From 813890db9bace35711cf04942db592180dcff7bb Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 9 Jun 2026 22:47:54 -0600 Subject: [PATCH 1/3] docs(v5.1): release notes, deployment tracking ops, deploy_component updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add 5.1.md release notes covering: models/AI, @embed directive, MCP server, deployment tracking, HNSW int8 quantization, and replication improvements - Update deploy_component docs: urlPath, install_allow_scripts params, deployment_id response - Document new deployment operations: list_deployments, get_deployment, get_deployment_payload, delete_deployment_payload - Document hdb_deployment record schema (fields, phases, peer_results) Note: models/AI detail, MCP reference, and HNSW quantization have separate PRs (#523, #507/#516, #508) — this PR adds the release notes overview and the deployment tracking operations which had no coverage. Co-Authored-By: Claude Sonnet 4.6 --- reference/operations-api/operations.md | 132 ++++++++++++++++++++++--- release-notes/v5-lincoln/5.1.md | 127 ++++++++++++++++++++++++ 2 files changed, 243 insertions(+), 16 deletions(-) create mode 100644 release-notes/v5-lincoln/5.1.md diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 5a54dca7..8fa9b9ae 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -577,27 +577,38 @@ Operations for deploying and managing Harper components (applications, plugins). Detailed documentation: [Components Overview](../components/overview.md) -| Operation | Description | Role Required | -| ---------------------- | ----------------------------------------------------------------------- | ------------- | -| `add_component` | Creates a new component project from a template | super_user | -| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | -| `package_component` | Packages a component project into a base64-encoded tar | super_user | -| `drop_component` | Deletes a component or a file within a component | super_user | -| `get_components` | Lists all component files and config | super_user | -| `get_component_file` | Returns the contents of a file within a component | super_user | -| `set_component_file` | Creates or updates a file within a component | super_user | -| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | -| `update_ssh_key` | Updates an existing SSH key | super_user | -| `delete_ssh_key` | Deletes an SSH key | super_user | -| `list_ssh_keys` | Lists all configured SSH key names | super_user | -| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | -| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | -| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | +| Operation | Description | Role Required | +| ---------------------------- | ----------------------------------------------------------------------- | ------------- | +| `add_component` | Creates a new component project from a template | super_user | +| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | +| `package_component` | Packages a component project into a base64-encoded tar | super_user | +| `drop_component` | Deletes a component or a file within a component | super_user | +| `get_components` | Lists all component files and config | super_user | +| `get_component_file` | Returns the contents of a file within a component | super_user | +| `set_component_file` | Creates or updates a file within a component | super_user | +| `list_deployments` | Lists deployment records with optional filters | super_user | +| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | +| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | +| `delete_deployment_payload` | Removes the stored tarball to free space | super_user | +| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | +| `update_ssh_key` | Updates an existing SSH key | super_user | +| `delete_ssh_key` | Deletes an SSH key | super_user | +| `list_ssh_keys` | Lists all configured SSH key names | super_user | +| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | +| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | +| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | ### `deploy_component` Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. +Additional parameters: + +- `urlPath` — override the HTTP URL path the component is mounted at (e.g. `"/api/v2"`) +- `install_allow_scripts` — set to `true` to allow npm pre/post install scripts (disabled by default) + +The response includes a `deployment_id` that can be used to query the deployment record: + ```json { "operation": "deploy_component", @@ -608,6 +619,95 @@ Deploys a component. The `package` option accepts any valid NPM reference includ } ``` +Response: + +```json +{ + "deployment_id": "a3f8c2d1...", + "message": "Component deployed successfully" +} +``` + +### Deployment Operations + +Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output. + +### `list_deployments` + +Returns a list of deployment records, newest first. All filter parameters are optional. + +| Parameter | Type | Description | +| ---------- | -------- | -------------------------------------------------- | +| `project` | string | Filter to a specific component project | +| `status` | string | Filter by status: `pending`, `success`, `failed` | +| `since` | number | Start of time range (Unix timestamp ms) | +| `until` | number | End of time range (Unix timestamp ms) | +| `limit` | number | Maximum number of results (default: 100) | +| `offset` | number | Pagination offset | + +```json +{ + "operation": "list_deployments", + "project": "my-app", + "status": "success", + "limit": 20 +} +``` + +Response includes a `deployments` array and a `total` count. The `payload_blob` field is stripped from list responses for size; use `get_deployment_payload` to retrieve the tarball. + +### `get_deployment` + +Returns a single deployment record by `deployment_id`. When called on an in-progress deployment via a request that accepts `text/event-stream`, the response streams live phase events and install output as Server-Sent Events, replaying the buffered event log then tailing until the deployment reaches a terminal status. + +```json +{ + "operation": "get_deployment", + "deployment_id": "a3f8c2d1..." +} +``` + +The deployment record includes: + +| Field | Description | +| ------------------ | ------------------------------------------------------------------------ | +| `deployment_id` | Unique identifier (content hash) | +| `project` | Component project name | +| `package_identifier` | Package reference or `payload` for tar uploads | +| `status` | `pending`, `success`, `failed`, or `rolled_back` | +| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | +| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | +| `peer_results` | Per-node outcome map for replicated deployments | +| `payload_hash` | SHA-256 hash of the deployment tarball | +| `payload_size` | Byte size of the deployment tarball | +| `started_at` | Timestamp when deployment began | +| `completed_at` | Timestamp when deployment finished | +| `user` | User who initiated the deployment | +| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | +| `error` | Error message for failed deployments | + +### `get_deployment_payload` + +Returns the raw tarball for a deployment. Useful for inspecting or re-deploying a specific version. + +```json +{ + "operation": "get_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + +### `delete_deployment_payload` + +Removes the tarball blob from a deployment record. The deployment record itself is retained; only the binary payload is deleted. Use this to reclaim storage after confirming a deployment is stable. + +```json +{ + "operation": "delete_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + ### `add_ssh_key` Adds an SSH key (must be ed25519) for authenticating deployments from private repositories. diff --git a/release-notes/v5-lincoln/5.1.md b/release-notes/v5-lincoln/5.1.md new file mode 100644 index 00000000..7b40ad66 --- /dev/null +++ b/release-notes/v5-lincoln/5.1.md @@ -0,0 +1,127 @@ +--- +title: '5.1' +--- + +# 5.1 Release Notes + +### Patch Releases + +All patch release notes for 5.1.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.1&expanded=true). + +## AI Models Integration + +Harper 5.1 introduces a built-in models layer that provides a unified interface for AI model backends. This enables embedding generation and text generation from directly within Harper applications, without managing external API connections per-application. + +The models layer is exposed via `scope.models` in application code: + +```js +// Generate embeddings +const vector = await scope.models.embed('text to embed', { model: 'my-embedding-model' }); + +// Generate text +const response = await scope.models.generate([{ role: 'user', content: 'Hello' }], { model: 'my-chat-model' }); + +// Streaming generation +for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) { + // ... +} +``` + +Supported backends are Anthropic, AWS Bedrock, OpenAI, and Ollama, configured under the `models` key in `harper-config.yaml`: + +```yaml +models: + anthropic: + apiKey: your-api-key + openai: + apiKey: your-api-key + baseUrl: https://api.openai.com/v1 # optional override + ollama: + baseUrl: http://localhost:11434 + bedrock: + region: us-east-1 +``` + +### `@embed` Schema Directive + +The `@embed` directive automates vector embedding at the schema level, eliminating the need for application code to compute and store embeddings on every write. Add it to any `Float` array field with a `source` pointing to the text field to embed: + +```graphql +type Document @table { + id: ID @primaryKey + content: String + embedding: [Float] @embed(source: "content", model: "my-embedding-model") + @indexed(type: "HNSW") +} +``` + +On every write, Harper automatically calls the specified model to compute the embedding from `source` and stores it on the record. The `@indexed(type: "HNSW")` index is attached automatically when not explicitly specified. + +## MCP Server + +Harper 5.1 includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server, allowing LLM clients such as Claude Desktop, Cursor, and Zed to connect directly to a Harper instance and interact with its data and operations. + +The MCP server exposes two profiles: + +- **Operations profile** — wraps Harper's operations catalog as tools, with a curated default allow-list of read-only operations +- **Application profile** — auto-generates tools from a Harper application's Resource verb methods + +The `harper mcp` CLI command provides a stdio bridge for use with MCP clients: + +```bash +# Generate a config block for Claude Desktop +harper mcp print-config --client claude-desktop + +# Run diagnostics against a running instance +harper mcp doctor +``` + +MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the [MCP reference documentation](../reference/mcp/overview.md) for full configuration options, authentication, and tool customization. + +## Deployment Tracking + +`deploy_component` now records a full audit trail for every deployment in the `system.hdb_deployment` system table. Each deployment gets a `deployment_id` and tracks phases (prepare → load → replicate → restart → success), per-node outcomes, and a bounded event log capturing install output. + +The response from `deploy_component` now includes a `deployment_id`: + +```json +{ + "deployment_id": "a3f8c2...", + "message": "Component deployed successfully" +} +``` + +New operations provide access to deployment history: + +- `list_deployments` — query deployment history with filters +- `get_deployment` — fetch a single deployment record; supports live SSE streaming for in-progress deploys +- `get_deployment_payload` — retrieve the stored tarball for a deployment +- `delete_deployment_payload` — free storage by removing the payload blob after deployment + +See [Deployment Operations](#deployment-operations) in the Operations API reference for details. + +## HNSW int8 Quantization + +HNSW vector indexes now support `int8` quantization, reducing index storage by approximately 3× and improving search throughput approximately 5× with around 1% recall loss at recall@10: + +```graphql +type Document @table { + embedding: [Float] @indexed(type: "HNSW", quantization: "int8") +} +``` + +Search uses asymmetric scoring: queries use full-precision float vectors while the index graph uses int8, and results are reranked against full-precision vectors before returning. The full-precision vector is always stored on the record itself. + +Per-query `ef` can be overridden at query time for applications that need to tune the recall/latency tradeoff dynamically. + +## Replication Improvements + +Several replication reliability improvements are included in 5.1: + +- **Resumable bulk clone** — interrupted full-table copies resume from where they left off rather than restarting from the beginning +- **Client-side receive watchdog** — dead WebSocket connections are now detected and recovered without waiting for a server-side timeout +- **Wedge detection and recovery** — stalled replication streams are detected and re-subscribed automatically +- **`isLeader` flag on `add_node`** — explicitly request a full-table copy when joining a cluster, independent of the normal subscription logic +- **`replication.pingInterval` / `replication.pingTimeout`** — configurable keepalive intervals for replication connections (values in milliseconds) + +Please see the [migration guide](./v5-migration.md) for suggestions on how to migrate from v4 to v5. From dfdb229a47742649d8224bc260a723e40ff8cb13 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Thu, 11 Jun 2026 16:16:27 -0600 Subject: [PATCH 2/3] style: run prettier on changed files Co-Authored-By: Claude Sonnet 4.6 --- reference/operations-api/operations.md | 88 +++++++++++++------------- release-notes/v5-lincoln/5.1.md | 17 +++-- 2 files changed, 52 insertions(+), 53 deletions(-) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8fa9b9ae..d755a047 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -577,26 +577,26 @@ Operations for deploying and managing Harper components (applications, plugins). Detailed documentation: [Components Overview](../components/overview.md) -| Operation | Description | Role Required | -| ---------------------------- | ----------------------------------------------------------------------- | ------------- | -| `add_component` | Creates a new component project from a template | super_user | -| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | -| `package_component` | Packages a component project into a base64-encoded tar | super_user | -| `drop_component` | Deletes a component or a file within a component | super_user | -| `get_components` | Lists all component files and config | super_user | -| `get_component_file` | Returns the contents of a file within a component | super_user | -| `set_component_file` | Creates or updates a file within a component | super_user | -| `list_deployments` | Lists deployment records with optional filters | super_user | -| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | -| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | -| `delete_deployment_payload` | Removes the stored tarball to free space | super_user | -| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | -| `update_ssh_key` | Updates an existing SSH key | super_user | -| `delete_ssh_key` | Deletes an SSH key | super_user | -| `list_ssh_keys` | Lists all configured SSH key names | super_user | -| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | -| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | -| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | +| Operation | Description | Role Required | +| --------------------------- | ----------------------------------------------------------------------- | ------------- | +| `add_component` | Creates a new component project from a template | super_user | +| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | +| `package_component` | Packages a component project into a base64-encoded tar | super_user | +| `drop_component` | Deletes a component or a file within a component | super_user | +| `get_components` | Lists all component files and config | super_user | +| `get_component_file` | Returns the contents of a file within a component | super_user | +| `set_component_file` | Creates or updates a file within a component | super_user | +| `list_deployments` | Lists deployment records with optional filters | super_user | +| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | +| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | +| `delete_deployment_payload` | Removes the stored tarball to free space | super_user | +| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | +| `update_ssh_key` | Updates an existing SSH key | super_user | +| `delete_ssh_key` | Deletes an SSH key | super_user | +| `list_ssh_keys` | Lists all configured SSH key names | super_user | +| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | +| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | +| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | ### `deploy_component` @@ -636,14 +636,14 @@ Harper records every `deploy_component` call in the `system.hdb_deployment` tabl Returns a list of deployment records, newest first. All filter parameters are optional. -| Parameter | Type | Description | -| ---------- | -------- | -------------------------------------------------- | -| `project` | string | Filter to a specific component project | -| `status` | string | Filter by status: `pending`, `success`, `failed` | -| `since` | number | Start of time range (Unix timestamp ms) | -| `until` | number | End of time range (Unix timestamp ms) | -| `limit` | number | Maximum number of results (default: 100) | -| `offset` | number | Pagination offset | +| Parameter | Type | Description | +| --------- | ------ | ------------------------------------------------ | +| `project` | string | Filter to a specific component project | +| `status` | string | Filter by status: `pending`, `success`, `failed` | +| `since` | number | Start of time range (Unix timestamp ms) | +| `until` | number | End of time range (Unix timestamp ms) | +| `limit` | number | Maximum number of results (default: 100) | +| `offset` | number | Pagination offset | ```json { @@ -669,22 +669,22 @@ Returns a single deployment record by `deployment_id`. When called on an in-prog The deployment record includes: -| Field | Description | -| ------------------ | ------------------------------------------------------------------------ | -| `deployment_id` | Unique identifier (content hash) | -| `project` | Component project name | -| `package_identifier` | Package reference or `payload` for tar uploads | -| `status` | `pending`, `success`, `failed`, or `rolled_back` | -| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | -| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | -| `peer_results` | Per-node outcome map for replicated deployments | -| `payload_hash` | SHA-256 hash of the deployment tarball | -| `payload_size` | Byte size of the deployment tarball | -| `started_at` | Timestamp when deployment began | -| `completed_at` | Timestamp when deployment finished | -| `user` | User who initiated the deployment | -| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | -| `error` | Error message for failed deployments | +| Field | Description | +| -------------------- | ----------------------------------------------------------------------- | +| `deployment_id` | Unique identifier (content hash) | +| `project` | Component project name | +| `package_identifier` | Package reference or `payload` for tar uploads | +| `status` | `pending`, `success`, `failed`, or `rolled_back` | +| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | +| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | +| `peer_results` | Per-node outcome map for replicated deployments | +| `payload_hash` | SHA-256 hash of the deployment tarball | +| `payload_size` | Byte size of the deployment tarball | +| `started_at` | Timestamp when deployment began | +| `completed_at` | Timestamp when deployment finished | +| `user` | User who initiated the deployment | +| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | +| `error` | Error message for failed deployments | ### `get_deployment_payload` diff --git a/release-notes/v5-lincoln/5.1.md b/release-notes/v5-lincoln/5.1.md index 7b40ad66..fdd30caa 100644 --- a/release-notes/v5-lincoln/5.1.md +++ b/release-notes/v5-lincoln/5.1.md @@ -23,7 +23,7 @@ const response = await scope.models.generate([{ role: 'user', content: 'Hello' } // Streaming generation for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) { - // ... + // ... } ``` @@ -35,7 +35,7 @@ models: apiKey: your-api-key openai: apiKey: your-api-key - baseUrl: https://api.openai.com/v1 # optional override + baseUrl: https://api.openai.com/v1 # optional override ollama: baseUrl: http://localhost:11434 bedrock: @@ -48,10 +48,9 @@ The `@embed` directive automates vector embedding at the schema level, eliminati ```graphql type Document @table { - id: ID @primaryKey - content: String - embedding: [Float] @embed(source: "content", model: "my-embedding-model") - @indexed(type: "HNSW") + id: ID @primaryKey + content: String + embedding: [Float] @embed(source: "content", model: "my-embedding-model") @indexed(type: "HNSW") } ``` @@ -86,8 +85,8 @@ The response from `deploy_component` now includes a `deployment_id`: ```json { - "deployment_id": "a3f8c2...", - "message": "Component deployed successfully" + "deployment_id": "a3f8c2...", + "message": "Component deployed successfully" } ``` @@ -106,7 +105,7 @@ HNSW vector indexes now support `int8` quantization, reducing index storage by a ```graphql type Document @table { - embedding: [Float] @indexed(type: "HNSW", quantization: "int8") + embedding: [Float] @indexed(type: "HNSW", quantization: "int8") } ``` From a2d5c8935f9c415d2d4f5a52fa1ff8913a203f51 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Thu, 11 Jun 2026 16:23:56 -0600 Subject: [PATCH 3/3] fix: remove cross-plugin MCP link that breaks Docusaurus build The release-notes and reference doc plugins are separate; relative .md links between them resolve incorrectly. Removing until PR #507 (MCP reference section) merges and can be linked with an absolute path. Co-Authored-By: Claude Sonnet 4.6 --- release-notes/v5-lincoln/5.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/v5-lincoln/5.1.md b/release-notes/v5-lincoln/5.1.md index fdd30caa..5aa67fc3 100644 --- a/release-notes/v5-lincoln/5.1.md +++ b/release-notes/v5-lincoln/5.1.md @@ -75,7 +75,7 @@ harper mcp print-config --client claude-desktop harper mcp doctor ``` -MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the [MCP reference documentation](../reference/mcp/overview.md) for full configuration options, authentication, and tool customization. +MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the MCP reference documentation for full configuration options, authentication, and tool customization. ## Deployment Tracking