diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 5a54dca7..d755a047 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -577,27 +577,38 @@ Operations for deploying and managing Harper components (applications, plugins). Detailed documentation: [Components Overview](../components/overview.md) -| Operation | Description | Role Required | -| ---------------------- | ----------------------------------------------------------------------- | ------------- | -| `add_component` | Creates a new component project from a template | super_user | -| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | -| `package_component` | Packages a component project into a base64-encoded tar | super_user | -| `drop_component` | Deletes a component or a file within a component | super_user | -| `get_components` | Lists all component files and config | super_user | -| `get_component_file` | Returns the contents of a file within a component | super_user | -| `set_component_file` | Creates or updates a file within a component | super_user | -| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | -| `update_ssh_key` | Updates an existing SSH key | super_user | -| `delete_ssh_key` | Deletes an SSH key | super_user | -| `list_ssh_keys` | Lists all configured SSH key names | super_user | -| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | -| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | -| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | +| Operation | Description | Role Required | +| --------------------------- | ----------------------------------------------------------------------- | ------------- | +| `add_component` | Creates a new component project from a template | super_user | +| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | +| `package_component` | Packages a component project into a base64-encoded tar | super_user | +| `drop_component` | Deletes a component or a file within a component | super_user | +| `get_components` | Lists all component files and config | super_user | +| `get_component_file` | Returns the contents of a file within a component | super_user | +| `set_component_file` | Creates or updates a file within a component | super_user | +| `list_deployments` | Lists deployment records with optional filters | super_user | +| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | +| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | +| `delete_deployment_payload` | Removes the stored tarball to free space | super_user | +| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | +| `update_ssh_key` | Updates an existing SSH key | super_user | +| `delete_ssh_key` | Deletes an SSH key | super_user | +| `list_ssh_keys` | Lists all configured SSH key names | super_user | +| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | +| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | +| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | ### `deploy_component` Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. +Additional parameters: + +- `urlPath` — override the HTTP URL path the component is mounted at (e.g. `"/api/v2"`) +- `install_allow_scripts` — set to `true` to allow npm pre/post install scripts (disabled by default) + +The response includes a `deployment_id` that can be used to query the deployment record: + ```json { "operation": "deploy_component", @@ -608,6 +619,95 @@ Deploys a component. The `package` option accepts any valid NPM reference includ } ``` +Response: + +```json +{ + "deployment_id": "a3f8c2d1...", + "message": "Component deployed successfully" +} +``` + +### Deployment Operations + +Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output. + +### `list_deployments` + +Returns a list of deployment records, newest first. All filter parameters are optional. + +| Parameter | Type | Description | +| --------- | ------ | ------------------------------------------------ | +| `project` | string | Filter to a specific component project | +| `status` | string | Filter by status: `pending`, `success`, `failed` | +| `since` | number | Start of time range (Unix timestamp ms) | +| `until` | number | End of time range (Unix timestamp ms) | +| `limit` | number | Maximum number of results (default: 100) | +| `offset` | number | Pagination offset | + +```json +{ + "operation": "list_deployments", + "project": "my-app", + "status": "success", + "limit": 20 +} +``` + +Response includes a `deployments` array and a `total` count. The `payload_blob` field is stripped from list responses for size; use `get_deployment_payload` to retrieve the tarball. + +### `get_deployment` + +Returns a single deployment record by `deployment_id`. When called on an in-progress deployment via a request that accepts `text/event-stream`, the response streams live phase events and install output as Server-Sent Events, replaying the buffered event log then tailing until the deployment reaches a terminal status. + +```json +{ + "operation": "get_deployment", + "deployment_id": "a3f8c2d1..." +} +``` + +The deployment record includes: + +| Field | Description | +| -------------------- | ----------------------------------------------------------------------- | +| `deployment_id` | Unique identifier (content hash) | +| `project` | Component project name | +| `package_identifier` | Package reference or `payload` for tar uploads | +| `status` | `pending`, `success`, `failed`, or `rolled_back` | +| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | +| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | +| `peer_results` | Per-node outcome map for replicated deployments | +| `payload_hash` | SHA-256 hash of the deployment tarball | +| `payload_size` | Byte size of the deployment tarball | +| `started_at` | Timestamp when deployment began | +| `completed_at` | Timestamp when deployment finished | +| `user` | User who initiated the deployment | +| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | +| `error` | Error message for failed deployments | + +### `get_deployment_payload` + +Returns the raw tarball for a deployment. Useful for inspecting or re-deploying a specific version. + +```json +{ + "operation": "get_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + +### `delete_deployment_payload` + +Removes the tarball blob from a deployment record. The deployment record itself is retained; only the binary payload is deleted. Use this to reclaim storage after confirming a deployment is stable. + +```json +{ + "operation": "delete_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + ### `add_ssh_key` Adds an SSH key (must be ed25519) for authenticating deployments from private repositories. diff --git a/release-notes/v5-lincoln/5.1.md b/release-notes/v5-lincoln/5.1.md new file mode 100644 index 00000000..5aa67fc3 --- /dev/null +++ b/release-notes/v5-lincoln/5.1.md @@ -0,0 +1,126 @@ +--- +title: '5.1' +--- + +# 5.1 Release Notes + +### Patch Releases + +All patch release notes for 5.1.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.1&expanded=true). + +## AI Models Integration + +Harper 5.1 introduces a built-in models layer that provides a unified interface for AI model backends. This enables embedding generation and text generation from directly within Harper applications, without managing external API connections per-application. + +The models layer is exposed via `scope.models` in application code: + +```js +// Generate embeddings +const vector = await scope.models.embed('text to embed', { model: 'my-embedding-model' }); + +// Generate text +const response = await scope.models.generate([{ role: 'user', content: 'Hello' }], { model: 'my-chat-model' }); + +// Streaming generation +for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) { + // ... +} +``` + +Supported backends are Anthropic, AWS Bedrock, OpenAI, and Ollama, configured under the `models` key in `harper-config.yaml`: + +```yaml +models: + anthropic: + apiKey: your-api-key + openai: + apiKey: your-api-key + baseUrl: https://api.openai.com/v1 # optional override + ollama: + baseUrl: http://localhost:11434 + bedrock: + region: us-east-1 +``` + +### `@embed` Schema Directive + +The `@embed` directive automates vector embedding at the schema level, eliminating the need for application code to compute and store embeddings on every write. Add it to any `Float` array field with a `source` pointing to the text field to embed: + +```graphql +type Document @table { + id: ID @primaryKey + content: String + embedding: [Float] @embed(source: "content", model: "my-embedding-model") @indexed(type: "HNSW") +} +``` + +On every write, Harper automatically calls the specified model to compute the embedding from `source` and stores it on the record. The `@indexed(type: "HNSW")` index is attached automatically when not explicitly specified. + +## MCP Server + +Harper 5.1 includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server, allowing LLM clients such as Claude Desktop, Cursor, and Zed to connect directly to a Harper instance and interact with its data and operations. + +The MCP server exposes two profiles: + +- **Operations profile** — wraps Harper's operations catalog as tools, with a curated default allow-list of read-only operations +- **Application profile** — auto-generates tools from a Harper application's Resource verb methods + +The `harper mcp` CLI command provides a stdio bridge for use with MCP clients: + +```bash +# Generate a config block for Claude Desktop +harper mcp print-config --client claude-desktop + +# Run diagnostics against a running instance +harper mcp doctor +``` + +MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the MCP reference documentation for full configuration options, authentication, and tool customization. + +## Deployment Tracking + +`deploy_component` now records a full audit trail for every deployment in the `system.hdb_deployment` system table. Each deployment gets a `deployment_id` and tracks phases (prepare → load → replicate → restart → success), per-node outcomes, and a bounded event log capturing install output. + +The response from `deploy_component` now includes a `deployment_id`: + +```json +{ + "deployment_id": "a3f8c2...", + "message": "Component deployed successfully" +} +``` + +New operations provide access to deployment history: + +- `list_deployments` — query deployment history with filters +- `get_deployment` — fetch a single deployment record; supports live SSE streaming for in-progress deploys +- `get_deployment_payload` — retrieve the stored tarball for a deployment +- `delete_deployment_payload` — free storage by removing the payload blob after deployment + +See [Deployment Operations](#deployment-operations) in the Operations API reference for details. + +## HNSW int8 Quantization + +HNSW vector indexes now support `int8` quantization, reducing index storage by approximately 3× and improving search throughput approximately 5× with around 1% recall loss at recall@10: + +```graphql +type Document @table { + embedding: [Float] @indexed(type: "HNSW", quantization: "int8") +} +``` + +Search uses asymmetric scoring: queries use full-precision float vectors while the index graph uses int8, and results are reranked against full-precision vectors before returning. The full-precision vector is always stored on the record itself. + +Per-query `ef` can be overridden at query time for applications that need to tune the recall/latency tradeoff dynamically. + +## Replication Improvements + +Several replication reliability improvements are included in 5.1: + +- **Resumable bulk clone** — interrupted full-table copies resume from where they left off rather than restarting from the beginning +- **Client-side receive watchdog** — dead WebSocket connections are now detected and recovered without waiting for a server-side timeout +- **Wedge detection and recovery** — stalled replication streams are detected and re-subscribed automatically +- **`isLeader` flag on `add_node`** — explicitly request a full-table copy when joining a cluster, independent of the normal subscription logic +- **`replication.pingInterval` / `replication.pingTimeout`** — configurable keepalive intervals for replication connections (values in milliseconds) + +Please see the [migration guide](./v5-migration.md) for suggestions on how to migrate from v4 to v5.