Skip to content

docs(v5.1): release notes, deployment tracking operations #524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kriszyp wants to merge 1 commit into
base: main
Choose a base branch
from
+243 −16
Open

docs(v5.1): release notes, deployment tracking operations #524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 116 additions & 16 deletions reference/operations-api/operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -577,27 +577,38 @@ Operations for deploying and managing Harper components (applications, plugins).

Detailed documentation: [Components Overview](../components/overview.md)

| Operation | Description | Role Required |
| ---------------------- | ----------------------------------------------------------------------- | ------------- |
| `add_component` | Creates a new component project from a template | super_user |
| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user |
| `package_component` | Packages a component project into a base64-encoded tar | super_user |
| `drop_component` | Deletes a component or a file within a component | super_user |
| `get_components` | Lists all component files and config | super_user |
| `get_component_file` | Returns the contents of a file within a component | super_user |
| `set_component_file` | Creates or updates a file within a component | super_user |
| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user |
| `update_ssh_key` | Updates an existing SSH key | super_user |
| `delete_ssh_key` | Deletes an SSH key | super_user |
| `list_ssh_keys` | Lists all configured SSH key names | super_user |
| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user |
| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user |
| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user |
| Operation | Description | Role Required |
| ---------------------------- | ----------------------------------------------------------------------- | ------------- |
| `add_component` | Creates a new component project from a template | super_user |
| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user |
| `package_component` | Packages a component project into a base64-encoded tar | super_user |
| `drop_component` | Deletes a component or a file within a component | super_user |
| `get_components` | Lists all component files and config | super_user |
| `get_component_file` | Returns the contents of a file within a component | super_user |
| `set_component_file` | Creates or updates a file within a component | super_user |
| `list_deployments` | Lists deployment records with optional filters | super_user |
| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user |
| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user |
| `delete_deployment_payload` | Removes the stored tarball to free space | super_user |
| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user |
| `update_ssh_key` | Updates an existing SSH key | super_user |
| `delete_ssh_key` | Deletes an SSH key | super_user |
| `list_ssh_keys` | Lists all configured SSH key names | super_user |
| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user |
| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user |
| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user |

### `deploy_component`

Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`.

Additional parameters:

- `urlPath` — override the HTTP URL path the component is mounted at (e.g. `"/api/v2"`)
- `install_allow_scripts` — set to `true` to allow npm pre/post install scripts (disabled by default)

The response includes a `deployment_id` that can be used to query the deployment record:

```json
{
"operation": "deploy_component",
Expand All @@ -608,6 +619,95 @@ Deploys a component. The `package` option accepts any valid NPM reference includ
}
```

Response:

```json
{
"deployment_id": "a3f8c2d1...",
"message": "Component deployed successfully"
}
```

### Deployment Operations

Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output.

### `list_deployments`

Returns a list of deployment records, newest first. All filter parameters are optional.

| Parameter | Type | Description |
| ---------- | -------- | -------------------------------------------------- |
| `project` | string | Filter to a specific component project |
| `status` | string | Filter by status: `pending`, `success`, `failed` |
| `since` | number | Start of time range (Unix timestamp ms) |
| `until` | number | End of time range (Unix timestamp ms) |
| `limit` | number | Maximum number of results (default: 100) |
| `offset` | number | Pagination offset |

```json
{
"operation": "list_deployments",
"project": "my-app",
"status": "success",
"limit": 20
}
```

Response includes a `deployments` array and a `total` count. The `payload_blob` field is stripped from list responses for size; use `get_deployment_payload` to retrieve the tarball.

### `get_deployment`

Returns a single deployment record by `deployment_id`. When called on an in-progress deployment via a request that accepts `text/event-stream`, the response streams live phase events and install output as Server-Sent Events, replaying the buffered event log then tailing until the deployment reaches a terminal status.

```json
{
"operation": "get_deployment",
"deployment_id": "a3f8c2d1..."
}
```

The deployment record includes:

| Field | Description |
| ------------------ | ------------------------------------------------------------------------ |
| `deployment_id` | Unique identifier (content hash) |
| `project` | Component project name |
| `package_identifier` | Package reference or `payload` for tar uploads |
| `status` | `pending`, `success`, `failed`, or `rolled_back` |
| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` |
| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) |
| `peer_results` | Per-node outcome map for replicated deployments |
| `payload_hash` | SHA-256 hash of the deployment tarball |
| `payload_size` | Byte size of the deployment tarball |
| `started_at` | Timestamp when deployment began |
| `completed_at` | Timestamp when deployment finished |
| `user` | User who initiated the deployment |
| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable |
| `error` | Error message for failed deployments |

### `get_deployment_payload`

Returns the raw tarball for a deployment. Useful for inspecting or re-deploying a specific version.

```json
{
"operation": "get_deployment_payload",
"deployment_id": "a3f8c2d1..."
}
```

### `delete_deployment_payload`

Removes the tarball blob from a deployment record. The deployment record itself is retained; only the binary payload is deleted. Use this to reclaim storage after confirming a deployment is stable.

```json
{
"operation": "delete_deployment_payload",
"deployment_id": "a3f8c2d1..."
}
```

### `add_ssh_key`

Adds an SSH key (must be ed25519) for authenticating deployments from private repositories.
Expand Down
127 changes: 127 additions & 0 deletions release-notes/v5-lincoln/5.1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: '5.1'
---

# 5.1 Release Notes

### Patch Releases

All patch release notes for 5.1.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.1&expanded=true).

## AI Models Integration

Harper 5.1 introduces a built-in models layer that provides a unified interface for AI model backends. This enables embedding generation and text generation from directly within Harper applications, without managing external API connections per-application.

The models layer is exposed via `scope.models` in application code:

```js
// Generate embeddings
const vector = await scope.models.embed('text to embed', { model: 'my-embedding-model' });

// Generate text
const response = await scope.models.generate([{ role: 'user', content: 'Hello' }], { model: 'my-chat-model' });

// Streaming generation
for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) {
// ...
}
```

Supported backends are Anthropic, AWS Bedrock, OpenAI, and Ollama, configured under the `models` key in `harper-config.yaml`:

```yaml
models:
anthropic:
apiKey: your-api-key
openai:
apiKey: your-api-key
baseUrl: https://api.openai.com/v1 # optional override
ollama:
baseUrl: http://localhost:11434
bedrock:
region: us-east-1
```

### `@embed` Schema Directive

The `@embed` directive automates vector embedding at the schema level, eliminating the need for application code to compute and store embeddings on every write. Add it to any `Float` array field with a `source` pointing to the text field to embed:

```graphql
type Document @table {
id: ID @primaryKey
content: String
embedding: [Float] @embed(source: "content", model: "my-embedding-model")
@indexed(type: "HNSW")
}
```

On every write, Harper automatically calls the specified model to compute the embedding from `source` and stores it on the record. The `@indexed(type: "HNSW")` index is attached automatically when not explicitly specified.

## MCP Server

Harper 5.1 includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server, allowing LLM clients such as Claude Desktop, Cursor, and Zed to connect directly to a Harper instance and interact with its data and operations.

The MCP server exposes two profiles:

- **Operations profile** — wraps Harper's operations catalog as tools, with a curated default allow-list of read-only operations
- **Application profile** — auto-generates tools from a Harper application's Resource verb methods

The `harper mcp` CLI command provides a stdio bridge for use with MCP clients:

```bash
# Generate a config block for Claude Desktop
harper mcp print-config --client claude-desktop

# Run diagnostics against a running instance
harper mcp doctor
```

MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the [MCP reference documentation](../reference/mcp/overview.md) for full configuration options, authentication, and tool customization.

## Deployment Tracking

`deploy_component` now records a full audit trail for every deployment in the `system.hdb_deployment` system table. Each deployment gets a `deployment_id` and tracks phases (prepare → load → replicate → restart → success), per-node outcomes, and a bounded event log capturing install output.

The response from `deploy_component` now includes a `deployment_id`:

```json
{
"deployment_id": "a3f8c2...",
"message": "Component deployed successfully"
}
```

New operations provide access to deployment history:

- `list_deployments` — query deployment history with filters
- `get_deployment` — fetch a single deployment record; supports live SSE streaming for in-progress deploys
- `get_deployment_payload` — retrieve the stored tarball for a deployment
- `delete_deployment_payload` — free storage by removing the payload blob after deployment

See [Deployment Operations](#deployment-operations) in the Operations API reference for details.

## HNSW int8 Quantization

HNSW vector indexes now support `int8` quantization, reducing index storage by approximately 3× and improving search throughput approximately 5× with around 1% recall loss at recall@10:

```graphql
type Document @table {
embedding: [Float] @indexed(type: "HNSW", quantization: "int8")
}
```

Search uses asymmetric scoring: queries use full-precision float vectors while the index graph uses int8, and results are reranked against full-precision vectors before returning. The full-precision vector is always stored on the record itself.

Per-query `ef` can be overridden at query time for applications that need to tune the recall/latency tradeoff dynamically.

## Replication Improvements

Several replication reliability improvements are included in 5.1:

- **Resumable bulk clone** — interrupted full-table copies resume from where they left off rather than restarting from the beginning
- **Client-side receive watchdog** — dead WebSocket connections are now detected and recovered without waiting for a server-side timeout
- **Wedge detection and recovery** — stalled replication streams are detected and re-subscribed automatically
- **`isLeader` flag on `add_node`** — explicitly request a full-table copy when joining a cluster, independent of the normal subscription logic
- **`replication.pingInterval` / `replication.pingTimeout`** — configurable keepalive intervals for replication connections (values in milliseconds)

Please see the [migration guide](./v5-migration.md) for suggestions on how to migrate from v4 to v5.
Loading