Repository navigation
feat: run agents through Fabric's fabric-agent-ctl - #13379
Merged
Merged
Conversation
Fabric v0.5.0's fabric-agent-ctl starts one agent runtime per sandbox, so a changed agent needs a new sandbox. Prepare for it on the current bridge: - Sandboxes are reconstructible: no prevent_destroy in generated graphs, and the openshell provider no longer refuses their replacement or removal. Durable data belongs on mounted volumes. - fabric_agent replaces on any configuration change, and planning refuses a replacement while its sandbox stays the same, explaining how to replace it. - Generated graphs hold each agent's configuration in a terraform_data that replaces its sandbox through replace_triggered_by. - fabric_agent no longer reports configuration changed inside the sandbox as drift; Fabric reports no applied configuration.
…acement # Conflicts: # crates/nemoclaw-openshell/src/lifecycle.rs
…acement # Conflicts: # docs/state.md
Agent images now run Fabric's own sandbox bridge, fabric-agent-ctl, instead of NemoClaw's Python bridge. fabric_agent starts the sandbox's one runtime (start), stops it on removal (stop), and exports its runtime_id for invocation (invoke). A sandbox runs one runtime, so the SDK replans with the sandbox replaced whenever a plan would create an agent in an existing sandbox. Fabric reports no runtime status or health, so fabric_agent_readiness, the bridge health path, and the stopped-runtime plan note are removed; apply now completes once agents start. The dummy reference image runs Fabric's runtime with a packaged NemoClaw test-only adapter.
The contract binary gained the sandbox replacement and agent removal tests, and measured 41.1 s summed on linux_arm64 against a 40 s budget.
The binary has grown to 62 tests and measured 197 s summed on linux_amd64 (175 s on linux_arm64) against a 190 s budget; this change adds the sandbox replacement tests.
…t size" This reverts commit 85ed9e8.
This reverts commit 38a231f.
The catalog label test imported /opt/nemoclaw/catalog.py through the PYTHONPATH the NemoClaw bridge set; with the bridge gone the import failed during image qualification.
FabricCatalog::bundled() parsed and checked the 50 KB generated catalog on every call, and every deployment compile reached it. That was about 40% of compiling the objects graph: the test compiling every example's deployment and teardown graphs dropped from 1.17 s to 0.86 s locally, and ran 8.7 s under CI load, over its 8 s budget.
# Conflicts: # crates/fabric-provider/src/agent.rs # crates/fabric-provider/src/configuration.rs # crates/fabric-provider/tests/contract/main.rs # crates/nemoclaw-e2e/tests/deployment.rs # crates/nemoclaw-e2e/tests/fabric_deployment.rs # crates/nemoclaw-test-fixtures/src/openshell.rs # docs/contributing/integration-tests.md # docs/provider.md # docs/troubleshooting.md
Contributor
|
v1 documentation preview: https://nvidia-preview-nemoclaw-v1-pr-13379.docs.buildwithfern.com/nemoclaw |
This was referenced Oct 17, 2026
gaveezy
added a commit
that referenced
this pull request
Oct 18, 2026
v1 removed crates/nemoclaw-e2e/tests/health.rs and its module line (#13379); this branch adds the fixture_keepalive module beside it. Keep fixture_keepalive and drop health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 18, 2026
cv
added a commit
that referenced
this pull request
Oct 18, 2026
…fusal (#13421) Part of #12731. Recreates #13161 by @cuongdangn on a fresh branch, because that branch doesn't allow maintainer edits. The commit keeps their authorship. ## Failure The live Kind tests would pass two regressions: - **SDK destroy:** the tests that deploy through the public SDK only checked that no destroy change named `kubernetes_storage`. They never read the cluster after `Deployment::destroy`, so they'd pass a destroy that recreated the namespace or key, deleted the gateway's claim, or left the gateway installed. - **Expired tokens:** the gateway test only checked that a token signed by an unknown key is refused. A gateway that stopped checking `exp` would still pass. ## Decision The same checks as #13161, adapted to current `v1`: - **Expired token:** after each install, the gateway test signs an expired development token with the deployment's own key and requires the gateway to refuse it. It also requires a current token from the same key to be accepted, so a refusal can only come from expiry. - **SDK destroy:** destroy must remove the gateway's Helm release record and StatefulSet. It must keep the namespace, the encryption key Secret and the gateway's PersistentVolumeClaim with unchanged UIDs. - **Rebased onto `v1`:** #13161 added a separate plain-Kubernetes deployment test for the destroy checks. Since #13379, apply completes through `fabric-agent-ctl`, and the registry test (`kubernetes_registry.rs`) already deploys an agent on Kubernetes through `deploy_image_and_export`. So the destroy checks run inside `deploy_image_and_export`, covering both the registry test and the OpenShift test, without adding another slow live test. Production code is unchanged. ## Validation - `cargo ci` passes locally: 1,589 unit tests and 153 lifecycle tests. - `cargo ci live-kind` passes locally: all 7 tests, including the new checks. Co-authored-by: Leo Dang <ledang@nvidia.com>
6 tasks
This was referenced Oct 18, 2026
cv
added a commit
that referenced
this pull request
Oct 18, 2026
…estroy checks (#13422) Part of #12731. Follows #13421, which merged before this narrowing reached it. ## Failure #13421 added three live Kind checks, two of which test other components: - **The gateway refusing an expired development token:** token verification is OpenShell's. - **The Helm release record and StatefulSet disappearing after SDK destroy:** that's Helm's and Kubernetes' behavior. NemoClaw's part, classifying an authentication failure and having its graph destroy the release, already has unit and plan-level coverage. ## Decision - **Removed:** both checks. - **Kept:** NemoClaw's own promise. After SDK destroy, the namespace, encryption key Secret and gateway PersistentVolumeClaims keep their UIDs, observed through #13421's more reliable `retained()`. - **Docs:** the live-tests guide and a stale test comment are updated. The comment claimed apply stops at the agent's health check, which hasn't been true since #13379. ## Validation This commit was tested on #13421's branch, whose code matches the merged #13421: - `cargo ci` passes locally: 1,589 unit tests and 153 lifecycle tests. - `cargo ci live-kind` passes locally: all 7 tests.
cv
added a commit
that referenced
this pull request
Oct 18, 2026
… Fabric (#13433) Closes #12016. ## Failure Downstream blueprints extend NemoClaw agent images: the VSS blueprint, for example, derives from one pinned by digest and adds skills, a domain CLI and workspace files. Nothing stated what such a derived image must keep. Trying it on current `v1` showed: - **A conforming derived image works:** labels carry over through `FROM`. - **A missing catalog label fails at plan time with a misleading message:** "Harness unavailable: The image declares no runtime for this adapter", then advice about `spec.gateway.engine`. - **A different Fabric revision in the image's catalog passes silently.** NemoClaw compared revisions, but the result, "unknown", was reported nowhere. - **An image from an older base fails at apply** with "sandbox unavailable: SANDBOX_PHASE_ERROR, reason unknown, exit code 1". OpenShell's supervisor log shows the cause: the image's launch command exited. An image from before the `fabric-agent-ctl` move lacks that command. ## Decision - **Fabric compatibility stays with Fabric.** NemoClaw no longer compares the image's Fabric revision with the release's. That comparison was NemoClaw judging another component, and it produced only a silent "unknown". The image's own `fabric-agent-ctl` validates the agent configuration and reports interface or validation errors when the agent starts (#13379, #13326). - **NemoClaw keeps checking its own contract,** with clearer messages: - **Missing catalog label:** "Harness unavailable" now names `io.nemoclaw.fabric.catalog` and the rule that a derived image keeps its base image's labels. The local-engine postcondition says the same. - **The adapter and the platform:** checked as before. - **Unexplained sandbox exits** now point to OpenShell's own log: "`openshell logs SANDBOX --source sandbox` shows why its process exited". Recognized OpenShell reasons keep their specific guidance. - **Docs:** a new [derived agent images](docs/derived-images.md) page: - deriving by digest; - what to keep: the catalog label, `/opt/fabric` with `fabric-agent-ctl`, UID and GID 10001, and the platform; - using added skills and tools; - rebuilding on upgrade; - which failures appear at plan time and which at apply. The compatibility, provider and build pages link to it. - **Base image publication** depends on releases (#12638). Until then, the page says to build the bases and push them to your own registry. ## Validation - **New tests**, each written before its change: - the missing-label message; - an unexplained sandbox exit pointing to `openshell logs`. - **Updated tests:** - a different Fabric revision now assesses as supported, with no `fabric_contract` check; - the prebuilt-images test allows exactly one build, the derived reference image. - **Live Docker:** the lifecycle test now runs an image derived from the reference image, with one added executable, and runs that executable inside the deployed sandbox. All 9 live Docker tests pass locally. - **By hand, with `nemoclaw plan` and `apply`:** a conforming derived image plans; a stale-revision image plans; a no-label image fails naming the label; a pre-`fabric-agent-ctl` image fails at apply, and its sandbox log shows why. - `cargo ci` passes locally: 1,600 unit tests and 149 lifecycle tests. The docs check reports 0 errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #13318. Closes #13031. Stacked on #13375; review that first.
Failure
Agent images ran NemoClaw's own Python bridge (
fabric.py,backend.py, and the bridge protocol, about 2,100 lines with tests) between OpenShell and Fabric. Fabric v0.5.0 ships its own sandbox bridge,fabric-agent-ctl, so NemoClaw was maintaining a second implementation of Fabric's job. Its health path also made every apply fail at the pinned Fabric, which has no health API (#13031).Decision
Images.
runtime.json's command is/opt/fabric/bin/fabric-agent-ctl; sandboxes launchfabric-agent-ctl serve. NemoClaw's bridge, its tests, and theio.nemoclaw.fabric.bridgelabel are gone.ADAPTER_PYTHONstays: Fabric reads it to choose the interpreter for Python adapters.Dummy reference image. It now runs Fabric's real runtime and
fabric-agent-ctl, with a NemoClaw test-only adapter (image/fabric/test-adapter) packaged like Fabric's own adapters. Catalogs come fromcatalog.pyfor every image; anadapters.jsonentry may name the descriptor file of an adapter Fabric's catalog package doesn't publish.fabric_agent.start --config -, retrying only whileserveisn't listening yet (host_unavailablewith no effects). It stores the returnedruntime_id, replacing therunningattribute.stop --runtime-id. A runtime already gone counts as stopped.runtime_exists: reported with guidance to replace the sandbox.One runtime per sandbox. A sandbox runs one runtime in its lifetime. After a stop, or a start that failed or lost its reply, it can't start another. When a plan would create an agent in a sandbox that already exists, the SDK plans again with that sandbox replaced (
-replace), and the plan shows it. Hand-written HCL users get the guidance instead.Readiness and health removed. This covers
fabric_agent_readiness,OperationResult.health,nemoclaw-backend's health types, the CLI's health output, and the plan's "agent runtime is not running" note. Apply now completes once agents start.Docs.
design/fabric-management.mdis rewritten aroundfabric-agent-ctl. The provider, usage, SDK, CLI, build, getting-started, Kubernetes and contributor docs are updated.Validation
cargo cipasses locally: 1530 unit tests, 149 lifecycle tests.cargo ci live-dockerpasses, 8 tests. This includes the dummy agent's full lifecycle through a managed Docker gateway with a realfabric-agent-ctl.cargo ci live-kindpasses, 5 tests.docker buildx bake checkpasses: lint, 15 unit tests, 5 reference tests drivingfabric-agent-ctl.cargo images qualify --lifecyclepasses for the dummy, Pi, Hermes and OpenClaw images: each starts, invokes and stops a runtime, and refuses a second.fern checkreports 0 errors.Not run: the Brev, Spark and hosted-parity live tests (hardware). They now expect a completed apply, and their health assertions are removed.
Follow-ups
runtime_id, which the docs read from OpenTofu state withjq. A CLI command to show it, or to invoke, would remove that step.