[Linux][Onboard] portable NemoHermes onboarding always fails its final secret-boundary check with a bare PRIVILEGED_CONTROL_UNAVAILABLE token #11892

Description

Portable NemoHermes onboarding never completes, so the portable profile cannot be used
with the Hermes agent at all on this build. Every stage succeeds - the rootless
Podman runtime starts, the sandbox is created and proves CUDA usability, the Hermes
Agent gateway reports healthy, the policy presets apply, and provider-neutral
inference is verified through the sandbox runtime - and then the final secret-boundary
check emits the bare token PRIVILEGED_CONTROL_UNAVAILABLE and the run is declared
incomplete.

The token is printed on a line by itself, with no sentence explaining what privileged
control was unavailable, what was expected, or what the operator should change. The
only guidance that follows is to inspect "the validator output above", but the token
is the entire validator output.

A managed (non-portable) NemoHermes onboard on the same host, same build, same agent
and same provider completes normally and reaches Ready; the secret-boundary check does
not appear in its output at all. The only difference between the two runs is
--experimental-profile portable, which makes this specific to the portable profile.

Context, because the marker itself is not new: a change titled "preserve secret-boundary
refusals through finalization" describes this same marker appearing on fresh Portable
Hermes onboarding while the run went on to print "Deployment verified" / "Hermes is
ready" and exit 0, and states that the marker means validation could not complete rather
than that a secret was exposed. That change made the refusal terminal, which is the
correct direction. What this report adds is that the underlying validation still cannot
complete on portable Hermes, so with the refusal now enforced the profile has no working
path at all - and that the enforced failure surfaces as a bare token with no explanation.

Platform scope: Reproduced on an x86_64 Ubuntu host with an NVIDIA A100 and rootless
Podman 5.7.0; other platforms not tested this pass. Portable
NemoHermes requires GPU acceleration authority, so a GPU-less host
cannot exercise this path.
Regression: Unknown - earlier builds not tested this pass.
OpenShell issue: No - the OpenShell gateway is reachable throughout, the sandbox is
created and reaches a healthy agent gateway, and the managed profile completes against
the same gateway. The failing check belongs to NemoClaw's portable onboarding flow.

Environment

Device:        x86_64 server with NVIDIA A100-SXM4-40GB
OS:            Ubuntu 25.04
Architecture:  x86_64
Kernel:        6.8.0-117-generic
Node.js:       v22.23.2
npm:           10.9.8
Podman:        podman version 5.7.0
Docker:        Docker version 29.5.2, build 79eb04c
OpenShell CLI: openshell 0.0.116
NemoHermes:    nemohermes v0.0.125

Steps to Reproduce

Start from a host with no portable state at all, so the failure cannot be attributed
to a leftover runner or a stale auth proxy.

1. Remove any previous portable state and stop the host Ollama service, so the
   portable runtime can bind its own Ollama on the loopback port:
     sudo systemctl stop ollama
     podman rm -f {every container}
     rm -rf ~/.nemoclaw/gateways ~/.nemoclaw/portable-inference
     rm -rf ~/.nemoclaw/hermes-portable-lifecycle ~/.nemoclaw/state
   Confirm nothing is listening on the Ollama port and no portable runner remains.

2. Pick an unused non-default gateway port and export NEMOCLAW_GATEWAY_PORT to it.

3. Run the portable onboard:
     nemohermes onboard --experimental-profile portable --fresh --name {sandbox} \
       --non-interactive --yes

4. Read the tail of the output, in particular the line printed after
   "Verifying provider-neutral inference through the sandbox runtime".

5. Control, on the same host and in the same session, without the portable profile:
     nemohermes onboard --fresh --name {other sandbox} --non-interactive --yes

Expected Result

Portable onboarding completes and the sandbox reaches Ready, the same way the managed
profile does on this host.

If the secret-boundary check genuinely cannot pass, the failure names what privileged
control was unavailable and what the operator should do about it, rather than printing
a bare token.

Actual Result

The portable run reaches the last step and then stops:

  [7/8] Setting up Hermes Agent inside sandbox
    Waiting for Hermes Agent gateway (up to 90s)...
    Hermes Agent gateway is healthy
  [8/8] Policy presets
    Applied preset: personal-open-internet
    Verifying provider-neutral inference through the sandbox runtime...
    Provider responds through inference.local inside the sandbox
    PRIVILEGED_CONTROL_UNAVAILABLE

    Secret-boundary check did not complete cleanly for Hermes gateway in '{sandbox}'.
    Refusing recovery; inspect the validator output above before re-running
    `nemoclaw {sandbox} recover`.
    Onboarding for '{sandbox}' is incomplete because a required process or
    secret-boundary check did not pass.

  exit 1

`doctor` disagrees with the onboarding result. Run against the same sandbox straight
after the failure it reports the lifecycle as healthy:

  NemoHermes doctor: {sandbox}
    [ok] Portable lifecycle: agent=Hermes; phase=active
    Summary: healthy

So a sandbox whose onboarding was declared incomplete is reported healthy by the
command the failure message tells the operator to run.

The control run on the same host, without the portable profile:

  nemohermes onboard --fresh --name {other sandbox}    exit 0,  Phase: Ready
  occurrences of PRIVILEGED_CONTROL_UNAVAILABLE                 0
  occurrences of any "Secret-boundary" line                     0

So the check that fails is only run by the portable profile, and the same host, build,
agent and provider complete successfully without it.

Logs

Portable run, stage by stage:

  rootless Podman runtime            started
  sandbox creation                   completed, CUDA usability proven
  Hermes Agent gateway               healthy within 90s
  policy presets                     personal-open-internet applied
  provider-neutral inference         responds through inference.local in the sandbox
  secret-boundary check              PRIVILEGED_CONTROL_UNAVAILABLE
  onboarding                         exit 1, reported incomplete

Managed run, same host and session:

  onboarding                         exit 0
  status                             Phase: Ready
  PRIVILEGED_CONTROL_UNAVAILABLE     not present
  "Secret-boundary"                  not present

Related

• fix(onboard): preserve secret-boundary refusals through finalization #11758 — "preserve secret-boundary refusals through finalization" describes this same
  marker appearing on fresh Portable Hermes onboarding while the run still printed
  "Deployment verified" / "Hermes is ready" and exited 0, and notes the marker means
  validation could not complete rather than that a secret was exposed. Making the refusal
  terminal is the right direction; this report is that the underlying validation still
  cannot complete on portable Hermes, so the profile now has no working path, and that the
  enforced failure surfaces as a bare token with no explanation.
• [Ubuntu 24.04][Onboard] Hermes Portable crashes with an uncaught exception on pre-existing gateway state instead of a clean recovery error #10486 — Hermes Portable crashes with an uncaught exception on pre-existing gateway
  state. That is about pre-existing state and an uncaught exception; this run starts from a
  fully cleared host and fails a specific named check with a controlled message.
• [DGX Spark][Onboard] every onboarding is refused when an empty ~/.config/nemoclaw/portable directory is not exactly mode 0700, and the error names no remedy #10740 — onboarding refused when an empty portable config directory is not exactly
  mode 0700. That is a permissions precondition rejected up front; here onboarding runs to
  the last step before failing.

Reproduced three times on this host, each from a fully cleared portable state (no leftover
runner, no stale auth proxy, host Ollama stopped so the portable runtime owns its loopback
port). Two earlier attempts that failed for other reasons — a port collision with the host
Ollama service, and a stale auth proxy bound to a non-loopback address — were eliminated
before this result was recorded.

Accepted implementation plan

• Decision: Accept.
• Reason: Current main can route Portable Hermes finalization through obsolete provider recovery after the native gateway is healthy. That path can emit PRIVILEGED_CONTROL_UNAVAILABLE, fail onboarding, and leave doctor reporting the same sandbox as healthy.
• Placement: main, in Portable Hermes finalization and doctor readiness reporting.
• Accountable maintainer: @rsliter.
• Outcome: Portable Hermes finalization and doctor use the same receipt-qualified native readiness result. Portable finalization does not use generic Docker/provider gateway recovery. Failures remain terminal but use redacted, actionable diagnostics.
• Acceptance:
  • A valid receipt and healthy native Hermes gateway complete onboarding without PRIVILEGED_CONTROL_UNAVAILABLE.
  • An authority, readiness, or secret-boundary failure keeps onboarding incomplete and prevents doctor from reporting healthy.
  • Managed Hermes, Portable OpenClaw, and custom-agent behavior do not change.
  • Focused tests and required CI pass before merge.
  • Prerequisite PR ci(e2e): add Portable Hermes GPU selector #12486 lands first to establish the trusted portable-hermes-finalization selector. PR fix(onboard): use native Hermes readiness in finalization #12244 then runs that self-contained selector against its unchanged exact head; passing live evidence is required before fix(onboard): use native Hermes readiness in finalization #12244 is ready for review.
• Exclusions: Do not restore the removed managed gateway controller. Do not add a public Portable profile support claim. Do not move this fix to the v1 redesign.
• Stop condition: Re-plan before changing native lifecycle ownership or adding a new supported product surface.

Live E2E validation amendment

• Missing semantic dimension: Portable Hermes finalization and doctor agreement on x86-64 NVIDIA GPU hardware with rootless Podman 5.7.
• Trusted selector owner: prerequisite PR ci(e2e): add Portable Hermes GPU selector #12486. The lane runs the existing portable-profile scenario on the reviewed RTX PRO 6000 runner, builds the pinned Podman 5.7 binary from the verified upstream source commit, and remains explicit-only.
• Behavior evidence owner: PR fix(onboard): use native Hermes readiness in finalization #12244. Its candidate scenario publishes exact receipt and registry authority, proves mismatched authority pauses onboarding and makes doctor fail, restores valid authority, verifies the authenticated Hermes health endpoint, and requires finalization and doctor to agree on healthy readiness.
• Dispatch: after ci(e2e): add Portable Hermes GPU selector #12486 lands, synchronize fix(onboard): use native Hermes readiness in finalization #12244 with main, then dispatch jobs=portable-hermes-finalization with targets= for the exact fix(onboard): use native Hermes readiness in finalization #12244 head and base commits. The selector chooses Podman 5.7 itself.
• Failure ownership: the live scenario owns receipt/registry authority disagreement; deterministic finalization and doctor tests retain unhealthy-gateway, missing-secret, lifecycle-lock, and redacted-diagnostic failure coverage.
• Scope: this explicit target is not part of default or release-required E2E, does not qualify inference, and does not widen Portable profile support.
• Timing decision: post-merge live evidence is accepted for prerequisite PR ci(e2e): add Portable Hermes GPU selector #12486 only because trusted main cannot dispatch a selector introduced by that candidate. This exception does not apply to PR fix(onboard): use native Hermes readiness in finalization #12244. After ci(e2e): add Portable Hermes GPU selector #12486 lands, exact-head live evidence remains required before fix(onboard): use native Hermes readiness in finalization #12244 is ready for review.