E2E: Portable Hermes registry cannot create a second OS thread on the GPU runner #13482

Investigation Summary

• Trusted portable-hermes-finalization failed on canonical main at merge commit b01bb7ffc1bc0e1e7c8b015c7103039998aa9a6e before any Hermes behavior ran.
• The retained registry artifact shows a running container and published 127.0.0.1:5000 mapping, but the registry process reports failed to create new OS thread (have 2 already; errno=11) and fatal error: newosproc.
• The same newosproc signature occurred on the exact PR head in run 38951030892. An exact-base comparison also failed in the same managed-registry readiness phase, so this is inherited infrastructure behavior rather than a regression from PR test(e2e): retain portable registry readiness evidence #13352.
• Current trusted setup creates a delegated user-level Podman service without recording LimitNPROC, TasksMax, or cgroup pids headroom. The managed registry launch has no explicit PID limit. Existing evidence cannot yet distinguish RLIMIT_NPROC, cgroup pids.max, or host PID exhaustion.
• No duplicate owns this failure. [All Platforms][Onboard] Portable profile's hardcoded registry IP has no recovery when Docker IPAM state gets stuck #11409 is stale Docker IPAM state; [SECURITY] No process limit (ulimit -u unlimited) — fork bomb unprotected #809/fix: harden sandbox image — remove gcc/netcat, add process limits, drop capabilities #830 and [All Platform][Security] Sandbox does not enforce nproc / nofile rlimits — ulimit -u = unlimited, ulimit -n = 1048576 inside sandbox shell #2173 concern product-sandbox process-limit enforcement, not the trusted E2E runtime.

Description

The explicit portable-hermes-finalization E2E cannot reach its Portable Hermes assertions on the reviewed x86-64 NVIDIA GPU runner. The rootless Podman runtime starts and accepts API requests, then the managed local registry container is created and remains reported as running. Its Go process cannot create a second OS thread, so the required unauthenticated HTTP 200 readiness check exhausts ten bounded read-only probes.

Expected behavior: trusted runtime setup provides and verifies enough bounded process/thread headroom for the pinned registry, or fails before candidate execution with an actionable infrastructure classification and retained controller evidence.

Actual behavior: the workflow reaches the live test, repeatedly observes connection refusal or reset, and fails before image publication or Hermes behavior. This blocks final canonical-main verification for #13129 and therefore blocks dependent PR #12244.

This is trusted CI infrastructure scope only. It does not establish a product change or justify modifying the production portable profile without a non-CI reproduction.

Reproduction Steps

Existing runs are sufficient evidence and must not be rerun unchanged.

1. Inspect focused canonical-main run 39256621467, attempt 1, job Portable Hermes finalization (Podman 5.7, NVIDIA GPU).
2. Confirm the trusted workflow and tested SHA are b01bb7ffc1bc0e1e7c8b015c7103039998aa9a6e.
3. Read retained artifact e2e-portable-hermes-finalization.
4. Observe:
   • portable-registry-readiness-retry.json: 10 exhausted read-only probes, each classified transient-external.
   • portable-registry-readiness-failure.json: final transport failure connection-reset.
   • container inspection: status=running, running=true, exit_code=0.
   • port mapping: 127.0.0.1:5000.
   • registry log tail: Go newosproc failure after two OS threads.
5. Compare exact-head run 38951030892, which retained the same newosproc signature.

Environment

• Workflow: E2E / E2E
• Selector: jobs=portable-hermes-finalization
• Runner label: linux-amd64-gpu-rtxpro6000-latest-1
• Architecture: x86_64
• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
• Runtime: pinned rootless Podman 5.7.0 with runc and cgroups v2
• Registry image: pinned registry:2 digest
• Canonical-main run: https://github.com/NVIDIA/NemoClaw/actions/runs/39256621467
• Exact-head comparison: https://github.com/NVIDIA/NemoClaw/actions/runs/38951030892
• Exact-base phase comparison: https://github.com/NVIDIA/NemoClaw/actions/runs/39163604619

Debug Output

Current source boundaries:

• The focused job and mandatory cleanup are owned by .github/workflows/e2e.yaml.
• .github/actions/setup-native-podman-e2e/action.yaml creates the delegated user systemd Podman service but does not capture or set effective task/process ceilings.
• src/lib/onboard/experimental/portable-host-preparation.ts launches the managed registry through the selected rootless runtime without a PID-limit argument.
• test/e2e/fixtures/portable-registry-readiness.ts correctly preserves bounded failure evidence and must remain fail-closed.

Confirmed failure class: infrastructure process/thread resource exhaustion at managed-registry startup.

Still unknown: which authoritative controller denies thread creation. The repair must identify the effective RLIMIT and cgroup state before choosing a limit change. It must not assume that increasing one limit fixes the actual enforcing boundary.

Logs

runtime: failed to create new OS thread (have 2 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

AssertionError: The managed local registry did not become ready.
See the bounded registry readiness artifacts.

Acceptance Criteria

• Trusted setup records bounded, credential-free evidence for the effective service/container process limits and current headroom needed to distinguish RLIMIT_NPROC, systemd TasksMax, cgroup pids.max, and host exhaustion.
• The authoritative trusted boundary supplies a reviewed, finite minimum headroom sufficient for the pinned Podman service and registry. Do not set an unlimited value unless a separate security decision explicitly requires it.
• Insufficient or inconsistent headroom fails before candidate behavior with an infrastructure classification and actionable evidence.
• Deterministic tests protect the systemd/service/container limit contract, ordering before candidate execution, redaction, and cleanup.
• A fresh focused canonical-main portable-hermes-finalization run reaches the registry HTTP 200 assertion and all correction-specific Hermes assertions.
• Artifact upload, registered-resource release, Docker restoration, cleanup-fixture removal, and Docker-auth cleanup remain mandatory and pass.
• Do not broaden retry signatures, rerun the failed workflow unchanged, weaken the HTTP readiness contract, or change production portable behavior without an independent reproduction.

Related Work

• test(e2e): retain portable registry readiness evidence #13352 added the bounded readiness and diagnostic evidence that exposed this cause; it did not remediate runner process limits.
• fix(ci): resolve portable OpenShell install path #13217 repaired the preceding OpenShell installation boundary.
• fix(ci): restore durable native Podman E2E staging #13129 owns the original Portable Hermes lifecycle and awaits successful canonical-main verification.
• fix(onboard): use native Hermes readiness in finalization #12244 remains dependent on that verification.
• [All Platforms][Onboard] Portable profile's hardcoded registry IP has no recovery when Docker IPAM state gets stuck #11409 affects the same registry but has a distinct stale-IPAM cause.
• [SECURITY] No process limit (ulimit -u unlimited) — fork bomb unprotected #809, fix: harden sandbox image — remove gcc/netcat, add process limits, drop capabilities #830, and [All Platform][Security] Sandbox does not enforce nproc / nofile rlimits — ulimit -u = unlimited, ulimit -n = 1048576 inside sandbox shell #2173 concern sandbox security limits and are not duplicates.

Checklist

• I confirmed this bug is reproducible.
• I searched existing issues and this is not a duplicate.