fix: reap the headroom proxy and read zombie pids as dead

docs/architecture.md

@@ -103,6 +103,8 @@ When a launch resolves `headroom` on, the launcher (synchronous end to end, righ
 
 The supervisor is the only thing that starts, stops, restarts, or upgrades headroom. Its whole lifecycle (install via `uv tool install` when the binary is missing or its version fails the configured source, crash restarts with bounded exponential backoff and a retry budget, drift restarts deferred until the session registry is empty, idle shutdown) lives in `src/headroom/supervisor.ts` as one pure-ish loop over injected `SupervisorPorts`, so every decision is tested against a fake clock, filesystem, and process table. Coordination between separate OS processes is entirely file-based (state, lock, session files) with pid liveness as the source of truth, which is what lets a synchronous launcher, a detached supervisor, and several concurrent sessions cooperate without any of them holding a socket open to another.
 
+Two facts about process death shape that liveness layer. First, the supervisor keeps every spawned headroom's `ChildProcess` handle and consumes its exit event: that event is both the crash signal and the reaping (a child nobody listens for is a child nobody reaps, and the loop's sleeps run on real timers precisely so the event loop can deliver it). Second, every liveness check anywhere in the coordination layer (crash detection, session pruning, state inspection, the identity lock's holder check) is zombie-aware via `ps`'s state column: a process that exited unreaped still answers `kill(pid, 0)` as alive, but holds no port and will never write state, so it must read as dead. Stops escalate SIGTERM to SIGKILL on a bounded grace, because the proxy has been observed to ignore SIGTERM outright. The supervisor handles SIGTERM and SIGINT explicitly and routes them through an orderly exit (Node skips `exit` handlers on unhandled signal death), so its own death takes its children with it; and a successor supervisor stops any orphan daemon a predecessor left running before starting its own, so nothing squatting on a port escapes supervision.
+
 ### Resolver mechanics
 
 For each `~/.claude` entry, walk the cascade to a boolean decision. If the decision is uniform for an entire subtree, symlink that directory in one shot. If a deeper path override splits the decision, materialise that directory as a real local directory instead of a symlink and recurse, repeating the check at each level — only directories with an actual split ever get exploded. A conditional entries key is never eligible for the uniform-symlink shortcut, since its decision can only be evaluated per-file.

docs/configuration-model.md

@@ -229,7 +229,7 @@ The ambient-credential guard (below) checks the parent environment and is unaffe
 
 `launch.headroom: true` (in a configuration profile, the global config, a directory rule, or a committed `.claude-use.json`, resolved through the same cascade as every other launch flag) or a one-off `CLAUDE_USE_HEADROOM=1 claude` routes the whole session through a local [headroom](https://github.com/ExaDev/headroom) daemon instead of straight to the provider: the child's `ANTHROPIC_BASE_URL` becomes the daemon's loopback address, `HEADROOM_PROXY_URL` names it too, and `ANTHROPIC_CUSTOM_HEADERS` gains `x-headroom-project-id` (the git repository root of the working directory, or the directory itself outside a repository) plus, when a provider is also selected, `x-headroom-base-url` carrying the provider's real upstream so one daemon can serve several providers per request.
 
-claude-use fully orchestrates the daemon; you never start, stop, or upgrade headroom by hand. The first launch that resolves headroom on spawns a detached supervisor (a background copy of the `claude-use` binary running a hidden internal subcommand), which installs headroom with `uv tool install` when the binary is missing or its version does not satisfy the configured source, starts `headroom proxy` on a free loopback port with `HEADROOM_ALLOWED_BASE_URLS` set to every provider's base URL plus `https://api.anthropic.com`, waits for its `/readyz` to answer, and only then records the port where launches can find it. A proxy that crashes is restarted with bounded exponential backoff; after five consecutive failures to become ready the supervisor records the error in its state and gives up, and the next launch fails loudly with the daemon log path rather than silently bypassing headroom. When the allowlist or install source drifts (a provider file changed, the configured source changed), the daemon is restarted only once no session is live, so a running session is never cut off; when no session has been live for `idleShutdownMinutes` (15 by default), the supervisor stops the daemon and exits, freeing its memory.
+claude-use fully orchestrates the daemon; you never start, stop, or upgrade headroom by hand. The first launch that resolves headroom on spawns a detached supervisor (a background copy of the `claude-use` binary running a hidden internal subcommand), which installs headroom with `uv tool install` when the binary is missing or its version does not satisfy the configured source, starts `headroom proxy` on a free loopback port with `HEADROOM_ALLOWED_BASE_URLS` set to every provider's base URL plus `https://api.anthropic.com`, waits for its `/readyz` to answer, and only then records the port where launches can find it. A proxy that crashes is restarted with bounded exponential backoff; after five consecutive failures to become ready the supervisor records the error in its state and gives up, and the next launch fails loudly with the daemon log path rather than silently bypassing headroom. Crash detection is driven by the proxy's own exit event (which is also what reaps it), and every pid liveness check in the coordination layer is zombie-aware, because a process that died unreaped still answers `kill(pid, 0)` as alive while holding no port; a deliberate stop escalates SIGTERM to SIGKILL on a bounded timeout, since the proxy does not reliably die on SIGTERM alone. When the allowlist or install source drifts (a provider file changed, the configured source changed), the daemon is restarted only once no session is live, so a running session is never cut off; when no session has been live for `idleShutdownMinutes` (15 by default), the supervisor stops the daemon and exits, freeing its memory.
 
 Coordination lives under `~/.claude-use/headroom/`: `state.json` (supervisor pid, daemon pid, port, version, allowlist hash, last error), an exclusive-create start lock so concurrent launches start at most one supervisor, and `sessions/<launcher-pid>.json` files as the session registry, pruned automatically when a launcher pid is no longer alive. `claude-use headroom status` reports all of it read-only, and `claude-use doctor` includes the daemon in its audit.
 

docs/testing.md

@@ -21,7 +21,7 @@ the resolver's cascade and materialisation logic is exactly the kind of thing th
 
 `identityManager.ts`, `configProfiles.ts`, `directoryRules.ts`, and `configure.ts` stay thin adapters over the resolver, so most of their correctness rides on the resolver's own test coverage above. The one exception is `listIdentities`, whose own tests cover a deliberate departure from the "throw a validation error and let it propagate" convention: an `identity.json` that is present but unreadable — malformed JSON, or valid JSON this version's `IdentitySchema` rejects — is reported as that one identity's own unreadable entry, so a single bad file never hides every *other* identity from `claude-use identity list` at the moment they most need to be visible. Only those two content-shaped failures are absorbed; a permission error still propagates. A wholly *absent* `identity.json` remains a silent skip rather than a problem, and both it and `doctor`'s own enumeration filter out directories whose name starts with `.`, since `IdentitySchema` requires an identity name to start with a letter or digit and a resync's own `.<identity>.scratch.<suffix>`/`.<identity>.previous.<suffix>` directories are therefore never identities to report on. `launcher.ts` carries three separately-testable responsibilities of its own that aren't covered by the resolver's purity, and need their own coverage: translating a resolved `Map<path, boolean>` into real filesystem side effects (creating/removing symlinks, materialising/collapsing directories, diffing against the farm's prior state, the per-identity lock and atomic-swap behaviour from [Directory rules](configuration-model.md#directory-rules)) against a fake/in-memory filesystem; invoking the real `claude` binary via an injected `spawn` function (argv/env construction, exit-code propagation), never a real subprocess in a unit test; and the ambient-credential guard — given a fake `process.env`, refusing to proceed when any of the six named variables is set and the active identity's `allowAmbientCredential` is unset/false, proceeding when it's true, and proceeding when `CLAUDE_USE_ALLOW_AMBIENT_CREDENTIAL=1` is set for that one call regardless of the identity's own setting. Provider selection gets the same launcher-level coverage against a fake providers map: the child env gaining the provider's base URL, token, static `env` entries, and a cleared `ANTHROPIC_API_KEY`; an unknown provider refusing with the known names listed (exit 1); an unset token environment variable refusing with exit 64; a `--provider` flag beating a cascade-pinned provider; and the guard standing aside for a provider launch (claude-use supplies the child's credential itself) while still refusing an ambient `ANTHROPIC_AUTH_TOKEN` with no provider selected. `providers.test.ts` covers the manager functions (`addProvider`/`listProviders`/`removeProvider`/`loadProvider`) over real temp directories per `identityManager.test.ts`'s convention, plus `resolveProvider` itself as a pure decision over an injected `FsPort`.
 
-Headroom routing keeps the same shape: no real process, port, clock, or HTTP call ever happens in a test. `headroom/ensure.test.ts` drives the launcher-side bring-up against a fake filesystem and scripted supervisor behaviour (first launch spawns the supervisor and waits for its ready state; a healthy daemon is reused without spawning; a dead supervisor pid is replaced; a live start lock held by another launcher is waited on while a dead holder's lock is taken over; a fatal `lastError` refuses immediately; a daemon that never appears times out naming the log path). `headroom/supervisor.test.ts` runs the whole supervisor loop over injected ports with a clock that only advances when the supervisor sleeps: install invoked when the binary is missing or the source changed and skipped when the installed version satisfies the source, the spawn allowlist containing every provider plus `api.anthropic.com`, crash restart while a session is live, the backoff schedule and retry budget ending in a recorded `lastError`, drift restarts deferred until the session registry empties, idle shutdown timing (and its reset while sessions are live), and dead-pid session pruning. `headroom/headers.test.ts` covers the `ANTHROPIC_CUSTOM_HEADERS` merge, `headroom/state.test.ts` the state/registry/allowlist primitives, and `doctor.test.ts` gains the headroom audit cases (never run, malformed state, healthy pids, dead supervisor, a `lastError` warning alongside a running replacement). The launcher's own headroom tests inject a fake `HeadroomPort` to cover the env wiring with and without a provider, the loud refusal when headroom resolved on with no port wired, and that the port is never touched when headroom resolved off.
+Headroom routing keeps the same shape: no real process, port, clock, or HTTP call ever happens in a test. `headroom/ensure.test.ts` drives the launcher-side bring-up against a fake filesystem and scripted supervisor behaviour (first launch spawns the supervisor and waits for its ready state; a healthy daemon is reused without spawning; a dead supervisor pid is replaced, including one that is an unreaped zombie signal 0 still reports alive; a live start lock held by another launcher is waited on while a dead holder's lock is taken over; a fatal `lastError` refuses immediately; a daemon that never appears times out naming the log path). `headroom/supervisor.test.ts` runs the whole supervisor loop over injected ports with a clock that only advances when the supervisor sleeps and a process model that keeps signal-0 existence, zombification, and the ChildProcess exit event distinct: install invoked when the binary is missing or the source changed and skipped when the installed version satisfies the source, the spawn allowlist containing every provider plus `api.anthropic.com`, crash restart driven by the exit event while a session is live, a zombie daemon treated as dead and restarted even though existence alone would still report it, the daemon fields cleared from state the moment a crash is detected, an orphan daemon left by a dead predecessor stopped before the successor starts its own, the backoff schedule and retry budget ending in a recorded `lastError`, drift restarts deferred until the session registry empties, idle shutdown timing (and its reset while sessions are live), and session pruning of dead and zombie launcher pids. `stopSupervisedProcess` gets its own table: never escalating when the process exits on SIGTERM, escalating to SIGKILL once the SIGTERM grace expires, reporting `still-running` when even SIGKILL does not clear it, and sending nothing further when the target is already gone. `headroom/headers.test.ts` covers the `ANTHROPIC_CUSTOM_HEADERS` merge, `headroom/state.test.ts` the state/registry/allowlist primitives, and `doctor.test.ts` gains the headroom audit cases (never run, malformed state, healthy pids, dead supervisor, a `lastError` warning alongside a running replacement). The launcher's own headroom tests inject a fake `HeadroomPort` to cover the env wiring with and without a provider, the loud refusal when headroom resolved on with no port wired, and that the port is never touched when headroom resolved off.
 
 `check.ts`'s three always-on diagnostics get their own tests too, independent of path/cascade resolution: the ambient-credential check against a fake `process.env` (same fixture as `launcher.ts`'s guard, since they share the same detection logic); the settings-secrets advisory against a fake settings.json with populated `env`/`hooks` fields, confirming it reports counts and key names only, never values; and — since Keychain access is real OS state, not something to fake — a manual/integration-only note that the Keychain-name lookup is exercised against a real `security` call in CI on macOS runners, not unit-tested with a mock.
 

src/cli.ts

@@ -26,7 +26,7 @@ import {
   realFarmFs,
   realFsPort,
   realHeadroomPort,
-  realIsProcessAlive,
+  realIsProcessRunning,
   realLogPort,
   realOwnExecutablePath,
   realProcPort,
@@ -103,7 +103,8 @@ function buildFarmRuntime(paths: LayoutPaths): {
         }).input,
       now: () => Date.now(),
       uniqueSuffix: `${String(process.pid)}.${randomUUID()}`,
-      lock: { pid: process.pid, isProcessAlive: realIsProcessAlive, sleep: realSleepSync },
+      // Zombie-aware on purpose: a previous launcher that crashed out of a resync without releasing the lock may sit unreaped, still answering signal 0 as alive, and must read as a dead holder so this launch takes the lock over instead of timing out.
+      lock: { pid: process.pid, isRunning: realIsProcessRunning, sleep: realSleepSync },
     },
     ...(selections.identity === undefined ? {} : { directoryIdentity: selections.identity }),
     ...(selections.configProfile === undefined ? {} : { directoryConfigProfile: selections.configProfile }),

src/doctor.test.ts

@@ -31,7 +31,7 @@ function baseParams(overrides: Partial<RunDoctorParams> = {}): RunDoctorParams {
     claudeShim: { state: undefined, targetExists: false },
     pathResolution: { ownExecutablePath: "/home/u/.local/bin/claude-use", claudeUse: { status: "ok" } },
     platform: "linux",
-    headroom: { state: { path: "/claude-use/headroom/state.json", raw: undefined }, isProcessAlive: () => false },
+    headroom: { state: { path: "/claude-use/headroom/state.json", raw: undefined }, isRunning: () => false },
     ...overrides,
   };
 }
@@ -69,7 +69,7 @@ describe("runDoctor: headroom", () => {
   });
 
   it("fails on a malformed state.json instead of guessing", () => {
-    const report = runDoctor(baseParams({ headroom: { state: { path: "/claude-use/headroom/state.json", raw: "{bad" }, isProcessAlive: () => false } }));
+    const report = runDoctor(baseParams({ headroom: { state: { path: "/claude-use/headroom/state.json", raw: "{bad" }, isRunning: () => false } }));
     expect(findingsFor(report, "headroom").some((finding) => finding.severity === "fail")).toBe(true);
   });
 
@@ -79,7 +79,7 @@ describe("runDoctor: headroom", () => {
       baseParams({
         headroom: {
           state: { path: "/claude-use/headroom/state.json", raw: JSON.stringify({ supervisorPid: ALIVE_SUPERVISOR_PID, headroomPid: ALIVE_DAEMON_PID, port: HEADROOM_PORT, version: "headroom 0.39.1" }) },
-          isProcessAlive: (pid: number) => alive.has(pid),
+          isRunning: (pid: number) => alive.has(pid),
         },
       }),
     );
@@ -93,7 +93,7 @@ describe("runDoctor: headroom", () => {
       baseParams({
         headroom: {
           state: { path: "/claude-use/headroom/state.json", raw: JSON.stringify({ supervisorPid: ALIVE_SUPERVISOR_PID, headroomPid: ALIVE_DAEMON_PID, port: HEADROOM_PORT }) },
-          isProcessAlive: () => false,
+          isRunning: () => false,
         },
       }),
     );
@@ -111,7 +111,7 @@ describe("runDoctor: headroom", () => {
             path: "/claude-use/headroom/state.json",
             raw: JSON.stringify({ supervisorPid: REPLACEMENT_SUPERVISOR_PID, headroomPid: REPLACEMENT_DAEMON_PID, port: HEADROOM_PORT, lastError: "previous crash" }),
           },
-          isProcessAlive: (pid: number) => alive.has(pid),
+          isRunning: (pid: number) => alive.has(pid),
         },
       }),
     );