Skip to content

fix: mail resumes a Codex thread Codex unloaded while idle, never one the person archived - #218

Open
reason-machines[bot] wants to merge 8 commits into
mainfrom
reason/network-resume-same-code-08bc7322
Open

reason-machines[bot] wants to merge 8 commits into
mainfrom
reason/network-resume-same-code-08bc7322

Conversation

@reason-machines

@reason-machines reason-machines Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Feedback (sline, #feedback): a durable Codex Desktop thread went idle for 3 h; Codex's app-server logged inactive_thread_unsubscribed, shut the thread down, and the Reason Network daemon logged session … ended at the same second. The session stayed listed offline with 3 unread messages until the person woke the same thread UUID by hand (Codex's own send_message_to_thread). The app and computer were running the whole time.

Root cause. Codex runs SessionEnd whenever a thread shuts down, including when the app-server unloads an idle one (openai/codex: hooks/src/events/session_end.rs always sends reason: "other", from core/src/session/handlers.rs shutdown). The daemon took the thread offline, which is right, since its route and name are kept. But its offline wake path ran codex queue --thread <id> --message …. That only adds the message to the thread's queue, and the queue is consumed only once something loads the thread again. Nothing did, so the mail waited for the person.

Fix.

  • codex queue now runs with --resume (src/daemon/mail.rs). This is Codex's supported way to do it (openai/codex cli/src/queue_cmd.rs, tui/src/session_queue_commands.rs::resume_if_unloaded). After queueing, Codex reads the thread and, only if it is NotLoaded, sends thread/resume for that same thread id. A thread that is already loaded is left alone. It never creates a new thread, chat or serve. This covers the offline path and the live idle path, where a thread can be unloaded while its process lives on.
  • A retired thread is never revived. A thread the person archived in Codex has its rollout moved to $CODEX_HOME/archived_sessions/rollout-<ts>-<id>.jsonl[.zst]. New codex::archived matches that exact id (not a suffix of one). For an archived thread, mail is queued without --resume, so it waits for the person to open the thread, as before. A thread Codex only unloaded keeps its rollout in sessions/ and is resumed. Folders that are off and failed codex queue handoffs keep their existing guards.
  • Older Codex keeps working. A codex queue that predates --resume refuses the flag (clap: unexpected argument '--resume' found) before it queues anything. The daemon then runs it again without the flag. Any other failure is still a failure.
  • The daemon's log for a Codex SessionEnd now says went offline; mail resumes it instead of ended. The sender's note says Codex resumes the same thread, which starts a turn with it, or it was archived… for an archived thread.
  • DESIGN.md's Codex delivery row is updated to match.

Builds on #217 (no-transcript Codex threads are never sessions) and merges cleanly with its head.

Checks (Reason cloud, Linux)

  • cargo test: 345 + 5 + 2 passed. New tests:
    • daemon::mail::tests::mail_resumes_a_codex_thread_codex_unloaded_but_never_one_the_person_archived reproduces the report: start, MCP register, SessionEnd, MCP exit. The unloaded thread's mail runs queue --thread t1 --resume, and the archived thread's runs queue --thread t2 with no --resume. With the --resume push disabled, it fails (panics at the t1 --resume assertion). With the fix, it passes.
    • a_codex_without_queue_resume_still_takes_an_offline_threads_mail: the fake old Codex refuses --resume, the daemon retries without it, and the mail is read and not marked failed.
    • only_a_codex_that_refuses_the_resume_flag_is_asked_again_without_it and codex::tests::only_a_thread_whose_rollout_codex_moved_to_archived_sessions_is_archived, which covers plain and .zst rollouts, exact-id matching and a missing home.
  • python3 tests/e2e/codex_busy.py: 59/59, including new checks that an offline thread is queued --resume by its own id and that an archived thread is queued without it.
  • python3 tests/e2e/hooks.py: 82/82. python3 tests/e2e/two_machines.py: 150/150 (its fake-queue assertion now expects --resume).
  • cargo fmt --check clean. cargo clippy --all-targets -- -D warnings clean.

Not verified

  • Not run against a live Codex Desktop on macOS. The behaviour comes from Codex source at ad3d352 (2026-11-10). resume_if_unloaded needs codex queue to reach a non-embedded app-server (the local daemon socket or --remote). If Codex Desktop's app-server isn't reachable that way, the message is still queued and Codex prints a warning but exits 0, so it waits for the thread to load, as it did before this change.
  • The daemon still needs a codex CLI on its PATH, or in the usual install folders, to hand off offline mail. Without one, mail waits for hooks, as before.
  • Offline wake still applies only to threads seen within the last day (CODEX_RESUME_MS), which is unchanged.

Shipping needs a reason-network release and the monorepo CLI's ENGINE_VERSION/digest bump.

— Feedback resolver

Review fixes

Independent review (GPT-6.1 Sol) raised three findings; all are fixed in this push.

  • P1: codex queue --resume exits 0 when it queued the message but could not resume the thread. Codex prints a could not resume thread warning on stderr and still exits 0. The daemon now detects that warning, logs it, does not requeue the mail, and remembers the thread as not resumed until a later resume succeeds or the daemon restarts. Senders are told the message is queued and Codex was asked to load that thread, not that it was resumed. New test: mail_codex_queued_but_could_not_resume_is_not_claimed_resumed.
  • P2: the archive lookup ran under the daemon's Inner lock. It now runs on a blocking worker, off the lock, capped at 5 s. A timeout counts as archived, so an uncertain thread is never revived.
  • P3: archive detection missed replacement rollout names. It now matches rollout-<ts>-<thread>_<rollout>.jsonl[.zst] as well as rollout-<ts>-<thread>.jsonl[.zst], by thread UUID only.
  • tests/e2e/codex_busy.py and DESIGN.md were updated to use the new sender wording.

Checks after the review fixes (Reason cloud, Linux)

  • cargo fmt --check: clean.
  • cargo clippy --all-targets --locked -- -D warnings: clean.
  • cargo test --locked: 346 + 5 + 2 passed.
  • python3 tests/e2e/codex_busy.py: 59/59 on three consecutive runs. The first run on a CPU-starved machine reported 58/59, and its failing check was not captured. It did not happen again in the reruns.
  • python3 tests/e2e/two_machines.py: 150/150.
  • python3 tests/e2e/hooks.py: 82/82.

Gaps

  • No test measures whether a slow archive lookup leaves other sessions unblocked. The lookup was moved off the lock, but this was not timed.
  • No live Codex Desktop run.
  • If Codex cannot reach the thread's app-server, the message stays queued, and senders are told it is queued and was not resumed.
  • One unexplained 58/59 codex_busy.py run, described above.

Second review round (GPT-6.1 Sol, re-review of 25425da)

  • P1 (still open at 25425da): mail Codex queued but couldn't resume sat in Codex. codex queue --resume can only load a thread through a shared app-server, the one on $CODEX_HOME/app-server-control/app-server-control.sock (openai/codex tui/src/session_queue_commands.rs::resume_if_unloaded, app-server-transport::app_server_control_socket_path). Without one, Codex queues the mail anyway, and the thread can't be loaded later without queueing it again. Fix: offline wake mail goes to Codex only while that socket answers (codex::app_server_up). Otherwise it stays unread here, is not in flight, and senders hear it is kept and tried again. It is offered again with backoff, 30 s doubling to 30 min (keep_for_codex / retry_codex, Inner::codex_retry), and goes once into that same thread by its UUID with --resume when an app-server answers. Mail sent meanwhile carries the kept mail with it, so nothing is queued in Codex twice. The retry stops when the thread starts again (SessionStart clears it, and its hooks show the mail), when the mail is read, or when codex queue takes it. If --resume still warns (the app-server went away between the probe and the call), the earlier warning path applies.
  • P2: an archive lookup that timed out counted as archived, and the wake was abandoned. archived_target now returns Option<bool>. A timeout is unknown: for offline mail it's kept and retried like P1, never handed over without --resume. For a live, loaded thread it is still queued without --resume, which never revives an uncertain thread.
  • P2: conflict with fix: Codex's ephemeral threads are never sessions on the network #217 (codex_unresumed vs no_transcript in Inner). Not reconciled here: fix: Codex's ephemeral threads are never sessions on the network #217 hasn't merged and main doesn't conflict. Once fix: Codex's ephemeral threads are never sessions on the network #217 lands, main will be merged into this branch keeping both fields and both initializers, and the suites rerun.
  • DESIGN.md's Codex row now describes the kept-and-retried path.

Checks at this head (Reason cloud, Linux)

  • cargo fmt --check: clean. cargo clippy --all-targets --locked -- -D warnings: clean.
  • cargo test --locked: 349 + 5 + 2 passed. New tests:
    • wake_mail_waits_here_until_a_codex_app_server_can_load_the_thread_then_goes_once: with no app-server, nothing is run, the mail is unread, the next sender hears it is kept, and retries keep it. Once the app-server answers, one queue --thread t1 --resume carries all three messages, they're read, the retry state is gone, and no more commands run.
    • an_archive_lookup_that_takes_too_long_is_not_known: a slow lookup gives None; archived and not-archived lookups give Some(true) and Some(false).
    • a_codex_app_server_is_up_only_while_its_control_socket_answers: no socket is down, a listening socket is up, and a stale socket file is down.
  • python3 tests/e2e/codex_busy.py: 61/61 (two new checks: with no app-server the mail is kept unread and the sender is told; once a socket listens, both messages are queued once with --resume).
  • python3 tests/e2e/two_machines.py: 150/150. python3 tests/e2e/hooks.py: 82/82.

Gaps

  • No live Codex Desktop run. The socket path and the resume behaviour come from openai/codex source on main, and the tests use a fake codex and a plain listening socket, not a real app-server.
  • On Windows, Codex's control socket is AF_UNIX, which tokio can't open, so an existing socket file counts as up. A stale file there would hand the mail to Codex, and if resume then fails, the warning path applies. The Windows path isn't exercised beyond CI's build and unit tests.
  • Kept mail is retried only while the daemon runs. After a restart it waits for the thread's hooks or the next wake mail to it.
  • No test measures whether a slow archive lookup leaves other sessions unblocked.

Third review round (GPT-6.1 Sol, re-review of cdf1629)

All five findings held, and head 178d108 fixes them.

  1. P1: Codex took the mail but couldn't load the thread.
    • Before: when codex queue --resume exited 0 with a "could not resume thread" warning, the daemon marked the mail read and never tried again.
    • Now: the daemon asks Codex to load that same thread again, and does nothing else, never queueing the mail a second time. It retries with backoff (30 s, doubling to 30 min) until the thread loads, starts again, or the person archives it.
    • How: through codex app-server proxy, which relays stdio to the control socket on every platform. Over it the daemon sends JSON-RPC over WebSocket: initialize and initialized, then thread/read, then thread/resume (excludeTurns) only for a thread that isn't loaded. That is the same sequence codex queue --resume uses (openai/codex tui/src/session_queue_commands.rs).
    • It also covers a stale socket file on Windows: the existence check lets the mail through, and the warning then starts these retries.
  2. P2: long CODEX_HOME. Codex advertises its control socket as a symlink to a short protected socket. The probe now connects to the link's target, and the proxy is given that target with --sock.
  3. P2: kept mail after a daemon restart.
    • Every offline handoff now carries the thread's unread wake mail along, oldest first, whether or not the in-memory retry map has an entry for the thread.
    • When the daemon starts, it offers such mail to Codex again by itself.
  4. P2: old Codex never reached its fallback.
  5. P2: empty CODEX_HOME. An empty value now means the default home, as Codex treats it. This holds both for the environment variable and for a home recorded for the thread.

DESIGN.md is updated to match.

Checks at this head (Reason cloud, Linux)

All passed:

  • cargo fmt --check and cargo clippy --all-targets --locked -- -D warnings: both clean.
  • cargo test --locked: 353 + 5 + 2 passed. New or changed tests:
    • resume_loads_an_unloaded_thread_over_the_control_socket_and_leaves_a_loaded_one: run against a fake WebSocket app-server that also sends a ping, a notification, a frame longer than 125 bytes and a fragmented reply. An unloaded thread gets exactly the sequence above, a loaded one is left alone, and a refusal comes back as an error.
    • an_app_server_behind_a_long_codex_home_answers_through_its_socket_link: connecting straight to the advertised path (longer than 108 bytes) fails, and the probe connects through the link.
    • an_empty_codex_home_is_the_default_one.
    • mail_codex_queued_but_could_not_resume_is_resumed_later_without_queueing_it_again: while resuming fails, the daemon keeps asking; once it succeeds it stops. Each message is queued exactly once, and a SessionStart also stops the asking.
    • wake_mail_kept_before_a_daemon_restart_still_reaches_the_codex_thread: after a restart, the next mail carries the kept message ahead of it. With no new mail, the retry at startup delivers it by itself.
    • a_codex_without_queue_resume_still_takes_an_offline_threads_mail: with old help text and no socket, the mail is queued once, without --resume.
  • python3 tests/e2e/codex_busy.py: 61/61.
  • python3 tests/e2e/two_machines.py: 150/150.
  • python3 tests/e2e/hooks.py: 82/82.

The e2e fake codex now answers queue --help with --resume. The first e2e run failed (57/61 and 149/150) because the fakes didn't answer queue --help yet. With that fixed, both pass.

Gaps

  • No live run against Codex Desktop or a real app-server. The WebSocket client is tested against a fake server built from Codex's source.
  • Windows still only checks that the socket file exists. A stale file now leads to resume retries rather than stuck mail, but CI is the only thing that runs the Windows path.
  • The resume-only call closes its connection after thread/resume, as codex queue --resume does. Whether Codex then starts the queued turn isn't verified live.
  • No test measures whether a slow archive lookup leaves other sessions unblocked.
  • The conflict with fix: Codex's ephemeral threads are never sessions on the network #217 in src/daemon/mod.rs (both PRs add a field and an initializer to Inner) is still there. It gets resolved once fix: Codex's ephemeral threads are never sessions on the network #217 lands.

Fourth review round (head 63f61f0)

The reviewer (GPT-6.1 Sol, Session 3bc06e82) found two more problems at 6af1a30. Both were real, and both are fixed.

  1. P1: a daemon restart forgot a thread that Codex had taken mail for but couldn't load. The mail is marked read once Codex takes it, so the record that the thread still needs loading lived only in memory (codex_unresumed). After a restart, nothing asked Codex to load it again. Fix: that record is now also kept in the store (key codex-unresumed:<session>), written when Codex takes the mail and removed once the thread loads, starts again, is archived or is no longer known. At startup the daemon reads it back and asks Codex to load the thread again with the same backoff. It only resumes the thread and never queues the mail again.
  2. P2: the 1-day cutoff dropped resumes Codex had already accepted. resume_codex checked the same "seen within the last day" rule that applies when new mail is sent. A thread that took mail just before the cutoff was dropped at its first retry. Fix: the cutoff now applies only when mail is sent. Finishing a handoff Codex accepted uses the thread and its home without the age check.

DESIGN.md is updated to match.

Checks (Reason cloud, Linux, same tree as 63f61f0)

  • cargo fmt --check: clean.
  • cargo clippy --all-targets --locked -- -D warnings: clean.
  • cargo test --locked: 354 + 5 + 2 passed.
    • New test: a_thread_codex_took_mail_for_is_loaded_again_after_a_restart_and_past_the_cutoff. Codex takes mail without loading the thread, then the thread ages past the 1-day cutoff and the daemon is restarted (memory cleared). Codex is asked to load the thread again both times. Once it can, the record is cleared, and the mail is queued exactly once.
    • The test fails with either old behavior put back. With the cutoff check: "no resume once past the cutoff". Without reading the record at startup: pending() fails after the restart.
  • python3 tests/e2e/codex_busy.py: 61/61.
  • python3 tests/e2e/two_machines.py: 150/150.
  • python3 tests/e2e/hooks.py: 82/82.

The earlier Ubuntu CI failure at 178d108 was in the restart test, which assumed the fake codex's log files come back in a fixed order. 6af1a30 fixed the test; the daemon code didn't change for it.

Gaps (in addition to the ones above)

  • The restart in the new test is simulated by clearing the daemon's memory and running its startup recovery. It doesn't stop and start a real daemon process.
  • A thread that keeps failing to load is asked again every 30 min at most until it starts again, is archived or is forgotten. There's no other limit.

reason-machines Bot and others added 2 commits November 11, 2026 00:00
… the person archived

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
… up archives off the lock; match replacement rollouts

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
reason-machines Bot and others added 5 commits November 11, 2026 02:09
…read; an archive lookup that times out is unknown, not archived

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
…; connect through Codex's socket link; keep old-Codex fallback; retry kept mail after a restart

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
…start test

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
Co-authored-by: qa-runner2-star <319404260+qa-runner2-star@users.noreply.github.com>
…ould not load; finish that handoff past the 1-day wake cutoff

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
Co-authored-by: qa-runner2-star <319404260+qa-runner2-star@users.noreply.github.com>
…ing again

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
Co-authored-by: qa-runner2-star <319404260+qa-runner2-star@users.noreply.github.com>
@reason-machines

Copy link
Copy Markdown
Contributor Author

Fifth review round (reviewer GPT-6.1 Sol, at 63f61f0), fixed in b18380f

P2: ordinary session expiry dropped an accepted pending resume. Once the 6 h startup grace ended, prune forgot offline sessions older than a day, including a Codex thread still in codex_unresumed. The next retry found no session row and cleared the marker, and the mail Codex held (already read here) had no automatic recovery.

Change: prune keeps the session row of every thread in codex_unresumed, and holds keeps that thread's name from other sessions while it waits. Archiving, the thread starting again, or a successful load still ends the wait.

Checks (Reason cloud, Linux):

  • a_thread_codex_took_mail_for_is_loaded_again_after_a_restart_and_past_the_cutoff now ends the startup grace and calls prune after aging the session. It fails without the fix ("pruned a thread still to be loaded") and passes with it.
  • cargo fmt --check: pass. cargo clippy --all-targets --locked -- -D warnings: pass.
  • cargo test --locked: 354 + 5 + 2 passed.
  • python3 tests/e2e/codex_busy.py 61/61, tests/e2e/two_machines.py 150/150, tests/e2e/hooks.py 82/82 (run with the prune fix, before the name-keeping change).

Gap: no test covers the name-keeping change in holds. The regression test's thread is only ever seen through its hooks, so its name is never kept either way.

— Feedback resolver

…on, and even if it was never reached

Co-authored-by: svemyh <40596752+svemyh@users.noreply.github.com>
Co-authored-by: qa-runner2-star <319404260+qa-runner2-star@users.noreply.github.com>
@reason-machines

Copy link
Copy Markdown
Contributor Author

Fifth review round (head 48e3662) — Feedback resolver

Fixes the two P2s the independent review (GPT-6.1 Sol) found at b18380f:

  1. Pending resume didn't hold the name of a never-reached thread. holds required r.kept() (attached) even for a thread Codex holds mail for. Now a thread that still owes a resume (Inner::resume_owed) keeps its name on its own, so a competing registration can't take over its row.
  2. Prune could delete a thread mid-handoff. Protection only started once codex queue returned. A new codex_handing count now covers the thread from dispatch until handed/keep_for_codex has recorded the outcome (and the codex_unresumed marker, when Codex couldn't load it), so the routine prune keeps its row and unread mail throughout.

Checks (Reason cloud, Linux):

  • New test a_thread_being_handed_mail_keeps_its_session_and_name_until_codex_answers: holds the fake codex queue open, ages the unattached thread past a day and prunes. Failed with the old holds/prune logic, passes with the fix.
  • cargo fmt --check, cargo clippy --all-targets --locked -- -D warnings: clean.
  • cargo test --locked: 355 + 5 + 2 passed.
  • tests/e2e/codex_busy.py 61/61, two_machines.py 150/150, hooks.py 82/82.

Not verified: live Codex Desktop, native Windows runtime. The known Inner field conflict with #217 gets resolved (keeping both fields) once #217 lands.

@reason-machines
reason-machines Bot requested a review from svemyh November 11, 2026 04:57
@reason-machines

Copy link
Copy Markdown
Contributor Author

Ready for approval. Head 48e3662: all required checks (macOS, Ubuntu, Windows tests, CodeQL) pass. Independent re-review (GPT-6.1 Sol) is clean on this exact head: both b18380f P2s resolved, codex_handing counter audited with no leak/underflow, cargo test --locked 362/362, codex_busy.py 61/61. GitHub requires one maintainer approval to merge; @svemyh please approve. — Feedback resolver

@reason-machines

Copy link
Copy Markdown
Contributor Author

@svemyh friendly nudge: #218 needs your approving review to merge. Head 48e3662 is unchanged since the clean independent re-review, and all required checks (macOS, Ubuntu, Windows) pass. Once approved I'll merge and hand the SHA to the v0.12.18 release.

— Feedback resolver

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants