Skip to content

fix(cross-model): attest the omp serving family from the session's recorded model - #2043

Merged
tmchow merged 7 commits into
EveryInc:mainfrom
anthogiuliano:p8o-omp-serving-family
Nov 9, 2026
Merged

tmchow merged 7 commits into
EveryInc:mainfrom
anthogiuliano:p8o-omp-serving-family

Conversation

@anthogiuliano

@anthogiuliano anthogiuliano commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2042

Summary

Follow-up to #1825 / #1867. Under oh-my-pi (omp) the host-attestation snippet leaves the serving family unknown, which is correct because omp sets CLAUDECODE=1 whatever model serves it. The cross-model worker then refuses an unknown family before any egress, so the cross-model pass is skipped under omp unless the orchestrator sets the family by hand. In one real session a six-lens review shipped without its independent lens.

omp records the serving model: PI_SESSION_FILE points at a session JSONL whose model_change entries carry provider/id. The ce-code-review and ce-doc-review workers now read it when the harness is omp and the family arrives as unknown.

  • First-party prefixes only: anthropic/claude-* -> claude, openai/gpt-*|codex-*|o<digit>* -> codex, xai/grok-* -> grok, google/gemini-* -> gemini. Everything else stays unknown, so the pass is skipped instead of mis-attested (gateway-served Claude, local models, missing or empty session file, no model_change).
  • The latest default-role entry names the model. Entries for other roles (smol, subagent:*) are ignored. If the latest relevant entry is a fallback or temporary switch, the serving model can't be known and the family stays unknown.
  • A jq parse failure (for example a torn trailing line) also fails closed to unknown.
  • The fix is worker-side because the shared attestation snippet is byte-identical across three skills under a parity test. The orchestrator prose in the ce-code-review and ce-doc-review references now says to pass unknown and still start the job under omp, and to report a worker skip of host serving family unattested plainly in Coverage instead of as a routine not-run.

Not done: ce-pov. Its orchestrator excludes automatic oracle candidates while the family is unknown, before the worker runs, so worker-side resolution cannot reach it. An attempted edit was reverted in this PR's history. It needs the family resolved before candidate discovery without breaking the snippet parity test.

Validation

Base: 8e5d1c87.

  • bun run test --changed-since=8e5d1c87, bun run test:skill-guards, bun run release:validate, bun run plugin:validate and bun run typecheck pass, apart from tests/omp-native-install.test.ts. That test fails on omp 18.18.x because omp install --dry-run --json no longer prints name and manifest; the test does not depend on this change.
  • New route tests in tests/skills/ce-code-review-cross-model-routes.test.ts and tests/skills/ce-doc-review-cross-model-routes.test.ts cover first-party mappings, ignored non-default roles, later default entries, and the fail-closed cases.
  • Fresh-agent evals of the prose (case 6a, stop-before-dispatch, omp markers plus a session file recording anthropic/claude-sonnet-5-5): ce-code-review and ce-doc-review each passed on Claude, Codex and OpenCode. Every cell declared harness omp, family unknown, target codex. Grok did not run: the eval driver's grok flags (--prompt-file, --verbatim, --cwd, --always-approve) do not exist in the grok CLI I have (1.1.7), and it has no credentials configured here. The pre-change arm was run once for ce-doc-review only: Codex skipped, and Claude declared family claude from its own model id.
  • Real runs with omp's markers and a session file recording anthropic/claude-sonnet-5-5 (real npm/acpx/Codex execution, synthetic fixture diffs only): the worker run directly logged host claude excluded, ran the codex route on gpt-6.1-terra and published independence_verified=true; the same worker launched through peer-job-runner.py start/wait/result reached done with the same receipt; and a full ce-code-review run (mode:agent, staged payment-handling fixture, depth full) started the peer with harness omp and family unknown without the orchestrator setting the family, and folded adversarial-codex into the review.
  • I replayed the final rule over 4,061 of my own omp session files, comparing the family it attests before each assistant turn with the family of the provider that served that turn. It attested on 78,703 of 94,689 turns served by a first-party provider and was wrong on none; in the last 30 days, 10,752 of 10,817, wrong on none. The replay's final state matched the real jq filter on a random sample of 60 files. Most turns come from entries that record no runtime.version, so this does not establish the rule for every omp release.

Open: verification on omp versions beyond 18.18.x, and ce-pov.

Security Disclosure

Shell is changed in the two cross-model worker scripts (cross-model-adversarial-review.sh, cross-model-doc-review.sh): they now run jq over the file named by PI_SESSION_FILE, read-only, and only when the harness is omp and the family is unknown. The path comes from the host's environment and is passed as an argument, not interpolated into a command. The extracted model id is only pattern-matched against a fixed list of first-party prefixes to pick a family name; it is never executed; any parse error, unreadable file or unrecognized id leaves the family unknown, which skips the pass before any egress. Residual risk: a session file that falsely records a first-party model would attest that family. The file is written by the host runtime, which is the same trust boundary as the existing CLAUDECODE/OMPCODE markers.

Agent Disclosure

  • Model: Claude Code · claude-sonnet-5-5 did the rebase, verification and publishing. The implementation commits were written in earlier sessions whose model is not visible from this one.

🤖 Generated with Claude Code

Maintainer follow-up (0a1b47c)

  • Nested-harness mis-attestation (Codex review P1), fixed. A Codex, Claude Code, or Grok session started inside omp inherits OMPCODE and PI_SESSION_FILE, so the worker read the parent's model and attested the wrong family. Same-family agreement could then be promoted as independent. The workers now read the session file only when no other harness's marker is present (CLAUDE_CODE_ENTRYPOINT/CLAUDE_CODE_SESSION_ID, CODEX_*, GROK_*, CURSOR_*, OPENCODE_TERMINAL), and otherwise skip as unattested. omp itself sets none of these. The new route tests failed on the previous head and pass now.
  • Subscription providers mapped. omp records openai-codex/<model> for ChatGPT logins and xai-oauth/<model> for xAI logins, and both previously stayed unknown. omp pins them to chatgpt.com and x.ai/grok.com, so they now map to codex and grok.
  • Two Coverage lines removed. In ce-code-review, the "never started" bullet described a job that did start, and the recovery branch already owns that case. In ce-doc-review, the existing clean-skip condition ("host un-attestable") already decides it.

Validation:

  • Route tests (253), skill guards, typecheck, release:validate and plugin:validate pass.
  • bun run test --changed-since=8e5d1c87: 5001 pass, 1 fail. The failure is tests/omp-native-install.test.ts: omp 18.18.5's dry-run JSON dropped name and manifest. It predates this PR and is skipped in CI.
  • Live runs on omp 18.18.5:
    • A clean session on anthropic/claude-sonnet-5-5 attested claude and ran the real Codex peer (gpt-6.1-terra) with independence_verified: true.
    • openai-codex/gpt-6.1-terra and xai-oauth/grok-4.20-0309-reasoning sessions attested codex and grok (dry run).
    • omp launched from inside Claude Code failed closed.
  • Prose eval not rerun: the prose edits only delete lines the existing conditions already decide, plus one parenthetical.

Agent disclosure for this follow-up: Claude Code · claude-opus-5-5.

anthogiuliano and others added 6 commits November 8, 2026 21:36
Real omp sessions record role=temporary model_change entries that the worker
jq skipped, so a temporary switch to another family left the host family at
the earlier default and could mark a peer independent against the wrong host.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T20:13:29.956568Z 0a1b47c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ebce11de3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A Codex, Claude Code, or Grok session started inside omp inherits OMPCODE
and PI_SESSION_FILE, so the workers read the parent's recorded model and
attested the wrong family, letting same-family agreement be promoted as
independent. The workers now read the session file only when no other
harness's marker is present, and fail closed otherwise.

Also map omp's first-party subscription providers (openai-codex -> codex,
xai-oauth -> grok), which omp records for ChatGPT and xAI logins, and drop
two Coverage lines that existing conditions already decide.
@tmchow
tmchow merged commit d293bf4 into EveryInc:main Nov 9, 2026
9 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recover the serving family under omp from the session file, so the cross-model pass runs again (follow-up to #1825)

2 participants