Repository navigation
Codex lock-in: lean home, engine lock, eval fixes; engine A/B - #7
edagher92-coder wants to merge 4 commits into
Conversation
…ash attribution - agents/codex-qwen.cmd is now the lean-home launcher with a preflight engine lock (agents/locks/qwen-lock.ps1), git identity label and an exec-only writable .git root; old launcher kept as codex-qwen-full.cmd; claude-qwen takes the same lock. New agents/codex-lean-home/ (config, compact instructions, preflight). - evalsuite 19 tasks: old 13/19 -> lean 15/19 (-49% wall on shared passes) -> 18/19 acceptance in 711 s (old 2,140 s); money rule then 6/6. Raw rows in data/codex-eval-20261018/. docs/CODEX-QWEN.md. - bench/ab-20261018: f16 32K baseline mean 42.04 tok/s, pp 471/520; q8_0 main KV mean 35.16 (-16%), rejected; GGML_VK_PERF_LOGGER attribution: 125.2 ms GPU vs 128.3 ms wall per DFlash round (host gap ~3 ms). - TUNING-LOG rows 28-33, CHANGELOG entry (SYCL fork audit parked, B60 tuning: nothing to change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4G2wCLwkaZUjYJd6T3uZV
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 878e465a67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| endlocal | ||
| set "WIN=" | ||
| set "LOCKPID=0" | ||
| for /f "usebackq tokens=1,2" %%a in (`powershell -NoProfile -ExecutionPolicy Bypass -File C:\AI\codex-local\preflight.ps1`) do ( |
There was a problem hiding this comment.
Invoke preflight from its documented deployment directory
On a fresh installation following docs/CODEX-QWEN.md, agents/codex-lean-home is deployed to C:\AI\codex-local\home, so this path does not contain the committed preflight.ps1; repo-wide search also finds no installer that copies it to the parent directory. PowerShell therefore emits no captured output, the wrapper falls back to WIN=24576 with LOCKPID=0, and Codex starts without the advertised engine lock or live window sizing.
Useful? React with 👍 / 👎.
| endlocal | ||
| set "SET_FILE=%LOCALAPPDATA%\Temp\claude-qwen-settings.json" | ||
| > "%SET_FILE%" echo {"autoCompactWindow": %WIN%, "autoCompactEnabled": true} | ||
| claude --bare --model qwen3.8-27b:code --settings "%SET_FILE%" %* |
There was a problem hiding this comment.
Keep the full Claude tool set when launching
For Claude coding sessions, --bare removes most of the core tools; the checked-in Claude guide specifically records that it leaves only three tools and says to avoid it. This regresses the prior launcher, which explicitly supplied Bash, Read, Edit, Write, Glob, Grep, WebFetch, Agent, and TodoWrite, so tasks that need file creation, discovery, search, or subagents no longer receive the documented harness.
Useful? React with 👍 / 👎.
| '-ngld', '99', '-md', 'C:/AI/models/qwen3.8-27b/Qwen3.8-27B-DFlash2-Q4_K_M.gguf', '--spec-type', 'draft-dflash', | ||
| '--spec-draft-n-max', '7', '-ub', '256', '-ctkd', 'q8_0', '-ctvd', 'q8_0', '--spec-draft-p-min', '0.4') + $Extra | ||
| $log = "$out\$Label.server.log" | ||
| $p = Start-Process -FilePath $Exe -ArgumentList $args2 -RedirectStandardOutput $log -RedirectStandardError "$log.err" -WindowStyle Hidden -PassThru |
There was a problem hiding this comment.
Stop the scratch server when a probe fails
If any operation after this launch throws—for example a PDH counter is unavailable, /tokenize times out, or the sustained evidence file cannot be parsed—$ErrorActionPreference = 'Stop' exits before the sole Stop-Process near the end. The scratch server then remains on port 10011 and can block or contaminate the next A/B run; wrap the post-launch work in try/finally so the process is always terminated.
Useful? React with 👍 / 👎.
| [string]$Exe = 'C:\AI\llama.cpp\vulkan-intelfa\llama-server.exe', | ||
| [string[]]$Extra = @(), | ||
| [hashtable]$Env = @{ GGML_VK_FORCE_MMVQ = '1' }, | ||
| [double]$AbortBelowTps = 0, |
There was a problem hiding this comment.
Honor the requested abort threshold
When callers set -AbortBelowTps to reject a slow variant after seed 1, the parameter is never referenced and sustained.py is still invoked with all $Runs. This defeats the documented early-abort behavior and spends the full benchmark duration on configurations the caller explicitly asked to stop; run the first seed separately and compare its throughput before continuing.
Useful? React with 👍 / 👎.
| if /i not "%~1"=="exec" goto :interactive | ||
| set "REST=%*" | ||
| call set "REST=%%REST:*exec=%%" | ||
| codex exec -c model_context_window=%WIN% -c model_auto_compact_token_limit=%COMPACT% %GITW% %REST% |
There was a problem hiding this comment.
Restore the client context cap in the Codex wrapper
The evaluation runners set QWEN_CLIENT_CAP from their --cap argument and record that value in every result, but this invocation now passes the uncapped live WIN to Codex and the wrapper never reads QWEN_CLIENT_CAP. Consequently a run labelled with cap: 32768 can actually use the 40–49K live client window, as the newly committed result rows demonstrate via their 40960/49152 contexts, invalidating cap-controlled comparisons and sustained-compaction tests.
Useful? React with 👍 / 👎.
| if (-not (Test-Path $LockPath)) { return $null } | ||
| try { $l = Get-Content $LockPath -Raw | ConvertFrom-Json; if ($l -and $l.pid) { return $l } } catch {} | ||
| # empty or half-written: another acquirer is writing it right now (CreateNew, then write) - treat as live for 10 s | ||
| return [pscustomobject]@{ pid = 0; kind = 'writing'; started = (Get-Item $LockPath).LastWriteTime.ToString('s') } |
There was a problem hiding this comment.
Treat a disappearing lock file as unlocked
When a holder releases the lock between Test-Path/Get-Content and this Get-Item, the latter throws under $ErrorActionPreference = 'Stop' because it sits outside the try. That race makes a waiting acquire process exit unexpectedly exactly as the previous job finishes; preflight then deliberately starts the next agent without a lock, allowing another waiter to enter concurrently. Catch the metadata lookup or return $null when the file has disappeared.
Useful? React with 👍 / 👎.
| $sus = "$out\$Label.sustained.jsonl" | ||
| if (Test-Path $sus) { Remove-Item $sus } | ||
| $py = Get-Command python | % Source | ||
| & $py "$out\sustained.py" --label $Label --output $sus --runs $Runs --stream --expected-context $Ctx 2>&1 | Out-Null |
There was a problem hiding this comment.
Reject incomplete sustained benchmark runs
If sustained.py fails after writing only one or two rows, PowerShell 5.1 does not turn its nonzero native exit code into a terminating error, and the following code reads the partial file, computes a mean, and publishes a normal result JSON. A transient request or metrics failure can therefore turn a required three-seed contract into an apparently valid one-seed result; check $LASTEXITCODE, the row count, and each row's long_enough status before recording the result.
Useful? React with 👍 / 👎.
| :done | ||
| endlocal | ||
| set "SET_FILE=%LOCALAPPDATA%\Temp\claude-qwen-settings.json" | ||
| > "%SET_FILE%" echo {"autoCompactWindow": %WIN%, "autoCompactEnabled": true} |
There was a problem hiding this comment.
Use a supported Claude compaction window
The live WIN produced here is approximately 24–57K, but Claude Code accepts autoCompactWindow only in the 100000–1000000 range, as also noted by the repository's pre-existing Claude documentation and the upstream Claude Code report. Every generated settings file is therefore outside the supported schema, so Claude may reject the settings or ignore the threshold and continue until the smaller local server returns a context-overflow error; retain the former CLAUDE_CODE_MAX_CONTEXT_TOKENS mechanism or use a supported control for these windows.
Useful? React with 👍 / 👎.
| '--cache-reuse', '256', '--metrics', '--reasoning-budget', '16384', '-fit', 'off', | ||
| '--chat-template-file', 'C:/AI/governor/qwen38-template.jinja', '--no-context-shift', | ||
| '-ngld', '99', '-md', 'C:/AI/models/qwen3.8-27b/Qwen3.8-27B-DFlash2-Q4_K_M.gguf', '--spec-type', 'draft-dflash', | ||
| '--spec-draft-n-max', '7', '-ub', '256', '-ctkd', 'q8_0', '-ctvd', 'q8_0', '--spec-draft-p-min', '0.4') + $Extra |
There was a problem hiding this comment.
Benchmark the actual production speculative threshold
This harness claims to copy the production command line and vary only the requested knobs, but governor/profiles.json does not set --spec-draft-p-min, whose upstream default is 0.00, while this line forces 0.4; the official llama.cpp server reference confirms that default. Because this threshold controls when speculative drafting stops and directly affects acceptance and generation speed, A0 is not the production baseline and the reported A/B result does not isolate only the main KV type.
Useful? React with 👍 / 👎.
| endlocal | ||
| set "SET_FILE=%LOCALAPPDATA%\Temp\claude-qwen-settings.json" | ||
| > "%SET_FILE%" echo {"autoCompactWindow": %WIN%, "autoCompactEnabled": true} | ||
| claude --bare --model qwen3.8-27b:code --settings "%SET_FILE%" %* |
There was a problem hiding this comment.
Retain Microsoft Store Claude executable discovery
For installations supplied by the Microsoft Store desktop app where claude.exe is under the package's LocalCache directory rather than exposed on PATH, this plain command now fails before starting a session. The replaced wrapper explicitly searched %LOCALAPPDATA%\Packages\Claude_* and the existing changelog documents that support, so preserve that resolver as a fallback instead of assuming every supported installation has a claude command alias.
Useful? React with 👍 / 👎.
- engine/patches/0003 (Codex's GDN state-snapshot fusion, speed-neutral) and 0004 (Intel rm_kq_int 1 -> 2 for K-quant mat-vec): 42.12 -> 51.22 tok/s, token-identical greedy output, test-backend-ops 695/695, soak 3x ~51.2. Promoted as vulkan-intelfa-kq2-20261018 (old engine kept). - VRAM spill: above ~22.6 GiB on the adapter Windows moves a ~0.94 GiB block of the model to system memory (26-30 tok/s). Live governor now counts shared usage, re-fits once on spill; margin 1.2 -> 1.45 GiB. Production after: 51.78 tok/s at 32K. Tuning-log rows 34-36. - Codex evalsuite on the new engine: 19/19 in 521 s (was 18/19 in 711 s) after four generic instruction rules; flaky three 9/9. - B60 hardware overclock: not applied (driver rejects the IGCL waiver). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4G2wCLwkaZUjYJd6T3uZV
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94064b245d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| & $py "$d\sustained.py" --label $Label --output "$d\$Label.sustained.json" --runs 3 --expected-context $ctx 2>&1 | Out-Null | ||
| $rows = Get-Content "$d\$Label.sustained.json" | ? { $_.Trim() } | % { $_ | ConvertFrom-Json } |
There was a problem hiding this comment.
Refuse to summarize stale production-check evidence
When this script is invoked again with the default or any previously used -Label, sustained.py rejects the existing output file, but PowerShell ignores the native nonzero exit and the next line parses the old file. The command therefore reports the previous run's throughput as if it came from the current server/context; remove or uniquely name the evidence file before launch and verify $LASTEXITCODE before reading it.
Useful? React with 👍 / 👎.
| 35c622fc30fa9c152324cea86eaf5321b7462fcc484fccbb03fa9179e93dffcc 0003-vulkan-fuse-the-Gated-DeltaNet-state-snapshot-copy-i.patch | ||
| 03d333aa94d5f63eb89c8b78a782db792e557585159d4bf767d2fcf3cf5320d7 0004-vulkan-Intel-Xe2-two-rows-per-workgroup-for-K-quant-.patch |
There was a problem hiding this comment.
Update the build's patch verification window
Adding these two patches makes a fresh build.ps1 checkout contain four applied commits, but engine/build.ps1 verifies the Intel FA patch using only git log -2. The newest two subjects are now 0004 and 0003, so the required Intel customized FA shader subject is absent and every fresh build exits at the verification check before compilation; inspect all applied patch commits or verify the resulting tree instead.
Useful? React with 👍 / 👎.
| if /i "%~1"=="exec" if not "%QWEN_GIT_WRITE%"=="0" for /f "usebackq delims=" %%g in (`git rev-parse --absolute-git-dir 2^>nul`) do set "GITDIR=%%g" | ||
| rem git prints C:/x/.git; pass the native C:\x\.git form to the Windows sandbox | ||
| if defined GITDIR set "GITDIR=%GITDIR:/=\%" | ||
| if defined GITDIR set "GITW=-c sandbox_workspace_write.writable_roots=['%GITDIR%']" |
There was a problem hiding this comment.
Allow the common Git directory for worktrees
When unattended exec runs inside a linked worktree, git rev-parse --absolute-git-dir returns only the per-worktree administrative directory, while objects and refs remain in the separate common directory; the Git documentation distinguishes this from --git-common-dir. Allowlisting only GITDIR therefore still leaves commits and branch creation unable to write the common object/ref storage, so resolve and add the absolute common directory as a writable root too.
Useful? React with 👍 / 👎.
- codex-qwen.cmd: the exec path re-parsed %* (text after & in a quoted prompt ran as a command; a .git path with spaces split into three arguments). The exec half moves to codex-qwen-exec.mjs, which gets already-split arguments and spawns codex.exe without a shell. Git config points at an empty folder for unattended runs (no more git/ignore warnings from the sandbox). - qwen-lock.ps1: grace covers the launcher's slot wait (was a 60 s takeover), compare-and-delete stale takeover (no double winners), PID start time recorded (reuse), delete-pending CreateNew retried instead of crashing. - test-qwen-lock.ps1: 14 checks including the takeover race; all pass. - Codex smoke on tool-git-branch, files-version-bump, tool-sha256, biz-price-update: 4/4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4G2wCLwkaZUjYJd6T3uZV
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1480f96eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| & "$d\run-ab.ps1" -Label P1-ctx49k-t8 -Ctx 49152 -Exe $exe -Runs 1 -SkipPP -Greedy | Out-Null; Row P1-ctx49k-t8 | ||
| & "$d\run-ab.ps1" -Label P2-ctx32k-t14 -Ctx 32768 -Exe $exe -Runs 1 -SkipPP -Greedy -Extra @('-t', '14', '-tb', '14') | Out-Null; Row P2-ctx32k-t14 | ||
| & "$d\run-ab.ps1" -Label P3-ctx40k-t8 -Ctx 40960 -Exe $exe -Runs 1 -SkipPP -Greedy | Out-Null; Row P3-ctx40k-t8 | ||
| 'probe done' | Add-Content $log |
There was a problem hiding this comment.
Stop the benchmark session after the probe
After all three probe runs succeed, the script exits here without invoking bench-session.ps1 stop. The earlier start action created a sleeper that holds qwen-engine.lock while bench.flag exists and switched the governor to OFF, so every normal probe completion leaves subsequent Qwen jobs blocked and automatic model management disabled until someone manually stops the session; put the stop action in a finally block so failures are covered too.
Useful? React with 👍 / 👎.
| $us = [regex]::Match($l, '([0-9.]+) us/run').Groups[1].Value | ||
| $bpw = if ($ta -eq 'q4_K') { 4.5 } else { 6.5625 } | ||
| $gbs = if ($us) { [math]::Round((4096.0 * 14336 * $bpw / 8) / ([double]$us * 1e-6) / 1e9, 1) } else { '' } | ||
| ("{0,-14} {1,-5} n={2} {3,9} us/run {4,9} GB/s" -f $c.name, $ta, $n, $us, $gbs) | Tee-Object -FilePath $Out -Append |
There was a problem hiding this comment.
Keep the sweep output in one encoding
Under the target Windows PowerShell 5.1 environment, the file is initialized as UTF-8 on line 15 but Tee-Object -FilePath ... -Append writes the subsequent rows as UTF-16LE. The committed mmv-sweep.txt demonstrates the result: a UTF-8 header followed by NUL-separated UTF-16 data, causing Git and ordinary UTF-8 analysis tools to treat the benchmark evidence as binary or unreadable. Append with an explicitly UTF-8 writer instead.
Useful? React with 👍 / 👎.
| set "QWEN_WIN=%WIN%" | ||
| set "QWEN_COMPACT_AT=%COMPACT%" | ||
| set "QWEN_GITDIR=%GITDIR%" | ||
| node "%~dp0codex-qwen-exec.mjs" %* |
There was a problem hiding this comment.
Ship the exec helper beside the launcher
On a fresh installation following docs/SETUP.md, only agents/codex-qwen.cmd is copied into %USERPROFILE%\bin; the newly required sibling codex-qwen-exec.mjs is not installed by that guide or by any repo installer. Every unattended invocation used by the eval runners therefore reaches this line and fails with Node's module-not-found error instead of starting Codex, so the helper must be deployed beside the wrapper or invoked from a documented installed location.
Useful? React with 👍 / 👎.
…n cap, nightly - llama-swap: default 27B ttl 7200 -> 0 so only the governor unloads (it saves the KV slot first). An explicit 0 is not replaced by globalTTL. - evalsuite/runner.py: --max-turns for Codex (default 25; passing tasks use <= 12), engine lock released after a killed run, event_log/turn_capped in each result row. - evalsuite: acceptance.txt (19 -> 27 tasks), prune_codex_trust.py (removes the per-workdir trust tables runs leave in the lean Codex home), nightly-codex.ps1, check_runner.py (offline checks, now a CI step). - Docs: tuning-log rows 37-40 (incl. the live governor hysteresis and the Open WebUI cache-eviction negative result), CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4G2wCLwkaZUjYJd6T3uZV
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b68709679f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| $busy = Get-CimInstance Win32_Process -Filter "Name='python.exe'" | ? { $_.CommandLine -match 'runner\.py|sustained\.py' } | ||
| $st = try { Get-Content C:\AI\governor\public\status.json -Raw | ConvertFrom-Json } catch { $null } | ||
| $mode = (Get-Content C:\AI\governor\mode.txt -ErrorAction SilentlyContinue | Select -First 1) | ||
| if ($busy -or ($st -and $st.processing) -or ($mode -eq 'OFF')) { L "SKIP: eval running, model busy or mode OFF"; exit 0 } |
There was a problem hiding this comment.
Skip nightlies while the engine lock is held
When an interactive Codex or Claude session is open but idle, status.json reports processing = false and this guard starts the nightly even though that session still owns qwen-engine.lock. Each task then waits for the preflight timeout and proceeds without the lock, so a 27-task run can spend hours in repeated 600-second waits and can overlap the interactive session when it becomes active again. Check the shared lock here and skip the nightly when it has a live holder.
Useful? React with 👍 / 👎.
| foreach ($c in $configs) { | ||
| foreach ($k in 'GGML_VK_TUNE_RM_KQ_INT', 'GGML_VK_TUNE_MMVQ_WG', 'GGML_VK_TUNE_RM_STDQ_INT', 'GGML_VK_TUNE_RM_KQ') { Remove-Item "env:$k" -ErrorAction SilentlyContinue } | ||
| foreach ($k in $c.env.Keys) { Set-Item "env:$k" $c.env[$k] } | ||
| & powershell -NoProfile -ExecutionPolicy Bypass -File "$dir\run-ab.ps1" -Label $c.label -Kv f16 -Ctx 32768 -Runs 3 -Exe $exe -Greedy | Out-Null |
There was a problem hiding this comment.
Record inherited tuning knobs in each result
For the K1/K2 runs, this driver sets the tuning variables in its own environment but invokes run-ab.ps1 without passing them through its -Env parameter; run-ab.ps1 therefore serializes only its default hashtable. The committed K0/K1/K2 result files consequently all report GGML_VK_FORCE_MMVQ=1 even though different GGML_VK_TUNE_* settings produced them, and kernel-ab2.ps1 has the same problem for W0/W1/W2. Record the effective inherited variables or pass a merged environment map so the benchmark evidence identifies the configuration actually measured.
Useful? React with 👍 / 👎.
| rem Lean MCP set for the local model: keep context7; cloud/heavy servers off (playwright on with QWEN_BROWSER=1) | ||
| set "MCPOFF=-c mcp_servers.render.enabled=false -c mcp_servers.openaiDeveloperDocs.enabled=false -c mcp_servers.headroom.enabled=false -c mcp_servers.node_repl.enabled=false" | ||
| if not "%QWEN_BROWSER%"=="1" set "MCPOFF=%MCPOFF% -c mcp_servers.local-ai-playwright.enabled=false" | ||
| codex --profile qwen -c model_context_window=%QWEN_WIN% -c model_auto_compact_token_limit=%QWEN_COMPACT% %MCPOFF% %* |
There was a problem hiding this comment.
Put the full Codex launcher under the engine lock
When the advertised MCP-heavy codex-qwen-full launcher is used, it starts Codex without the shared preflight or qwen-engine.lock. Because it targets the same single-slot Qwen server, a lean Codex, Claude, or eval job can start while this session is idle between turns and both sessions then interleave requests at roughly half speed; conversely, this full session can start while another lock holder is active. Acquire and release the same lock around this invocation.
Useful? React with 👍 / 👎.
What
agents/codex-qwen.cmdruns the lean Codex home (about 3K base prompt tokens) behind a shared engine lock, so only one Qwen job runs at a time.claude-qwentakes the same lock, and the old launcher is kept ascodex-qwen-full.cmd. Write-up:docs/CODEX-QWEN.md.bench/ab-20261018/, TUNING-LOG rows 28-33.Evidence (all measured on ELZYDLAB, 18 Oct)
Not in this PR
🤖 Generated with Claude Code
https://claude.ai/code/session_01G4G2wCLwkaZUjYJd6T3uZV