Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/selftest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ jobs:
- name: Validate all 100 hidden checkers (untouched must fail, reference must pass)
working-directory: evalsuite
run: python selftest.py --ci
- name: Runner turn cap, acceptance set and trust pruning (offline)
working-directory: evalsuite
run: python check_runner.py


gateway:
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Changelog

## Unreleased - 2026-10-18 Codex lock-in and engine A/B

- **Codex is the one harness.** `agents/codex-qwen.cmd` now runs the lean Codex home ([agents/codex-lean-home/](agents/codex-lean-home/), about 3K base prompt tokens) with a preflight that takes a shared engine lock (`agents/locks/qwen-lock.ps1`, `C:\AI\locks\qwen-engine.lock`) so only one Qwen job runs at a time; `claude-qwen` takes the same lock. The old heavy launcher is kept as `codex-qwen-full.cmd`. Details: [docs/CODEX-QWEN.md](docs/CODEX-QWEN.md).
- **evalsuite, same 19 tasks:** old home 13/19; lean home 15/19 and 49% faster on the 12 tasks both passed (930 s -> 477 s), 85% fewer server prompt tokens; after five traced fixes the acceptance run scored **18/19, all 19 in 711 s** (old: 2,140 s), and the last failure (money rounding) passed 3/3 after a tighter rule. Raw rows in `data/codex-eval-20261018/`.
- Fixes found from the Codex event logs: PowerShell `>` writes UTF-16; the compact prompt's own patch example taught a stray space; Codex's Windows sandbox keeps `.git` read-only and a root-level `writable_roots` override is ignored under `exec -s` (now placed after `exec`, unattended runs only); repeat-the-failing-command loops; float money rounding; positional CSV parsing.
- **Engine A/B on production Vulkan 11526 (ctx 32768, 3-seed contract, `bench/ab-20261018/`):** f16 KV baseline 38.95 / 43.01 / 44.16 tok/s (mean 42.04), pp 471 (2K) / 520 (8K). q8_0 main KV with DFlash2: 34.34 / 33.62 / 37.52 (mean 35.16, -16%), pp 433 / 381, saves 0.94 GiB: **rejected again**, so 48K/56K q8_0 was not run.
- **DFlash round attribution (GGML_VK_PERF_LOGGER):** 125.2 ms/round of GPU time against 128.3 ms/round wall, so the host readback/injection/sync gap is about 3 ms: the round is kernel-bound. The verify graph (101 ms/round) is n=8 mat-vec over Q4_K/Q6_K at about 160 GB/s, about 35% of the card's 456 GB/s; that kernel efficiency is the remaining lever, not the DFlash bridge.
- **SYCL fork review (valarauca/llama.cpp `feature/intel`, MIT):** source audit only, not built. Its large prefill gains target Q4_0/Q8_0; Q4_K/Q6_K get about 20% faster matmul, so the 2x prompt-speed gate for this Q4_K_M model was not expected to pass (INFERRED +10-30%), and speculative decoding on SYCL has an open upstream issue (#23203). Parked; the earlier 1.2% figure came from a Vulkan port of one idea, not the SYCL build.
- **B60 hardware tuning:** the Intel control library (IGCL 1.2.298) lists overclock ranges for the card, but its overclock waiver call returns `CTL_RESULT_ERROR_UNSUPPORTED_FEATURE`, so nothing was applied and the card runs stock (VERIFIED by readback). The card runs at about 141 W of a 200 W cap at 2.4 GHz.
- **Engine: two rows per workgroup for Intel K-quant mat-vec (`engine/patches/0004`).** The DFlash2 verify batch is an n=8 mat-vec that ran at about 35% of the card's bandwidth. `rm_kq_int` 1 -> 2: q4_K 178.6 -> 133.8 us, q6_K 288 -> 253 us (m=4096 k=14336); end to end **42.12 -> 51.22 tok/s** mean (3 seeds, ctx 32768, 127 -> 104 ms/round), token-identical greedy output, test-backend-ops 695/695, soak 51.19 / 51.22 / 51.09. Promoted as `C:\AI\llama.cpp\vulkan-intelfa-kq2-20261018` (one-line governor `exe` change; old engine kept for rollback). `engine/patches/0003` is Codex's GDN state-snapshot fusion: correct, speed-neutral, kept. Tuning-log rows 34-35.
- **VRAM spill found and fixed (governor).** Production ran 26-30 tok/s at 49K context: once the adapter passes about 22.6 GiB, Windows moves a ~0.94 GiB block of the model to system memory. The governor only counted dedicated memory, so the spill looked like a perfect fit and its calibration shrank the next margin further. Live governor: margin 1.2 -> 1.45 GiB, pressure threshold 1.1 -> 1.3, calibration counts llama shared usage, a spill queues one idle-time re-fit; 4 new tests (195 of 196 pass; the remaining failure is an older `others_vram_max_gib` drift). Production after the fix: **51.78 tok/s** mean (47.83 / 53.11 / 54.41) at 32K. This corrects the 1.2 GiB margin from the context tune below. Tuning-log row 36.
- **Codex on the new engine: 19/19 in 521 s** (18/19 in 711 s before the engine change), after four generic rules in `agents/codex-lean-home/instructions-compact.md`: an unindented apply_patch example, one-step rounding to 5 cents, edge cases beyond the given tests, and checking every patched file for a stray leading space. The three tasks that had been flaky passed 9/9. Details in [docs/CODEX-QWEN.md](docs/CODEX-QWEN.md), raw rows `data/codex-eval-20261018/codex-kq2-*`.
- **Review fixes (launcher and lock).** `agents/codex-qwen.cmd` re-parsed its arguments to drop `exec`, so text after `&` inside a quoted prompt ran as a command and a `.git` path with spaces split; the exec half now lives in `agents/codex-qwen-exec.mjs`, which receives the arguments already split and passes them to `codex.exe` untouched (it also points git at an empty config folder, silencing the sandbox's `git/ignore` warnings). `agents/locks/qwen-lock.ps1`: a launcher waiting for the slot was taken over after 60 s (grace now covers the wait), two waiters could both take over a stale lock (compare-and-delete under an exclusive handle), reused PIDs kept a lock (start time recorded), and a delete-pending lock file crashed the acquirer. New `agents/locks/test-qwen-lock.ps1` (14 checks, including the takeover race). Codex smoke on four tasks: 4/4.
- **Unloads, profile flapping, eval runner (19 Oct).** llama-swap `ttl` for the default 27B is now 0 so only the governor unloads (it saves the KV slot first; an explicit 0 is not replaced by `globalTTL`). Live governor: MAX/BALANCED hysteresis (enter MAX at RAM >= 10 GB and CPU < 25%, leave below 8.5 GB or above 40%) after 16 flips in one evening, and an optional upgrade reload needs at least +8K context; 217/217 governor tests. The suspected Open WebUI cache eviction was not real (task generation off, no chats; the only other client is Codex's Qwen MCP gateway), so nothing changed there. evalsuite: `--max-turns` for Codex (default 25), the engine lock is released after a killed run, `prune_codex_trust.py` removes the per-workdir trust tables each run leaves in the lean Codex home, acceptance set 19 -> 27 in `evalsuite/acceptance.txt` (the 8 added tasks 8/8 on kq2), and `nightly-codex.ps1` runs it nightly with a regression comparison. `evalsuite/check_runner.py` covers the cap and pruning offline and runs in CI. Tuning-log rows 37-40.

## Unreleased — 2026-10-18 context tune

- Context sweep on the production launch line (32K/40K/48K/56K, short and 80%-full requests): speed holds to 48K (43.6 tok/s with 39K tokens in context) and falls off at 56K (22.4 tok/s, about 1 GiB moved to system memory). New `bench/ctx-sweep.py`, raw results in `data/ctx-sweep-20261018.jsonl`, tuning-log rows 24-27.
Expand Down
78 changes: 28 additions & 50 deletions agents/claude-qwen.cmd
Original file line number Diff line number Diff line change
@@ -1,55 +1,33 @@
@echo off
rem Claude Code (real CLI) on the local Qwen3.8-27B (Arc B60, llama-swap, loads on request).
rem Usage: claude-qwen [claude args]. Default preset qwen3.8-27b:code (fast agent turns, no thinking).
rem Deep planning: set QWEN_MODEL=qwen3.8-27b:plan Balanced: qwen3.8-27b Huge docs: qwen3.8-27b-long
rem claude-qwen: Claude Code running on the local Qwen3.8-27B (ELZYDLAB Arc B60) through llama-swap's
rem Anthropic-compatible /v1/messages endpoint. --bare keeps the prompt small (no plugins, hooks, connectors,
rem CLAUDE.md auto-discovery). Your normal "claude" command and settings are not changed.
rem Usage: claude-qwen [claude args] e.g. claude-qwen or claude-qwen -p "explain this repo"
rem Add context explicitly if you want it: --add-dir . (CLAUDE.md), --mcp-config <file>
setlocal
if "%QWEN_MODEL%"=="" set "QWEN_MODEL=qwen3.8-27b:code"
if "%QWEN_BASE%"=="" set "QWEN_BASE=http://127.0.0.1:8090"
set "CLAUDE_BIN="
if exist "%USERPROFILE%\.local\bin\claude.exe" set "CLAUDE_BIN=%USERPROFILE%\.local\bin\claude.exe"
if not defined CLAUDE_BIN for /f "usebackq delims=" %%p in (`powershell -NoProfile -Command "$r=@($env:APPDATA+'\Claude\claude-code')+@(Get-ChildItem ($env:LOCALAPPDATA+'\Packages') -Directory -Filter 'Claude_*' -ErrorAction SilentlyContinue | ForEach-Object { $_.FullName+'\LocalCache\Roaming\Claude\claude-code' }); ($r | Where-Object { Test-Path $_ } | ForEach-Object { Get-ChildItem $_ -Recurse -Force -Filter claude.exe -ErrorAction SilentlyContinue } | Sort-Object LastWriteTime -Descending | Select-Object -First 1).FullName"`) do set "CLAUDE_BIN=%%p"
if not defined CLAUDE_BIN ( echo Claude Code not found. Install it: powershell -c "irm https://claude.ai/install.ps1 | iex" & exit /b 2 )
if "%LOCAL_QWEN_KEY%"=="" for /f "usebackq delims=" %%k in (`powershell -NoProfile -Command "[Environment]::GetEnvironmentVariable('LOCAL_QWEN_KEY','User')"`) do set "LOCAL_QWEN_KEY=%%k"
if not defined LOCAL_QWEN_KEY ( echo LOCAL_QWEN_KEY is not set - see docs\SETUP.md step 3. & exit /b 3 )
set "ANTHROPIC_BASE_URL=%QWEN_BASE%"
set "ANTHROPIC_AUTH_TOKEN=%LOCAL_QWEN_KEY%"
set "ANTHROPIC_API_KEY="
set "ANTHROPIC_MODEL=%QWEN_MODEL%"
set "ANTHROPIC_DEFAULT_OPUS_MODEL=%QWEN_MODEL%"
set "ANTHROPIC_DEFAULT_SONNET_MODEL=%QWEN_MODEL%"
set "ANTHROPIC_DEFAULT_HAIKU_MODEL=%QWEN_MODEL%"
set "CLAUDE_CODE_SUBAGENT_MODEL=%QWEN_MODEL%"
rem Live context window from the governor (loaded ctx or expected ctx, minus 8k headroom)
set "QWEN_WIN=50000"
set "WINARG="
if /i "%QWEN_MODEL%"=="qwen3.8-27b-long" set "WINARG=long"
for /f "usebackq delims=" %%w in (`powershell -NoProfile -ExecutionPolicy Bypass -File C:\AI\governor\qwen.ps1 window %WINARG%`) do set "QWEN_WIN=%%w"
if "%QWEN_CLIENT_CAP%"=="" set "QWEN_CLIENT_CAP=32768"
if %QWEN_WIN% GTR %QWEN_CLIENT_CAP% set "QWEN_WIN=%QWEN_CLIENT_CAP%"
if /i "%QWEN_MODEL%"=="qwen3.8-27b-big" set "QWEN_WIN=120000"
set "CLAUDE_CODE_MAX_CONTEXT_TOKENS=%QWEN_WIN%"
rem Efficient sessions: 8k output reserve (more room before auto-compact), capped tool output
set "CLAUDE_CODE_MAX_OUTPUT_TOKENS=8000"
set "BASH_MAX_OUTPUT_LENGTH=15000"
set "MAX_MCP_OUTPUT_TOKENS=8000"
set "CLAUDE_CODE_ATTRIBUTION_HEADER=0"
rem One Qwen job at a time: preflight takes C:\AI\locks\qwen-engine.lock (kind claude) and prints "<window> <lockHolderPid>".
set "WIN=" & set "LOCKPID=0"
for /f "usebackq tokens=1,2" %%a in (`powershell -NoProfile -ExecutionPolicy Bypass -File C:\AI\codex-local\preflight.ps1 -Kind claude`) do (
set "WIN=%%a"
set "LOCKPID=%%b"
)
if not defined WIN set "WIN=28672"
set "ANTHROPIC_BASE_URL=http://127.0.0.1:8090"
set "ANTHROPIC_API_KEY=%LOCAL_QWEN_KEY%"
set "ANTHROPIC_AUTH_TOKEN="
set "ANTHROPIC_MODEL=qwen3.8-27b:code"
set "ANTHROPIC_DEFAULT_OPUS_MODEL=qwen3.8-27b:code"
set "ANTHROPIC_DEFAULT_SONNET_MODEL=qwen3.8-27b:code"
set "ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3.8-27b:code"
set "ANTHROPIC_SMALL_FAST_MODEL=qwen3.8-27b:code"
set "CLAUDE_CODE_SUBAGENT_MODEL=qwen3.8-27b:code"
set "CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192"
set "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1"
set "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1"
rem (The "v2" extra settings tested on 2026-10-10 scored lower - 93/100 vs 97/100 - and are not used. See docs/RELIABILITY.md.)
set "API_TIMEOUT_MS=900000"
rem Compaction is governed by CLAUDE_CODE_MAX_CONTEXT_TOKENS above. CLAUDE_CODE_AUTO_COMPACT_WINDOW is deliberately NOT set:
rem its documented valid range is 100000-1000000, so the 32K-57K windows used here are out of range.
rem Browser tools (playwright) are opt-in: set QWEN_BROWSER=1 and create C:\AI\claude-local\mcp-browser.json yourself (not shipped: browser tool schemas are large and load on every turn)
set "MCPFILE=C:\AI\claude-local\mcp.json"
if "%QWEN_BROWSER%"=="1" if exist "C:\AI\claude-local\mcp-browser.json" (set "MCPFILE=C:\AI\claude-local\mcp-browser.json") else (echo mcp-browser.json not found in C:\AI\claude-local - continuing without browser tools.)
rem DEFAULT = FAST mode: full core toolset (Bash Read Edit Write Glob Grep WebFetch Agent TodoWrite + context7) with a compact local system prompt instead of Claude Code's ~19k-token one.
rem set QWEN_FULL=1 standard Claude Code prompt (lean config dir, hooks/skills discovery)
if "%QWEN_FULL%"=="1" goto full
set "CLAUDE_CONFIG_DIR=C:\AI\claude-local"
"%CLAUDE_BIN%" --model %QWEN_MODEL% --system-prompt-file C:\AI\claude-local\LOCAL-SYSTEM.md --tools "Bash,Read,Edit,Write,Glob,Grep,WebFetch,Agent,TodoWrite" --mcp-config "%MCPFILE%" --strict-mcp-config %*
goto done
:full
set "CLAUDE_CONFIG_DIR=C:\AI\claude-local"
"%CLAUDE_BIN%" --model %QWEN_MODEL% --mcp-config "%MCPFILE%" --disallowedTools WebSearch %*
:done
endlocal
set "SET_FILE=%LOCALAPPDATA%\Temp\claude-qwen-settings.json"
> "%SET_FILE%" echo {"autoCompactWindow": %WIN%, "autoCompactEnabled": true}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use a supported Claude compaction window

The live WIN produced here is approximately 24–57K, but Claude Code accepts autoCompactWindow only in the 100000–1000000 range, as also noted by the repository's pre-existing Claude documentation and the upstream Claude Code report. Every generated settings file is therefore outside the supported schema, so Claude may reject the settings or ignore the threshold and continue until the smaller local server returns a context-overflow error; retain the former CLAUDE_CODE_MAX_CONTEXT_TOKENS mechanism or use a supported control for these windows.

Useful? React with 👍 / 👎.

claude --bare --model qwen3.8-27b:code --settings "%SET_FILE%" %*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the full Claude tool set when launching

For Claude coding sessions, --bare removes most of the core tools; the checked-in Claude guide specifically records that it leaves only three tools and says to avoid it. This regresses the prior launcher, which explicitly supplied Bash, Read, Edit, Write, Glob, Grep, WebFetch, Agent, and TodoWrite, so tasks that need file creation, discovery, search, or subagents no longer receive the documented harness.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the requested Claude model override

When a user sets QWEN_MODEL=qwen3.8-27b:plan (the documented way to request deep planning), this hard-coded CLI argument overrides it and always selects the code preset; the surrounding model environment variables are also hard-coded. This removes the existing plan/general/vision selection behavior without providing another launcher option, so the wrapper silently runs the wrong sampling and thinking configuration.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain Microsoft Store Claude executable discovery

For installations supplied by the Microsoft Store desktop app where claude.exe is under the package's LocalCache directory rather than exposed on PATH, this plain command now fails before starting a session. The replaced wrapper explicitly searched %LOCALAPPDATA%\Packages\Claude_* and the existing changelog documents that support, so preserve that resolver as a fallback instead of assuming every supported installation has a claude command alias.

Useful? React with 👍 / 👎.

set "RC=%ERRORLEVEL%"
if not "%LOCKPID%"=="0" powershell -NoProfile -ExecutionPolicy Bypass -File C:\AI\locks\qwen-lock.ps1 release -HolderPid %LOCKPID% >nul 2>&1
endlocal & exit /b %RC%
Loading
Loading