Skip to content

Commit f05a9fc

Browse files
authored
ci: post recursion guest cycle+accelerator diff (main vs PR) on /bench-verify (#808)
* feat(cli): report keccak/ecsm accelerator call counts under `execute --cycles` The cycle counter only counted retired instructions, so a keccak or ecsm accelerator call — one ecall that runs a whole permutation / scalar-mul in a single cycle — was invisible. `execute --cycles` now also prints `Keccak calls` and `Ecsm calls`, classified exactly as the prover's CpuOperation::from_log does (a7 == KECCAK/ECSM syscall number on an EcallEbreak), tallied in the memory-safe streaming loop. * feat(bench): deterministic recursion-guest cycle/accelerator comparison Add scripts/bench_recursion_cycles.sh. The in-VM recursion verifier's cost for a fixed (guest ELF, input blob) is fully deterministic, so comparing two git refs is two EXACT integer readings — total guest cycles, Keccak accelerator calls, Ecsm accelerator calls — with no A/B/B/A interleaving. For each ref the script: adds an isolated worktree at the ref's SHA, builds the recursion guest via `make compile-recursion-elfs` (auto-detecting recursion-<PRESET>.elf, falling back to recursion.elf on refs that build a single unnamed guest), generates that ref's own input blob via the ignored test_dump_recursion_input, and measures it with a measuring CLI built once from this branch (`execute --cycles`, which reports the keccak/ecsm counts). Results are cached per ref SHA so re-runs are fast; worktrees are kept so a re-measure is a cargo no-op. It fails with a clear, ref-named message when a ref lacks the dump test or a recursion guest artifact. Prints an exact comparison table (negative delta = PR fewer = better, matching bench_verify's sign convention) plus a machine-parseable raw-number block for downstream tooling. Usage: scripts/bench_recursion_cycles.sh REF_A [REF_B=origin/main] [PRESET=min] * ci(bench-verify): post recursion guest cycle+accelerator diff (main vs PR) After the verifier benchmark, run scripts/bench_recursion_cycles.sh and append its deterministic recursion-guest cycle/keccak/ecsm comparison to the same /bench-verify PR comment (same marker comment; not a second one). The step is additive and isolated from the verifier bench: continue-on-error plus `if: !cancelled()` keep a recursion-bench failure from failing the job or clobbering the verifier verdict, and it still runs if the verifier step failed. Bump the job timeout to 90 min for the cold recursion guest builds (~10-20 min/ref the first time; the measurement itself is ~1 min). * bench(recursion-cycles): harden result cache + workflow timeout/build-sharing Address review findings on the recursion-guest cycle bench. Script (scripts/bench_recursion_cycles.sh): - Corrupt-cache correctness: write result files atomically (tmp + mv) and validate every cached record on read (numeric keys integer, elf non-empty) via a new valid_result(); an incomplete/partial cache is now discarded and re-measured, and the final table/RAW emit is guarded so a truncated record fails loudly instead of posting a bogus -100%. - Cache key now folds in the MEASURE_CLI source SHA so the two ref sides can never be compared across different counters. - Bounded, self-limiting worktree cache: keep the newest PRUNE_KEEP (default 10) ref worktrees, prune older ones + their results/blobs/logs at startup (touch-on-use tracks recency), and remove a worktree whose guest build fails mid-run so partial trees never linger. - Docs: correct the determinism claim (each ref dumps its own blob, so a proof-format change conflates guest-code and proof-structure deltas); explain that main->recursion.elf vs PR->recursion-min.elf is the expected comparison (both under min proof options) and surface the differing artifacts prominently in the output; reconcile the MEASURE_CLI "built from main in CI" comment. Workflow (.github/workflows/bench-verify.yml): - Add a step-level timeout-minutes: 70 to the recursion step (fail-fast well under the 90-min job cap; continue-on-error absorbs it so the verifier verdict is still posted). - Set GUEST_TARGET_DIR / HOST_TARGET_DIR so the RV64 build-std and host cargo target are shared across ref worktrees (cuts cold time and per-worktree disk). - Update the timeout comments to reflect the real cold cost (MEASURE_CLI + per-ref guest build + per-ref prover-test build for the blob dump). - Bound the success-path raw-log fallback in the Post-result script the same way the failure branch is bounded; note each ref dumps its own blob.
1 parent a81c44e commit f05a9fc

2 files changed

Lines changed: 413 additions & 4 deletions

File tree

‎.github/workflows/bench-verify.yml‎

Lines changed: 64 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,14 @@ jobs:
2626
startsWith(github.event.comment.body, '/bench-verify') &&
2727
contains(fromJSON('["MEMBER","OWNER","COLLABORATOR"]'), github.event.comment.author_association)
2828
runs-on: [self-hosted, bench]
29-
timeout-minutes: 60
29+
# Job cap. The verifier bench is ~4 min and the recursion measurement itself ~1 min,
30+
# but on a cold runner the recursion BUILDS dominate: MEASURE_CLI (release cli) once,
31+
# plus PER REF a guest build (~10-20 min) and a prover-test build for the blob dump.
32+
# All cached in /tmp for later runs, and build-std / the host cargo target are shared
33+
# across ref worktrees (see the recursion step's env) to keep the cold run under this
34+
# cap. The recursion step also has its own tighter timeout so a runaway build there
35+
# can't burn the whole job and lose the already-captured verifier result.
36+
timeout-minutes: 90
3037
steps:
3138
- name: Acknowledge (react + occupancy notice)
3239
uses: actions/github-script@v7
@@ -85,27 +92,80 @@ jobs:
8592
scripts/bench_verify.sh "$HEAD_SHA" origin/main "$PAIRS" 2>&1 | tee /tmp/verify_out.txt
8693
sed -n '/=== Verify ABBA result/,$p' /tmp/verify_out.txt > /tmp/verify_result.txt
8794
95+
# Additive: deterministic recursion-guest cycle+accelerator diff (PR vs main).
96+
# The measurement is one exact `execute --cycles` reading per ref (~1 min total, no
97+
# ABBA). Cold wall time is dominated by BUILDS: MEASURE_CLI (release cli) once, plus
98+
# per ref a guest build (~10-20 min) and a prover-test build for the blob dump; the
99+
# GUEST_TARGET_DIR / HOST_TARGET_DIR below share build-std and the host cargo target
100+
# across the two ref worktrees so the second ref is much cheaper (and per-worktree
101+
# disk shrinks). continue-on-error + `!cancelled()` keep this fully isolated from the
102+
# verifier bench above: a failure OR timeout here never fails the job nor clobbers the
103+
# verifier verdict, and it still runs if the verifier step failed.
104+
- name: Run recursion guest cycle benchmark
105+
id: recursion
106+
if: ${{ !cancelled() }}
107+
continue-on-error: true
108+
# Fail-fast well under the 90-min job cap so a runaway recursion build can't burn
109+
# the whole job and lose the already-captured verifier result (continue-on-error
110+
# absorbs the timeout; the job still posts the verifier verdict). Sized with
111+
# headroom over a cold shared-build run (MEASURE_CLI + 2 guest builds + 2 blob-dump
112+
# builds).
113+
timeout-minutes: 70
114+
env:
115+
HEAD_SHA: ${{ steps.cfg.outputs.head_sha }}
116+
# Share build-std and the host cargo target across ref worktrees, rooted under
117+
# the script's /tmp cache dir: cuts cold build time and per-worktree disk.
118+
GUEST_TARGET_DIR: /tmp/recursion_cycles_run/shared_guest_target
119+
HOST_TARGET_DIR: /tmp/recursion_cycles_run/shared_host_target
120+
run: |
121+
export SYSROOT_DIR="$HOME/.lambda-vm-sysroot"
122+
set -o pipefail
123+
scripts/bench_recursion_cycles.sh "$HEAD_SHA" origin/main min 2>&1 | tee /tmp/recursion_out.txt
124+
sed -n '/=== Recursion-guest cycle/,$p' /tmp/recursion_out.txt > /tmp/recursion_result.txt
125+
88126
- name: Post result
89127
if: always()
90128
uses: actions/github-script@v7
91129
env:
92130
HEAD_SHA: ${{ steps.cfg.outputs.head_sha }}
93131
PAIRS: ${{ steps.cfg.outputs.pairs }}
94132
OUTCOME: ${{ steps.run.outcome }}
133+
RECURSION_OUTCOME: ${{ steps.recursion.outcome }}
95134
with:
96135
script: |
97136
const fs = require('fs');
98137
const read = (p) => { try { return fs.readFileSync(p, 'utf8').trim(); } catch { return ''; } };
138+
// Bound any raw-log fallback so a future header rename (empty *_result.txt)
139+
// can't dump the entire build log into the PR comment.
140+
const tail = (s, n) => s.split('\n').slice(-n).join('\n');
99141
const head = (process.env.HEAD_SHA || '').slice(0, 10), pairs = process.env.PAIRS;
100142
let body = `## Verifier benchmark — \`${head}\` vs \`main\` (${pairs} pairs)\n\n`;
101143
if (process.env.OUTCOME === 'success') {
102-
const res = read('/tmp/verify_result.txt') || read('/tmp/verify_out.txt');
144+
const res = read('/tmp/verify_result.txt') || tail(read('/tmp/verify_out.txt'), 30);
103145
body += res + '\n';
104146
body += '\n<sub>Drift-free interleaved A/B/B/A measurement. - = PR faster. ';
105147
body += 'Trust the verdict when paired-t and Wilcoxon agree.</sub>\n';
106148
} else {
107-
const tail = read('/tmp/verify_out.txt').split('\n').slice(-30).join('\n');
108-
body += `❌ Run failed. Last log lines:\n\n` + '```\n' + tail + '\n```\n';
149+
body += `❌ Run failed. Last log lines:\n\n` + '```\n' + tail(read('/tmp/verify_out.txt'), 30) + '\n```\n';
150+
}
151+
// Additive recursion-guest cycle section, kept clearly separated from the
152+
// verifier verdict above so a failure here can't change how the bench reads.
153+
body += '\n---\n\n## Recursion guest cycles (main vs PR)\n\n';
154+
if (process.env.RECURSION_OUTCOME === 'success') {
155+
const rec = read('/tmp/recursion_result.txt') || tail(read('/tmp/recursion_out.txt'), 20);
156+
if (rec) {
157+
body += rec + '\n';
158+
body += '\n<sub>Deterministic in-VM verifier: one exact `execute --cycles` reading per ref ';
159+
body += '(no ABBA needed). - = PR does fewer cycles/calls = better. keccak/ecsm are 0 when ';
160+
body += 'the verifier uses Poseidon2. Each ref dumps its own input blob, so for a PR that ';
161+
body += 'changes the proof format the delta reflects both guest-code and proof-structure changes.</sub>\n';
162+
} else {
163+
body += '_No recursion comparison output was captured._\n';
164+
}
165+
} else {
166+
const rtail = tail(read('/tmp/recursion_out.txt'), 20);
167+
body += '⚠️ Recursion cycle bench did not complete (does not affect the verifier verdict above).';
168+
body += rtail ? ' Last log lines:\n\n' + '```\n' + rtail + '\n```\n' : '\n';
109169
}
110170
const { data: comments } = await github.rest.issues.listComments({
111171
owner: context.repo.owner, repo: context.repo.repo,

0 commit comments

Comments
 (0)