feat(*): self-evolution stack: evolver, appworld bench, unified entry#140
Merged
cyfyifanchen merged 4 commits intoJul 17, 2026
Merged
Conversation
ethanwwwwwwww
force-pushed
the
feat/port_evolver_appworld_bench
branch
from
July 17, 2026 03:33
c54e60e to
37b6b00
Compare
Port the harness self-evolution framework onto Raven and grow it into a
runnable, benchmark-pluggable stack:
- raven/evolver/: the evolution machinery - orchestrator state machine
(seven-step funnel), three verification gates (infra validity, beacon
attribution, paired significance), sealed-test runner with retention,
git-backed candidate tree, judge, taxonomy, activation ledger
- benchmarks/appworld/ (repo-root bench area, out of the wheel): the AppWorld bench (agent_cli, batch
scorer with trial-level resume, infra-rerun ladder) + evolve glue
(designer, diagnosis, adapter, precheck with real-generation probing)
- unified entry: python -m raven.evolver {run,check,status,finalize}
drives cold start -> rounds -> terminate -> unseal as a resumable
state machine; config drift and unseal one-wayness are mechanized in
run meta; benches plug in via a documented contract, loaded from the
subject repo root (bench plugins live outside the installed package)
- designer feedback: inert-death loop (pruned_inert status, per-WHY
history, gentle decay) so unreachable triggers teach the next attempt
- launch hygiene: check runs the bench Gate0 precheck (a dead subject
endpoint fails at check time, not at cold start); ephemeral eval/edit
worktrees live under work_dir/tmp and hard-kill leftovers are swept
on the next launch of the same run
- docs: SOP translation (internal details redacted), SOP-to-code map,
bench contract, user README with security notes and known limitations
- verified: 56 unit/e2e tests plus a real AppWorld smoke through the
unified entry (cold start, resume, one full round, honest Gate-b
verdict, termination)
Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
ethanwwwwwwww
force-pushed
the
feat/port_evolver_appworld_bench
branch
from
July 17, 2026 05:09
37b6b00 to
7f91bfb
Compare
Add 70 unit tests over the evolver's decision core and side-effect boundaries, which were previously covered only transitively by the fake-bench e2e and real smoke runs: - gates: paired z statistics against hand-computed vectors (deterministic win/regression, sqrt(3) mid case, missing-task-scores-zero), Fisher exact hand-checked values and degenerate margins, three-shield pipeline narrowing (infra reported but never dropped, Gate-b fail-open vs fail-closed), and both gate policies end to end - including the dual condition where a fired subset win must not promote a full-train regression - scoring: the SOP 0 infra-rerun ladder (missing task = infra, keep fewest-infra measurement, persistent infra survives at K and scores 0) and flip_summary accounting - git_ops: commit_files_as_child leaves the main working tree and HEAD untouched, worktree_at cleans up on both exit paths, evolver refs anchor candidates against gc, ephemeral-root redirection - sandbox: whitelist write refusal, origin-repo bash boundary, scope_restore reverting only out-of-whitelist edits, exact change and deletion capture, import smoke resolving from the edited worktree - drivers: claude_cli transcript serialization, argv/stdin shape with tools off, retry and rate-limit stretch, and the Gate0 endpoint probe mapping each failure mode to its actionable message Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
0xKT
previously approved these changes
Jul 17, 2026
The pre-commit CI job lints every file a PR touches, which put the whole ported evolver/bench tree in scope; make lint-python only covers the repo hygiene scripts, so these findings never surfaced locally. - fix real findings: PassFailResult was referenced in an annotation but only imported inside the function (F821) - import it at module level; drop a dead _run call in git_ops.branch_exists and an unused local in tree_loader (F841); rename three ambiguous `l` loop variables (E741) - register the idiomatic patterns as per-file-ignores in pyproject with reasons, following the existing sections: S607 for the git/bash subprocess drivers, N803/N806 for K/T as SOP domain vocabulary, S105 for outcome-label constants misread as credentials, S101 joining the tracked convert-vs-keep assert batch - apply ruff format across the ported tree (68 files, mechanical) Verified with the CI-exact toolchain (ruff 0.15.12): check and format --check clean over all 96 changed files; the 126-test evolver suite passes; a real `check` run against a live endpoint still passes. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
cyfyifanchen
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port the harness self-evolution framework onto Raven and grow it into a
runnable, benchmark-pluggable stack: a budget-bounded loop that diagnoses
failing trajectories, designs candidate patches as real git commits, screens
them cheaply, confirms survivors, and promotes only what passes three
verification gates -- with a sealed test set for an honest generalisation
number.
raven/evolver/): orchestrator state machine(seven-step funnel), three verification gates (infra validity, beacon
attribution, paired significance), sealed-test runner with retention,
git-backed candidate tree, judge, taxonomy, activation ledger.
benchmarks/appworld/): agent_cli, batch scorerwith trial-level resume and an infra-rerun ladder, plus the evolve glue
(designer, diagnosis, adapter, Gate0 precheck with real-generation probing).
python -m raven.evolver {check,run,status,finalize}drives cold start -> rounds -> terminate -> unseal as a resumable state
machine. Config drift and unseal one-wayness are mechanized in run meta;
benches plug in via a documented contract (
docs/specs/evolve-bench-contract.md).history, gentle decay) so unreachable triggers teach the next attempt.
checkruns the bench Gate0 precheck, so a deadsubject endpoint fails at check time instead of at cold start; ephemeral
eval/edit worktrees live under
work_dir/tmpand hard-kill leftovers areswept on the next launch of the same run.
bench plugin contract, and a user README with security notes and known
limitations.
Type
Verification
Commands run:
uv run pytest tests/test_evolver_launch.py tests/test_evolver_launch_e2e.py -q-> 56 passed (config, guards, trial-level idempotency, interrupt -> resume ->
finalize with a fake bench, check precheck pass/fail, worktree sweep).
uv run ruff checkon all touched files -> no new findings versus base(one S607 partial-path git call matching the file's existing idiom).
written by following the README user path:
check->run --smoke(~16 min:cold start on 3 tasks, one full round with real diagnose/design, a real
1-line candidate commit, focused probe + confirm, promoted with an honest
credited=FalseGate-b verdict) -> termination.checkboth ways: live subject endpoint ->bench precheck: OK;config pointed at a dead (NXDOMAIN) endpoint -> exit 1 at check time with
an actionable
subject endpoint unreachablemessage.benchmarks/, outside the wheel, matchingthis directory's decoupling convention) was verified with a second real
smoke: all three phases ran against a checkout of this layout, worktree
evals resolved
python -m benchmarks.appworld.batchfrom the checkout,and the designer wrote its candidate to the new whitelist path
(
benchmarks/appworld/agent_cli.py) as a real commit.Risk
Security: this system has an LLM edit and execute code. Candidate edits are
constrained to a per-bench path whitelist, an immutable kernel protects the
measurement surface (the evolver itself and the scorer), and the README
documents the threat model plus the recommendation to run evolutions in an
isolated environment. Backward compatibility: purely additive -- new packages
plus one optional
BenchBundle.precheckfield; nothing existing changesbehavior. Rollback: revert the branch; no migrations, no config changes to
existing features.
Related Issues
N/A
Reviewer notes:
hours) and the sealed-test unseal/retention path on a real benchmark are
covered by tests but not yet by a production run; the README states this
under "Status and known limitations".
example (external scorer) is planned as a follow-up.