Skip to content

feat(*): self-evolution stack: evolver, appworld bench, unified entry#140

Merged
cyfyifanchen merged 4 commits into
EverMind-AI:mainfrom
ethanwwwwwwww:feat/port_evolver_appworld_bench
Jul 17, 2026
Merged

feat(*): self-evolution stack: evolver, appworld bench, unified entry#140
cyfyifanchen merged 4 commits into
EverMind-AI:mainfrom
ethanwwwwwwww:feat/port_evolver_appworld_bench

Conversation

@ethanwwwwwwww

@ethanwwwwwwww ethanwwwwwwww commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Port the harness self-evolution framework onto Raven and grow it into a
runnable, benchmark-pluggable stack: a budget-bounded loop that diagnoses
failing trajectories, designs candidate patches as real git commits, screens
them cheaply, confirms survivors, and promotes only what passes three
verification gates -- with a sealed test set for an honest generalisation
number.

  • Evolution machinery (raven/evolver/): orchestrator state machine
    (seven-step funnel), three verification gates (infra validity, beacon
    attribution, paired significance), sealed-test runner with retention,
    git-backed candidate tree, judge, taxonomy, activation ledger.
  • AppWorld bench (benchmarks/appworld/): agent_cli, batch scorer
    with trial-level resume and an infra-rerun ladder, plus the evolve glue
    (designer, diagnosis, adapter, Gate0 precheck with real-generation probing).
  • Unified entry: python -m raven.evolver {check,run,status,finalize}
    drives cold start -> rounds -> terminate -> unseal as a resumable state
    machine. Config drift and unseal one-wayness are mechanized in run meta;
    benches plug in via a documented contract (docs/specs/evolve-bench-contract.md).
  • Designer feedback: inert-death loop (pruned_inert status, per-WHY
    history, gentle decay) so unreachable triggers teach the next attempt.
  • Launch hygiene: check runs the bench Gate0 precheck, so a dead
    subject endpoint fails at check time instead of at cold start; ephemeral
    eval/edit worktrees live under work_dir/tmp and hard-kill leftovers are
    swept on the next launch of the same run.
  • Docs: SOP translation (internal details redacted), SOP-to-code map,
    bench plugin contract, and a user README with security notes and known
    limitations.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Commands run:

  • uv run pytest tests/test_evolver_launch.py tests/test_evolver_launch_e2e.py -q
    -> 56 passed (config, guards, trial-level idempotency, interrupt -> resume ->
    finalize with a fake bench, check precheck pass/fail, worktree sweep).
  • uv run ruff check on all touched files -> no new findings versus base
    (one S607 partial-path git call matching the file's existing idiom).
  • Real AppWorld end-to-end through the unified entry, from a fresh config
    written by following the README user path: check -> run --smoke (~16 min:
    cold start on 3 tasks, one full round with real diagnose/design, a real
    1-line candidate commit, focused probe + confirm, promoted with an honest
    credited=False Gate-b verdict) -> termination.
  • Real check both ways: live subject endpoint -> bench precheck: OK;
    config pointed at a dead (NXDOMAIN) endpoint -> exit 1 at check time with
    an actionable subject endpoint unreachable message.
  • The bench placement (repo-root benchmarks/, outside the wheel, matching
    this directory's decoupling convention) was verified with a second real
    smoke: all three phases ran against a checkout of this layout, worktree
    evals resolved python -m benchmarks.appworld.batch from the checkout,
    and the designer wrote its candidate to the new whitelist path
    (benchmarks/appworld/agent_cli.py) as a real commit.

Risk

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Security: this system has an LLM edit and execute code. Candidate edits are
constrained to a per-bench path whitelist, an immutable kernel protects the
measurement surface (the evolver itself and the scorer), and the README
documents the threat model plus the recommendation to run evolutions in an
isolated environment. Backward compatibility: purely additive -- new packages
plus one optional BenchBundle.precheck field; nothing existing changes
behavior. Rollback: revert the branch; no migrations, no config changes to
existing features.

Related Issues

N/A


Reviewer notes:

  • Exercised at smoke scale only: a full-size run (90 tasks x K=3, multi-round,
    hours) and the sealed-test unseal/retention path on a real benchmark are
    covered by tests but not yet by a production run; the README states this
    under "Status and known limitations".
  • One bench ships today (AppWorld, built-in scorer line); a framework-line
    example (external scorer) is planned as a follow-up.

@ethanwwwwwwww
ethanwwwwwwww force-pushed the feat/port_evolver_appworld_bench branch from c54e60e to 37b6b00 Compare July 17, 2026 03:33
Port the harness self-evolution framework onto Raven and grow it into a
runnable, benchmark-pluggable stack:

- raven/evolver/: the evolution machinery - orchestrator state machine
  (seven-step funnel), three verification gates (infra validity, beacon
  attribution, paired significance), sealed-test runner with retention,
  git-backed candidate tree, judge, taxonomy, activation ledger
- benchmarks/appworld/ (repo-root bench area, out of the wheel): the AppWorld bench (agent_cli, batch
  scorer with trial-level resume, infra-rerun ladder) + evolve glue
  (designer, diagnosis, adapter, precheck with real-generation probing)
- unified entry: python -m raven.evolver {run,check,status,finalize}
  drives cold start -> rounds -> terminate -> unseal as a resumable
  state machine; config drift and unseal one-wayness are mechanized in
  run meta; benches plug in via a documented contract, loaded from the
  subject repo root (bench plugins live outside the installed package)
- designer feedback: inert-death loop (pruned_inert status, per-WHY
  history, gentle decay) so unreachable triggers teach the next attempt
- launch hygiene: check runs the bench Gate0 precheck (a dead subject
  endpoint fails at check time, not at cold start); ephemeral eval/edit
  worktrees live under work_dir/tmp and hard-kill leftovers are swept
  on the next launch of the same run
- docs: SOP translation (internal details redacted), SOP-to-code map,
  bench contract, user README with security notes and known limitations
- verified: 56 unit/e2e tests plus a real AppWorld smoke through the
  unified entry (cold start, resume, one full round, honest Gate-b
  verdict, termination)

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
@ethanwwwwwwww
ethanwwwwwwww force-pushed the feat/port_evolver_appworld_bench branch from 37b6b00 to 7f91bfb Compare July 17, 2026 05:09
Add 70 unit tests over the evolver's decision core and side-effect
boundaries, which were previously covered only transitively by the
fake-bench e2e and real smoke runs:

- gates: paired z statistics against hand-computed vectors (deterministic
  win/regression, sqrt(3) mid case, missing-task-scores-zero), Fisher
  exact hand-checked values and degenerate margins, three-shield pipeline
  narrowing (infra reported but never dropped, Gate-b fail-open vs
  fail-closed), and both gate policies end to end - including the dual
  condition where a fired subset win must not promote a full-train
  regression
- scoring: the SOP 0 infra-rerun ladder (missing task = infra, keep
  fewest-infra measurement, persistent infra survives at K and scores 0)
  and flip_summary accounting
- git_ops: commit_files_as_child leaves the main working tree and HEAD
  untouched, worktree_at cleans up on both exit paths, evolver refs
  anchor candidates against gc, ephemeral-root redirection
- sandbox: whitelist write refusal, origin-repo bash boundary,
  scope_restore reverting only out-of-whitelist edits, exact change and
  deletion capture, import smoke resolving from the edited worktree
- drivers: claude_cli transcript serialization, argv/stdin shape with
  tools off, retry and rate-limit stretch, and the Gate0 endpoint probe
  mapping each failure mode to its actionable message

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
@0xKT
0xKT requested review from 0xKT and arelchan July 17, 2026 07:48
0xKT
0xKT previously approved these changes Jul 17, 2026
The pre-commit CI job lints every file a PR touches, which put the whole
ported evolver/bench tree in scope; make lint-python only covers the repo
hygiene scripts, so these findings never surfaced locally.

- fix real findings: PassFailResult was referenced in an annotation but
  only imported inside the function (F821) - import it at module level;
  drop a dead _run call in git_ops.branch_exists and an unused local in
  tree_loader (F841); rename three ambiguous `l` loop variables (E741)
- register the idiomatic patterns as per-file-ignores in pyproject with
  reasons, following the existing sections: S607 for the git/bash
  subprocess drivers, N803/N806 for K/T as SOP domain vocabulary, S105
  for outcome-label constants misread as credentials, S101 joining the
  tracked convert-vs-keep assert batch
- apply ruff format across the ported tree (68 files, mechanical)

Verified with the CI-exact toolchain (ruff 0.15.12): check and
format --check clean over all 96 changed files; the 126-test evolver
suite passes; a real `check` run against a live endpoint still passes.

Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
@cyfyifanchen
cyfyifanchen merged commit c8f24f2 into EverMind-AI:main Jul 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants