Skip to content

Commit 54d2ce5

Browse files
committed
docs(eval): record the fixed track order as a timing confound
1 parent 2a5b408 commit 54d2ce5

1 file changed

Lines changed: 14 additions & 4 deletions

File tree

‎eval/skill/README.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Chromium** — not what the agent said it did.
1010
```bash
1111
pnpm eval:skill --agent oracle --track all # reference solutions (must be 100%)
1212
pnpm eval:skill --agent claude --track polycss,polycss-noskill # the real control
13-
pnpm eval:skill --agent claude --track all # one agent, both tracks, all tasks
13+
pnpm eval:skill --agent claude --track all # one agent, all three tracks
1414
pnpm eval:skill --agent all --track all --keep --json out.json
1515
pnpm eval:skill --task 03-cube-with-shadow --agent claude,codex
1616
pnpm eval:skill --agent claude --reuse --run r1 # re-grade only; never calls an agent
@@ -40,6 +40,13 @@ For each (agent, task) pair:
4040
| `polycss-noskill` | The **control**. Same library, task and contract; skill withheld. `polycss` minus this is the skill's effect. |
4141
| `three` | An external **baseline**, not a control. |
4242

43+
Track order is fixed: skill-less tracks always run first, so no control ever
44+
executes while an installed skill exists on disk. That buys isolation at the
45+
cost of confounding *timing* — every control runs before its intervention, so
46+
cache warming, rate limits and provider variation are inseparable from any
47+
duration difference. Treat wall-clock deltas as a fixed-order observation, not
48+
an effect of the skill. Only the per-check scores are comparable.
49+
4350
The Three.js track changes the library, its API and its authoring contract at
4451
the same time as it removes the skill, so a polycss-vs-three delta conflates the
4552
skill with how familiar and how costly each library is for that model. It still
@@ -143,9 +150,12 @@ failed to say something, said it somewhere the agent did not look, or the agent
143150
ignored it. The first two are fixable in `packages/skills/skill/`.
144151

145152
Use `--keep` to leave the workspaces under `$TMPDIR/polycss-skill-eval/<run>`
146-
and read what the agent actually wrote. Runs are namespaced by `--run <id>`
147-
(default `current`) so a later run cannot wipe evidence you kept, and
148-
concurrent runs do not collide.
153+
and read what the agent actually wrote. Runs are namespaced by `--run <id>` so a later
154+
run cannot wipe evidence you kept, and concurrent runs do not collide.
155+
156+
Run ids are random by default (`run-a1b2c3d4`) and printed at the top of the
157+
run, so nothing can overwrite anything; `--reuse` therefore requires an explicit
158+
`--run <id>`.
149159

150160
`--reuse` re-grades a kept run and **never invokes an agent**: a case with no
151161
`scene.mjs` is skipped and reported, not re-run. That is how to iterate on a

0 commit comments

Comments
 (0)