Add repeated fixture samples and generic CEL-extracted metric comparisons

[adityathebe]

Context

Captain needs a provider/model/mode matrix that can detect both behavioral inconsistencies (reasoning effort, permissions, tools, MCP) and numeric regressions (duration, cost, token usage). Gavel already provides Markdown fixtures, command execution, JSON-aware CEL assertions, result trees, and Go benchmark statistics, but every fixture row currently executes exactly once and fixture results have no generic metric samples or comparisons.

This issue adds the generic Gavel half of the design. Gavel must remain command- and JSON-oriented: it must not import Captain types or hard-code Captain JSON paths.

Related Captain work:

• Captain bench captain#7
• Pull up fixture capturing into prompts or gavel fixture recording captain#97
• Companion observation contract: Add versioned single-runtime observations for Gavel conformance fixtures captain#99

Ownership boundary

Gavel owns:

• explicit matrix rows;
• repeated execution and per-sample evidence;
• CEL assertions;
• numeric extraction and aggregation;
• absolute thresholds and row-to-row baseline comparisons;
• the final fixture pass/fail/error verdict.

The command under test owns execution and structured observations. For Captain, one Gavel row will invoke exactly one provider/mode/model runtime and receive one versioned JSON observation.

Proposed fixture schema

Keep explicit Markdown rows as the matrix mechanism in v1. Add file/command frontmatter defaults and a known table Repeat override:

---
exec: captain
args:
  - prompt
  - observe
  - "{{ .prompt }}"
  - --runtime
  - "{{ .selector }}"
repeat: 5
timeout: 20m

metrics:
  - name: reasoning_tokens
    extract: json.metrics.usage.buckets.reasoningTokens
    unit: tokens
    aggregate: median
    direction: none

  - name: duration_ms
    extract: json.metrics.durationMs.value
    unit: ms
    aggregate: median
    direction: lower
    baseline: API high
    threshold:
      regressionPercent: 25
      max: 120000
---

Example matrix:

| Name | selector | prompt | Repeat | CEL Validation |
|---|---|---|---:|---|
| API high | api:gpt-5.6-sol:high | testdata/reasoning.prompt | 5 | json.execution.state == "completed" && json.controls.reasoningEffort.observed.state == "known" && json.controls.reasoningEffort.observed.value == "high" |
| CLI high | cli:gpt-5.6-sol:high | testdata/reasoning.prompt | 5 | json.execution.state == "completed" && json.controls.reasoningEffort.observed.state == "known" && json.controls.reasoningEffort.observed.value == "high" |

Schema semantics

• repeat defaults to 1; a row value overrides command/file frontmatter.
• extract is CEL evaluated against each sample's existing context: json, stdout, stderr, exitCode, temporary files, and fixture properties.
• Extraction must return one finite numeric value. Missing/null, boolean, string, NaN, and infinity are metric errors.
• unit is required and is used as a label and compatibility check; v1 performs no unit conversion.
• aggregate supports mean (default), median, min, max, and p95.
• direction is lower, higher, or none.
• threshold.min and threshold.max apply to the selected aggregate.
• baseline names another logical fixture row in the same run.
• threshold.regressionPercent is the maximum allowed worsening relative to the baseline:
  • lower is better: (current - baseline) / abs(baseline) * 100;
  • higher is better: (baseline - current) / abs(baseline) * 100.
• Relative comparison to a zero baseline is an error; use an absolute threshold instead.
• Baseline row names must be unique. Metric name, unit, aggregate, and direction must match before comparison.

Sample and outcome model

Keep one test node per Markdown row; repeats must not inflate test counts.

Add additive result data equivalent to:

FixtureResult
  Samples[]
    index, duration, exit code, stdout, stderr, error
    command expectation result
    CEL result/error
    extracted metric values
  Metrics{}
    raw valid samples
    selected aggregate
    baseline/comparison
    metric status/error
  Outcomes
    command
    assertions
    metrics

Avoid recursively storing FixtureResult inside each sample.

Evaluation and failure semantics

1. Execute base command and stream/exit expectations for every sample.
2. If command expectations fail, CEL and metric extraction for that sample are not_evaluated.
3. If command expectations pass, evaluate CEL and metric extraction independently from the same parsed context.
4. A false CEL assertion must not discard otherwise valid metric evidence.
5. Valid metric values may still be summarized for diagnostics, but thresholds/baseline comparisons are valid only when every required sample produced a value.
6. Continue after an ordinary sample FAIL to expose stochastic reliability. Stop after ERR, cancellation, or exhaustion of the logical-row timeout.
7. Existing CEL must pass for every evaluated sample.
8. Final precedence:
   • extraction/configuration/CEL runtime error: ERR;
   • any command, CEL, absolute threshold, or regression failure: FAIL;
   • otherwise: PASS.
9. The fixture timeout is a total budget shared by all repeats, not silently multiplied per sample.
10. OnResult is called once per logical row with the finalized, comparison-aware result.

No aggregate-level CEL lifecycle is needed in v1; aggregate conditions use metric thresholds.

Implementation plan

Phase G1: schema, parsing, and validation

Likely files:

• fixtures/types.go
  • add repeat/metric configuration;
  • add non-recursive sample, metric summary/comparison, and outcome result types;
  • clean the new known frontmatter keys from inline metadata.
• fixtures/parser.go / fixtures/parser_ast.go
  • parse file and command-block metric configuration;
  • recognize the Repeat table column as a typed override rather than a custom property.
• Add pre-execution validation for duplicate names, invalid aggregates/directions, invalid thresholds, and missing/ambiguous baselines.

Phase G2: shared sample evaluation and metric extraction

Likely files:

• Refactor fixtures/expectations.go so JSON is decoded once whenever CEL or metrics require it, while preserving the existing json CEL variable and metadata.json behavior.
• Adjust fixtures/types/exec.go to retain process evidence separately from the final logical-row verdict.
• Add fixtures/metrics.go for numeric CEL extraction, finite-number validation, aggregates, absolute limits, and direction-aware comparisons.

Phase G3: repeat orchestration and finalization

Likely files:

• fixtures/runner.go
  • execute samples serially within one row task;
  • enforce one total row timeout;
  • keep a single tree node/result per row;
  • finalize cross-row baseline comparisons after all row tasks complete;
  • update tree statistics and invoke callbacks only with finalized results.
• Extend FixtureResult.Pretty/Clicky rendering and JSON output with samples and metric summaries without making ordinary fixture output noisy.

Statistics decision

Do not reuse testrunner/bench.BenchRun or bench.Compare directly. They encode Go benchmark assumptions: ns/op formatting, lower-is-better, significance tests, and geometric means.

For v1, keep fixture metric types and the small aggregate set in fixtures/metrics.go. If another consumer later needs the same pure algorithms, extract only those algorithms to a neutral internal/stats package and leave compatibility wrappers in testrunner/bench.

Do not add a shared benchmark/metric result hierarchy now.

Acceptance criteria

• repeat: 3 invokes a command exactly three times, stores three samples, counts one fixture row, and calls OnResult once.
• File/command frontmatter repeat works and a table Repeat value overrides it.
• Existing fixtures without repeat/metrics preserve current behavior; new serialized fields are additive and omitted when empty.
• JSON is parsed for metric extraction even when a fixture has no CEL assertion.
• CEL is evaluated per eligible sample and every evaluated sample must pass.
• Metric extraction preserves raw sample values and reports mean, median, min, max, and p95 correctly.
• Absolute min/max thresholds and both direction-aware regression formulas work.
• Missing/duplicate/zero baselines and metric unit/spec mismatches produce actionable errors.
• CEL failures and metric regressions are reported separately and both affect final status.
• The existing fixture timeout is enforced as the total repeat budget instead of the current hard-coded row timeout.
• A shell command producing a Captain-shaped JSON document proves the integration without importing Captain.

Non-goals for v1

• Cartesian-product matrix expansion; use explicit table rows.
• Captain-specific fixture types or JSON paths.
• Model-answer grading.
• Significance tests, geometric means, warmups, retries, outlier removal, or unit conversion.
• Automatic baseline recording/storage or an external benchmark database.
• Treating unknown/unavailable observations as an implicit skip.
• Arbitrary host-wide HTTP interception.

Risks

• Repeats can multiply paid-provider cost and side effects; fixtures must choose deterministic prompts, explicit repeat counts, and disposable environments.
• Real-provider timing/cost is noisy; initial paid-provider matrices should be scheduled/non-blocking until thresholds are calibrated.
• Storing every stdout/stderr sample can increase report size; keep existing output truncation/display controls and avoid embedding large artifacts in metric summaries.