Add versioned single-runtime observations for Gavel conformance fixtures

[adityathebe]

Context

Captain needs reliable regression and consistency coverage across API, CLI, app-server/agent, provider, model, reasoning-effort, permission, tool, MCP, and sandbox combinations. The same prompt can currently complete through one runtime while silently losing a control or behaving differently through another; #81 is a recent reasoning-effort example.

The existing Claude-specific fixture runner already repeats runs, aggregates metrics, and captures selected MCP/Kubernetes traffic, but matrix orchestration, generic metric aggregation, comparisons, assertions, and the final test verdict belong in Gavel.

Related work:

• Gavel companion: Add repeated fixture samples and generic CEL-extracted metric comparisons gavel#84
• Existing Captain benchmark motivation: Captain bench #7
• Existing capture/matrix sketch: Pull up fixture capturing into prompts or gavel fixture recording #97

This issue supersedes the implementation design in #97 while retaining its provider/model/sandbox matrix and proxy-capture goals. It does not require closing #7 or #97 immediately.

Ownership boundary

Captain owns:

• resolving and executing one requested runtime;
• provider-specific request/dispatch instrumentation;
• terminal execution outcome;
• normalized usage/cost/timing observations;
• permission and tool lifecycle evidence;
• Captain-owned MCP/Kubernetes interception and redacted artifacts;
• honest completeness/unknown/unsupported states.

Gavel owns:

• matrix rows and repeats;
• CEL assertions;
• sample aggregation and metric comparisons;
• baseline/threshold policy;
• the final fixture pass/fail verdict.

Captain must report facts and evidence, never a conformance passed value.

Proposed command contract

Add an additive, machine-oriented action rather than changing the existing human/batch result contract:

captain prompt observe <prompt> --runtime <selector>

The command must:

• resolve exactly one provider/mode/model runtime;
• reject batch/multi-model, fallback-chain, async, and chat execution in v1;
• emit exactly one captain.observation/v1 JSON document to stdout;
• write diagnostics only to stderr;
• include a stable execution outcome and all available observations;
• exit 0 whenever a complete and trustworthy observation document was produced, including an observed runtime failure, unavailable runtime, or unsupported requested capability;
• exit non-zero only when no trustworthy observation can be produced, such as invalid arguments/configuration, recorder/serialization failure, or process crash.

Process success therefore means “the observation protocol succeeded.” execution.state records whether the model run succeeded. Gavel asserts the latter.

A separate action avoids inheriting the current PromptRunResult limitations:

• only coarse input/output usage fields are serialized;
• duration is formatted text rather than a raw number;
• rich tool/permission events are discarded;
• effort reflects selected configuration rather than provider-native dispatch evidence;
• a multi-model batch can return status: partial without a command error.

Ordinary captain prompt run, including its existing JSON and human output, must remain compatible.

Versioned observation schema

Illustrative v1 document:

{
  "schemaVersion": "captain.observation/v1",
  "observationId": "019...",
  "runtime": {
    "requested": {
      "selector": "api:gpt-5.6-sol:high"
    },
    "resolved": {
      "provider": "openai",
      "backend": "openai",
      "mode": "api",
      "model": "gpt-5.6-sol"
    }
  },
  "availability": {
    "state": "available"
  },
  "execution": {
    "state": "completed",
    "durationMs": 1842,
    "error": null
  },
  "controls": {
    "reasoningEffort": {
      "requested": {"state": "known", "value": "high"},
      "resolved": {"state": "known", "value": "high"},
      "observed": {
        "state": "known",
        "value": "high",
        "evidenceRefs": ["dispatch-1"]
      }
    }
  },
  "capture": {
    "dispatch": {
      "status": "complete",
      "events": [{
        "id": "dispatch-1",
        "attempt": 1,
        "boundary": "openai.responses.create"
      }]
    },
    "permissions": {"status": "complete", "events": []},
    "tools": {"status": "complete", "events": []},
    "mcp": {"status": "not_requested", "events": []},
    "kubernetes": {"status": "not_requested", "events": []}
  },
  "metrics": {
    "durationMs": {"state": "known", "value": 1842, "unit": "ms"},
    "costUSD": {
      "state": "known",
      "value": 0.0124,
      "unit": "USD",
      "source": "provider"
    },
    "usage": {
      "state": "known",
      "semantics": "disjoint-v1",
      "buckets": {
        "inputTokens": 423,
        "outputTokens": 92,
        "reasoningTokens": 188,
        "cacheReadTokens": 0,
        "cacheWriteTokens": 0
      }
    }
  },
  "artifacts": []
}

The exact Go shape can remain additive within captain.observation/v1, but the field semantics below are contractual.

Requested, resolved, and observed semantics

Every control stage uses an explicit fact state:

State	Meaning
known	The value is known and value is present; zero/false/empty remain valid values.
unset	Captain positively observed that no value was supplied/emitted.
unknown	Captain cannot determine the value reliably.
unsupported	The resolved runtime explicitly cannot apply the control; include a stable reason code.

Stages:

• requested: literal caller intent before defaults.
• resolved: value after aliases, defaults, model/runtime selection, and translation.
• observed: value captured at the provider-native dispatch boundary.

Rules:

• Never populate observed by copying requested or resolved.
• observed: unset is valid only when complete instrumentation positively proves omission.
• Missing instrumentation must produce unknown, never a false success.
• For API providers, observe the request/options handed to the provider client.
• For CLI modes, observe argv/config at successful process start.
• For app-server/agent modes, observe the provider-native SDK or JSON-RPC initialization/turn request.
• An unsupported requested control should normally prevent dispatch and produce:
  • resolved.state: unsupported;
  • observed.state: unsupported;
  • execution.state: not_started;
  • process exit 0, because the observation is trustworthy.

For example, the existing PromptRunItem.Effort is useful requested/resolved metadata, but it is not evidence that the OpenAI API body or Codex CLI invocation actually carried that effort.

Capture completeness and normalized evidence

Each capture channel has a completeness status:

complete | partial | not_requested | unavailable | unsupported

An empty event list means “zero observed events” only when status is complete. It must not silently mean “not instrumented.”

Normalize stable event fields and allow additive provider-specific detail. Example denied permission/tool lifecycle:

{
  "capture": {
    "permissions": {
      "status": "complete",
      "events": [{
        "id": "permission-1",
        "toolCallId": "call-1",
        "tool": "sentinel.write",
        "decision": "denied",
        "decidedBy": "captain_broker"
      }]
    },
    "tools": {
      "status": "complete",
      "events": [{
        "id": "tool-1",
        "toolCallId": "call-1",
        "name": "sentinel.write",
        "execution": {"state": "not_started"}
      }]
    }
  }
}

Observation mode must wrap the permission callback and record both the request and returned decision. Correlate permission and tool events by tool-call ID.

MCP/Kubernetes evidence should expose only bounded, redacted facts such as method, tool/resource/path, status, timing, correlation ID, and body hash when useful. Do not emit prompts, authorization headers, credentials, raw tokens, or unbounded bodies.

There is no generic arbitrary-HTTP capture promise: Captain can only capture traffic routed through a Captain-controlled interception point.

Implementation plan

Phase C1: contract and one-shot command

Likely files/boundaries:

• Add versioned API types, for example pkg/api/runtime_observation.go.
• Add the action and registration, for example:
  • pkg/cli/prompt_observe.go;
  • pkg/cli/prompt_entity.go.
• Reuse prompt rendering, attachment resolution, runtime resolution, session persistence, and single-run execution internals from pkg/cli/prompt_run.go without routing through the multi-model aggregate result.
• Add an internal context-carried recorder, for example under pkg/ai/observation/, rather than adding fixture concerns to every public provider interface.
• Collect terminal state, raw duration, full disjoint usage buckets, cost source, and existing tool/result events.
• Guarantee one JSON stdout document and stderr-only diagnostics.

Phase C2: first conformance vertical slice

Instrument enough provider-native boundaries to demonstrate the original inconsistency:

1. one OpenAI/API dispatch path for reasoning effort;
2. one Codex/CLI or app-server dispatch path for reasoning effort;
3. brokered permission request/decision plus correlated tool lifecycle.

Likely touch points include:

• API provider invocation under pkg/ai/provider/genkit;
• CLI argument/process builders in pkg/ai/provider/cli.go and provider-specific CLI files;
• Codex app-server initialization/turn parameters;
• Claude agent initialization/turn parameters;
• permission callback construction around Config.CanUseTool;
• event accumulation currently used by pkg/cli/prompt_run_events.go and pkg/cli/prompt_run_live.go.

Use existing capability/model tables to report resolved: unsupported, but never use capability metadata to claim observed: known.

Phase C3: MCP and Kubernetes evidence

Reuse the existing Captain-owned proxy packages and correlation logic under pkg/ai/fixture:

• extract only enough orchestration to make kube/MCP capture available to observation mode;
• keep the proxies and provider-specific rewriting in Captain;
• report unsupported, unavailable, or partial for modes that cannot be routed completely;
• emit bounded normalized events and artifact references rather than embedding unbounded logs.

C3 must not block C1/C2 or the first reasoning/permission matrix.

Phase C4: Captain-hosted fixture corpus and CI rollout

After flanksource/gavel#84 is available:

• keep the evolving provider/mode matrix fixture corpus in Captain;
• pin a released/known Gavel version in Captain CI;
• add at least:
  • one API and one CLI reasoning-effort row;
  • one deterministic permission/tool-denial row;
• start real paid-provider matrices as scheduled/non-blocking jobs until variance, cost, and support expectations are calibrated.

End-to-end conformance examples

Reasoning effort

Gavel should be able to assert:

json.schemaVersion == "captain.observation/v1" &&
json.execution.state == "completed" &&
json.controls.reasoningEffort.requested.state == "known" &&
json.controls.reasoningEffort.requested.value == "high" &&
json.controls.reasoningEffort.resolved.value == "high" &&
json.controls.reasoningEffort.observed.state == "known" &&
json.controls.reasoningEffort.observed.value == "high" &&
size(json.controls.reasoningEffort.observed.evidenceRefs) > 0

A provider that completes and reports reasoning tokens while silently dropping the requested control must fail this assertion while retaining its numeric metric samples.

Permission/tool denial

A deterministic sentinel tool in a disposable workspace should allow Gavel to assert:

json.execution.state == "completed" &&
json.capture.permissions.status == "complete" &&
json.capture.permissions.events.exists(p,
  p.tool == "sentinel.write" && p.decision == "denied") &&
json.capture.tools.events.exists(t,
  t.name == "sentinel.write" && t.execution.state == "not_started")

Captain reports the request, decision, and lifecycle; Gavel decides whether those facts conform.

Acceptance criteria

• One observation invocation resolves no more and no fewer than one runtime.
• Exactly one versioned JSON document is emitted to stdout; diagnostics do not corrupt it.
• Runtime failure/unavailability/unsupported controls still produce trustworthy JSON and process exit 0; observation setup/protocol failure exits non-zero.
• requested, resolved, and observed are distinguishable, and provider-boundary evidence proves observed is not copied from configuration.
• Omitted native control produces observed: unset only with complete dispatch instrumentation; absent instrumentation produces unknown.
• Unsupported controls are reported before dispatch with a stable reason.
• All five disjoint token buckets survive observation output without double-counting.
• Cost includes whether it was provider-reported or Captain-estimated.
• Permission request and returned allow/deny decision correlate with tool execution state.
• Empty complete capture is distinguishable from unavailable/partial capture.
• MCP/Kubernetes events and artifacts are bounded and redact secrets.
• Existing captain prompt run human and JSON output remain unchanged.
• A Gavel fixture from Add repeated fixture samples and generic CEL-extracted metric comparisons gavel#84 can compare API and CLI reasoning effort and run a deterministic permission/tool case without Captain declaring test pass/fail.

Non-goals for v1

• Captain-owned fixture pass/fail or answer-quality grading.
• Batch/multi-model, fallback-chain attribution, retries, chat, or async observation.
• Cartesian matrix expansion.
• Moving MCP/Kubernetes proxies into Gavel.
• Arbitrary host-wide HTTP interception, including arbitrary curl traffic.
• Automatic metric baselines or external benchmark storage.
• Treating unknown, unavailable, or unsupported as an implicit skip; fixtures must state expected support explicitly.
• Immediate removal of the existing Claude-specific pkg/ai/fixture runner.

Risks

• Provider SDKs may hide final native requests. Those paths must report unknown unless instrumented at a trustworthy boundary.
• Capture can leak secrets. Redaction and bounded payloads are contract requirements, not optional presentation behavior.
• Proxy coverage differs by runtime (for example HTTP MCP can be rewritten while stdio MCP cannot); capture completeness must make this explicit.
• Repeated fixtures can create cost and side effects; use deterministic prompts, explicit repeats, and disposable environments.