Skip to content

Run the host probe protocol on all four hosts and replace the deferred:not-yet-run records #21

Description

@ivklgn

Rescoped — since this was filed, the protocol, the harness and the records table all landed (commit 5ec9c00, "give the probe protocol a spec, a harness, and honest records"). What did not land is the thing the issue is actually about: running the probes. Every record row still reads deferred:not-yet-run.

Problem

.archcore/plugin/host-adapter-contract.spec.md makes a dated probe record item 6 of what every adapter MUST provide. Four hosts ship (Claude Code, Cursor, Codex CLI, GitHub Copilot CLI) and none has an empirical record. Live-session guarantees — that the host loads the config at all, that its matcher fires on the tool name the model actually chose, that a deny is honored rather than merely displayed, and that writes performed by delegated sub-agents are covered — are asserted, not verified.

Current state of .archcore/plugin/host-probe-protocol.spec.md:

Date Host P0 A A-d B C D
claude-code deferred:not-yet-run
cursor deferred:not-yet-run
codex deferred:not-yet-run
copilot deferred:not-yet-run

What already exists (do not re-build)

  • .archcore/plugin/host-probe-protocol.spec.md (accepted) — six probes (P0, A, A-d, B, C, D), outcome vocabulary, method notation, ten normative rules, append-only records table.
  • test/probe/mkprobe — builds a disposable probe tree: a copy of plugins/archcore with the three launchers wrapped, a sibling probe project, a sentinel version 0.0.0-probe stamped into every manifest, and a tab-separated invocation log.
  • test/structure/probe-records.bats — every enrolled host owes a well-formed row (host list derived from host-coverage-matrix.bats).
  • test/structure/probe-hygiene.bats — no probe residue under plugins/.
  • test/unit/probe-wrapper.bats — the wrapper is transparent (same stdout, stderr, exit status) against every fixture.

Expected result

Run the protocol on each shipped host and append a dated row per run, with the captured log pasted verbatim into the commit body per rule 8. A-d — the delegated-write probe — is the one that answers the original question about Task-dispatched sub-agent coverage; record n/a:no-delegation-surface (<host> <ver>) where the host offers none.

Per rule 9, any host stdin captured during a run must be added to test/fixtures/stdin/<host>/ so the payload shape becomes a CI assertion. Only if a payload differs from what bin/lib/normalize-stdin.sh expects does this issue grow a code change.

Impact

Either confirms four adapters' coverage with evidence, or surfaces a real gap. A fail on A-d is also the documented trigger to revisit sub-agent snapshot injection (#17). Low effort per host, high assurance.

Proposed approach

Claude Code first (richest evidence grade), then Copilot (structurally the most divergent adapter, and the one with a documented extra P0 failure mode), then Codex and Cursor.

  • claude-code — run P0→D, append row, paste log in the commit body.
  • copilot — same. Note: archcore: plugin root unresolved at P0 is not a coverage gap; per the protocol, do not record a row for that run.
  • codex — same.
  • cursor — same. Rule 5 applies: additional_context may be dropped, so an agent's claim it saw context grades report, never log.
  • Only if a payload shape differs: extend bin/lib/normalize-stdin.sh and pin it with a bats case.
  • Drive-by: the spec's Surface table still names the pre-v0.7.0 scripts (bin/check-code-alignment, bin/check-archcore-write, bin/validate-archcore) in its "Guard exercised" column, while mkprobe correctly wraps the three launchers. Fix the column to bin/{session-start,pre-tool-use,post-tool-use}.

Acceptance criteria

  • No deferred:not-yet-run cell remains for any enrolled host; every outcome is pass, fail, or a reasoned n/a:.
  • Each row's evidence pointer resolves to a commit whose body carries the verbatim log.
  • probe-records.bats and probe-hygiene.bats stay green.

References

.archcore/plugin/host-probe-protocol.spec.md, host-adapter-contract.spec.md (item 6), jtbd1-phase2-hardening-delegated.plan.md. Harness: test/probe/mkprobe. Verification chore — the spec forbids committing probe lines into plugins/.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:hooksPreToolUse/PostToolUse/SessionStart hooksarea:testsTest suite (bats fixtures/harness)priority:P2Valuable follow-upsize:SA few hours

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions