Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Changelog

## 0.12.0 — Unreleased

- Add read-only preflight and task-owned Claude/Codex capture inspection.
- Separate configuration and execution evidence in explicit routing schema v3.
- Bind qualification to objective and detect host/alias/evidence drift.
- Document desktop capture limitations without treating requests as observation.

## 0.11.0 — Unreleased

- Add offline, scoped subscription-usage normalization with replay and retry accounting.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Guildhall

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Plugin](https://img.shields.io/badge/plugin-v0.11.0-green.svg)](plugin/README.md)
[![Plugin](https://img.shields.io/badge/plugin-v0.12.0-green.svg)](plugin/README.md)

**Give a coding task to a team of AI specialists that writes tests, implements the
change, reviews it and prepares a PR draft.**
Expand All @@ -20,7 +20,7 @@ code, tests, review evidence and a PR draft. Publishing a PR is a separate actio
## Install

The commands below install Guildhall from its published main branch. **Version
0.11.0 adds installed routing documentation, evaluation and privacy fixes.**
0.12.0 adds installed routing documentation, evaluation and privacy fixes.**
Choose one route for your coding app. No repository clone, build or IDD
installation is required.

Expand Down
2 changes: 1 addition & 1 deletion docs/contributing-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ workers or claim synthetic savings as observed results. See the

## Distribution and versioning

The usage-contract release is 0.11.0; increment all three manifests for further
The usage-contract release is 0.12.0; increment all three manifests for further
installer-visible changes. Existing native Claude agents/commands/hooks remain
preserved. Never rewrite historical plans to claim newer evidence. No repository
change implicitly installs personally, publishes a release or edits the separate
Expand Down
28 changes: 28 additions & 0 deletions docs/plans/2026-09-22-attribution-decision.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Decision: explicit host evidence levels

The user approved the issue-resolution plan, including a separately qualified,
explicit configuration-evidence lane. Routing schema3 implements that proposal.
Execution-observed remains the setup default; configuration_verified must be
selected explicitly and changes the activation hash. Neither a helper report nor
this implementation approval activates routing in any project.

Version1/2 policies and hashes remain unchanged. No existing verified flag is
silently migrated. V3 qualification binds its objective and evidence requirement,
plus existing profile, host, report, revision and expiry checks. Concrete model
and effort observations stay separate. Missing nonnull requested effort prevents
strong qualification. Configuration-based qualification makes no served-model claim.

Native-Claude transcript and Codex app-server capture adapters only inspect
explicitly supplied task-owned records. The Codex desktop collaboration tool is
not presumed to expose app-server capture. If unsupported, preflight reports that
limitation instead of changing host/worker tool or relaxing the policy silently.

Local inspection on 2026-09-22 found Codex.app binary0.142.0-alpha.1 and running
ChatGPT.app Codex runtime0.154.0-alpha.6.2. Their schemas were generated without
model calls into temporary directories. The latter Thread schema labels model
and effort configuration; completions omit executed identity. The issue's
0.155 desktop environment was not reproduced. No personal logs were searched.

This is a capability and record-validation implementation, not a live adaptive
qualification. Independently reviewed records and quality measurements remain
required for each enabled host/profile/role/category.
22 changes: 22 additions & 0 deletions docs/reviews/2026-09-22-routing-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,25 @@ Receipts: `guildhall-install-5cvv_j4o/report.json` (native+pinned) and
`guildhall-install-s0r7n7dj/report.json` (skills1.7.0). Synthetic usage example gives
120 host tokens, null monetary cost and unknown router usage. No quota conversion
or real savings is claimed. PR #38's four GitHub checks passed, including Python3.12.

## Host evidence — 0.12.0

Addresses #36 and #37 finding4. Added read-only preflight, bounded task-owned
Claude transcript and Codex app-server capture inspection, explicit evidence
levels in routing schema3, and alias/configuration/evidence-loss suspension.
The decision record documents the user-approved configuration lane and retains
execution-observed as setup default. No project was activated by this work.

104 tests pass, including mixed/missing responses, wrong-worker/session rejection,
interruption, repeated telemetry, rerouting, alias drift, objective binding and
explicit stronger/weaker qualification. Existing v1/v2 tests remain intact.
Local Codex schemas were generated and inspected without model calls. Running
ChatGPT-owned runtime0.154.0-alpha.6.2 and separately installed Codex.app0.142.0-alpha.1
are distinguished; neither certifies issue #36's different0.155 build. The desktop
collaboration interface does not itself expose the required capture. That route
remains unsupported for this adapter unless task-owned capture access is established.

Ten isolated installation checks passed with both evidence collectors executing
synthetic inputs after source removal. Receipts are retained in the OS temp
installation directories. No personal transcript search, provider probe or live
adaptive qualification occurred. PR #39's four hosted checks passed.
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "guildhall",
"version": "0.11.0",
"version": "0.12.0",
"description": "The Guildhall — a gathering place for adventurers. A TDD-ordered coding agent harness for Claude Code, tuned for Opus-tier orchestration (Opus 5 recommended seat). The /quest slash command runs Mordain the Guildmaster, who writes a durable plan file, then dispatches 18 specialist adventurers across three tiers: Opus (architecture-reviewer, security-reviewer, reliability-reviewer, migration-safety-reviewer), Sonnet (test-author, feature-implementer, ui-test-author, docs-writer, pr-author, prototype-builder, debug-investigator, observability-reviewer, performance-reviewer, ops-readiness-reviewer, accessibility-reviewer), and Haiku (refactorer, plugin-validator, fog-cartographer). Post-green reviews fan out in parallel — two always-on (security, docs) plus six gated production-readiness reviewers (observability, reliability, performance, ops-readiness, migration-safety, accessibility) that fire only when their trigger matches the diff. Rook (pr-author) closes the quest with a platform-agnostic PR draft and folds the runbook into the body. Integrates with IDD-framework specs.",
"author": { "name": "GrillerGeek" },
"homepage": "https://github.com/GrillerGeek/guildhall",
Expand Down
2 changes: 1 addition & 1 deletion plugin/.codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "guildhall",
"version": "0.11.0",
"version": "0.12.0",
"author": {
"name": "GrillerGeek"
},
Expand Down
6 changes: 3 additions & 3 deletions plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ Full character sheets in [`CHARACTERS.md`](CHARACTERS.md).

## Installation

Version **0.11.0 includes the installed routing tools**. The [installation guide](https://github.com/GrillerGeek/guildhall/blob/main/docs/installation.md)
Version **0.12.0 includes the installed routing tools**. The [installation guide](https://github.com/GrillerGeek/guildhall/blob/main/docs/installation.md)
covers this repository's Codex, Claude and standalone routes, updates and removal.
The `main` route receives 0.11.0 after merge; the separate marketplace below is
The `main` route receives 0.12.0 after merge; the separate marketplace below is
not updated by this change.

### From the marketplace (recommended)
Expand Down Expand Up @@ -172,7 +172,7 @@ change reviewers, permissions, lifecycle gates or retry budgets.

## Portable quest candidate

Version 0.11.0 ships `skills/guildhall-quest/SKILL.md` for capable
Version 0.12.0 ships `skills/guildhall-quest/SKILL.md` for capable
non-Claude hosts. It bundles its role references and uses host-native independent
workers. The Claude command/agent/hook route documented above is preserved;
standalone installation does not register those native Claude components.
Expand Down
2 changes: 2 additions & 0 deletions plugin/commands/quest.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,3 +401,5 @@ The tone is a Guildmaster's fireside account, not a machine's log. Keep it truth
- If an adventurer returns "I can't complete this because the spec is ambiguous" or "I need to read implementation code" (Seraphine only), STOP. Route to the user or the relevant IDD agent. Do not dispatch a different adventurer to work around the blocker.
- If an `Oriana` finding is `high` severity, STOP before Rook. Surface to the user.
- If the token cost of a quest is exceeding your rough expectation (e.g., more than 3× a comparable quest), report back to the user mid-flow. Cost awareness is part of the contract.

For schema-v3 routing, read `${CLAUDE_PLUGIN_ROOT}/skills/guildhall-quest/references/host-evidence.md`. Run preflight before paid studies; compare reviewed observations after each worker and suspend on drift without replay.
2 changes: 1 addition & 1 deletion plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "guildhall",
"version": "0.11.0",
"version": "0.12.0",
"author": {
"name": "GrillerGeek"
},
Expand Down
146 changes: 146 additions & 0 deletions plugin/portable/references/host-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# Host evidence and preflight

Version 0.12.0 separates requested settings, host-recorded configuration and
host-reported execution identity. These are different guarantees. A worker's own
answer is never an evidence source. Evidence helpers are read-only: they inspect
explicitly supplied task-owned files, do not search your sessions, run models,
change host settings, or qualify profiles automatically.

## Check before paying for a study

Use the [preflight template](../resources/examples/preflight.json) with the actual
host application's executable/build, worker tool and capability facts. Confirm
identity through an active-process association or a host handshake; a convenient
`codex --version` elsewhere on PATH is not sufficient. Mark uncertain association
`unconfirmed`. The helper diagnoses the supplied facts; it does not authenticate
them or discover processes itself.

```sh
python3 "$GUILDHALL_SKILL_ROOT/scripts/routing_evidence.py" < preflight.json
```

`record_access` is `none`, `claude-owned-transcript` or `codex-owned-jsonrpc`.
The report identifies missing runtime, controls, capture access, review and study
requirements. A candidate evidence lane is a possible next step, not activation.
The default available modes remain off and shadow pending evidence review.
Preflight works without an API key or live probe.

## Read a task-owned capture

The examples are deliberately synthetic: [Claude](../resources/examples/claude-capture.json)
and [Codex](../resources/examples/codex-capture.json). Supply actual host identity,
worker/session IDs, requested settings, complete turn/attempt/response inventory,
and records. Missing or incomplete inventory cannot establish attribution.

```sh
python3 "$GUILDHALL_SKILL_ROOT/scripts/routing_evidence.py" < capture.json
```

For an existing JSONL trace, prepare the same context without its `records` field:

```sh
python3 "$GUILDHALL_SKILL_ROOT/scripts/routing_evidence.py" \
--context reviewed-context.json --records task-owned-trace.jsonl
```

Inputs are bounded to 16 MiB and 100,000 records. No raw prompts, content blocks,
transcripts or credentials appear in the output. Keep capture/report hashes and
review the underlying local evidence before using them. Synthetic reports never
advertise adaptive availability. `qualification` always remains false.

## Native Claude

Use the exact subagent transcript identified by the host, not the parent's final
text. The [Claude hook reference](https://code.claude.com/docs/en/hooks) describes
subagent transcript paths; [subagent documentation](https://code.claude.com/docs/en/sub-agents)
describes their lifecycle. The pilot in issue #37 reports `message.model` as a
concrete executed ID. This adapter treats it as host-reported observation, not
independent provider attestation.

`claude-transcript-v1` reads assistant records with matching `agentId` and
`sessionId`, `message.id`, `message.model` and `message.usage`. A single bounded
turn can use the context's turn ID; multi-turn captures must provide explicit
`turn_id` correlation. Every expected response must be present, with one
consistent model across the assignment. Missing/mixed identities or incomplete
turns yield unknown evidence. Effort remains unknown unless a supported source
is added; model identity alone does not prove effort.

Repeated message IDs are normalized once, with monotonic usage updates. Cache
read/write inputs are separate from uncached input for this adapter. Incompatible
counter shapes fail rather than inventing consumption. This field profile has
synthetic regression coverage; revalidate it against the actual client version
before qualifying a deployment. This release does not assert a new live Claude
qualification run.

## Codex

The supported `codex-app-server-v2` input is a task-owned, complete JSON-RPC
capture with both outgoing requests and incoming responses/notifications. Pair
request IDs for `thread/start` and `turn/start`, correlate the acknowledged
worker/session/turn, and use `turn/completed` agent-message IDs and statuses.
The starting thread must have no prior turns. Unacknowledged model/effort
overrides, missing records or reported substitutions make attribution unknown.

The acknowledged thread configuration is **configuration_verified**, never
observed execution identity. Thread usage totals are cumulative; normalize
monotonic per-turn differences for sequential turns instead of summing snapshots.
Reject interleaved counter streams that cannot be attributed safely. Unknown
billing stays unknown. Response item IDs are not described as provider response IDs.

[Codex app-server documentation](https://learn.chatgpt.com/docs/app-server)
describes version-specific schema generation, usage updates and reroute events.
Inspected local runtime 0.154.0-alpha.6.2 explicitly describes Thread.model and
Thread.reasoningEffort as configuration rather than per-turn telemetry.
Its completion schema has no executed model field. `model/rerouted` exposes
`highRiskCyberActivity`; its absence does not prove that no substitution occurred.
This is evidence about that inspected build, not certification of the different
0.155.0-alpha.9.2 desktop build reported in issue #36.

The current `collaboration.spawn_agent` result does not itself supply a complete
app-server capture. If your desktop provides no supported task-owned capture,
use `record_access: none`: adaptive qualification remains unavailable through
this adapter. Do not silently replace desktop workers with CLI sessions. A future
adapter may use reviewed native session records after their actual build-specific
contract is established; this release does not infer one from private filenames.

## Explicit qualification lanes (routing schema v3)

V1 and V2 policies retain their existing attribution behavior. V3 adds
`policy.required_evidence` and `host.evidence_level`; the required level defaults
to **execution_observed in setup**, and the schema requires an explicit value.
Qualification adds `evidence_level`, `observed_model`, `observed_effort` and
`objective`, in addition to all prior hashes, host revision, scope and expiry.

- `execution_observed`: a reviewed concrete model observation is mandatory.
Nonnull requested effort additionally requires matching observed effort.
- `configuration_verified`: a separately reviewed study may qualify effective
host configuration, while served model/effort remains unknown. The user must
explicitly select this weaker requirement and activate the changed policy.

Both lanes require actual model controls, fresh independent workers, reviewed
capture/report hashes, per-role quality evidence, matching objective and profile
hashes, unexpired scope, and the existing router identity checks. Changing the
objective invalidates V3 qualification. Reading a report does not add its hash to
approved evidence automatically. Do not label a configuration study as a
served-model benchmark or infer that every Codex host supports this lane.

See [policy v3](../resources/schemas/policy-v3.schema.json) and
[request v3](../resources/schemas/request-v3.schema.json). Migration is explicit:
set both versions to 3, keep V2 scoped measurements, select the evidence requirement,
clear old qualifications, obtain matching review and renew activation. Never
reinterpret legacy `verified` as one of the new levels automatically.

## Detect drift after execution

Record the alias requested and the concrete ID observed when available. Compare
new evidence with the previously reviewed report for that profile. The importable
`drift(previous, current)` returns suspension reasons for changed host/configuration,
lost evidence or changed observed model/effort, and always sets `replay: false`.
The coordinator carries `adaptive_suspended: true` to future routing decisions.
Preserve existing workers and partial work. Reevaluate and reactivate explicitly.

If identity only arrives after execution, no pre-dispatch guarantee against alias
drift exists. Keep that limitation visible. For a host without suitable telemetry,
an upstream request should identify the actual build, worker/turn/response IDs,
the missing executed model/effort fields, substitution coverage and a minimal
redacted schema/capture example. Do not attach private transcript content.
3 changes: 3 additions & 0 deletions plugin/portable/references/hosts.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,3 +91,6 @@ uses the same generated helper through its plugin root. Never select a different
skill or modify host configuration to obtain controls. Unknown runtime/credential
or unsupported controls uses an eligible baseline or holds, according to the
shared contract. Absent/off policy skips Python entirely.

Use [host preflight and evidence](host-evidence.md) for optional schema-v3 routing;
retain worker scopes, gates, state and explicit activation.
8 changes: 8 additions & 0 deletions plugin/portable/references/model-routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,3 +279,11 @@ basis and scope, and always leaves cost unknown. Jev calls do not necessarily
consume the host subscription quota. No usage report qualifies or activates a
profile. A collector must not combine incompatible token definitions under one
scope or omit interrupted/retried attempts from the expected inventory.


## Host preflight and evidence

Use the bundled [host evidence guide](host-evidence.md) before benchmarking.
It explains native-Claude observations, Codex configuration evidence, explicit
qualification lanes, capture import and alias drift. Missing telemetry is shown
before users spend on a qualification study.
3 changes: 3 additions & 0 deletions plugin/portable/references/quest.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,3 +199,6 @@ Report in Mordain's chronicle voice with concrete artifacts, actual verification
selected/skipped reviewers, limitations, partial work, open items and applicable
recovery decisions. Never report completed independence, enforcement or model
routing solely from a role's own assertion.

Use [host preflight and evidence](host-evidence.md) for optional schema-v3 routing;
retain worker scopes, gates, state and explicit activation.
14 changes: 14 additions & 0 deletions plugin/portable/references/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,3 +234,17 @@ task category; absent observations remain unknown. Duplicate scopes are rejected
Changing policy version or measurements requires renewed activation and reviewed
qualification. The installed [guide](model-routing.md#subscription-efficiency-0110)
explains subscription setup and the bundled usage normalizer.


## Host evidence schema v3

Follow the [host evidence guide](host-evidence.md) and run preflight before any
paid study. V3 keeps scoped metrics and adds explicit required/available evidence
levels. Setup defaults to execution_observed; configuration_verified is a
separately reviewed opt-in and never relabels requested values as observation.
Before approval, inspect task-owned capture and quality reports, not just their
hashes. After each completed worker, compare profile evidence with `drift()`;
carry suspension forward on host/configuration or observed identity changes.
Preserve partial work and never replay an uncertain dispatch. In the configuration
lane, missing served identity is expected; loss of required configuration evidence
still suspends routing. Do not fabricate observed model or effort in either lane.
Loading
Loading