Skip to content

feat(gate): enforce the method contract — a selected method could go undelivered with every gate green - #5

Closed
MyAlterLego wants to merge 1 commit into
mainfrom
feat/method-contract-gate
Closed

MyAlterLego wants to merge 1 commit into
mainfrom
feat/method-contract-gate

Conversation

@MyAlterLego

Copy link
Copy Markdown
Contributor

Closes the hole that let a selected method go undelivered while every gate passed green.

The hole

The intake asks which analysis to run and writes the answer to model.json as scope.methods. Nothing read that field:

$ grep -c methods Tools/{ScopeGate,VerifyGate,DiscoveryGate,ComposeChains,Prioritize}.ts
0 0 0 0 0

The only STRIDE string in RenderReport.ts is boilerplate prose in the §0 primer. So "STPA + STRIDE" was an unenforced promise.

How it failed in practice

A full-surface run against a ~3,600-file TypeScript monorepo recorded STPA + STRIDE, never executed the STRIDE pass, and instead grew a hand-written nine-row table at the end of 05-constraints.md — which the scope prose then described as "a STRIDE per-element sweep was run". Every gate passed. Scope contract honoured, plan complete, composition rated, report rendered.

The requester found it by reading the finished report and asking where STRIDE was. That is the one place a method gap should never be discovered first.

This is the Step-4 lesson repeating: the workflow said to do it, the workflow was not enough, and it only stopped being skipped when RenderReport began exiting 5 without 04-scenarios.md. A method selection that no gate checks is a control action with no feedback channel — the defect class this toolkit exists to find.

What the gate checks

check why
1 CONTRACT — scope.methods recorded at all Absent fails rather than assuming STPA-only. That assumption is the hole.
2 DELIVERY — 00-stride-seed.md exists with the five sections A table folded into another document doesn't count; the artifact is what the renderer shows and what the next cycle diffs.
3 RECONCILE — cross-check in both directions STRIDE→STPA is the half people write. STPA→STRIDE names the emergent findings STRIDE could not produce, and is the half that gets dropped.
4 PROVENANCE — how the pass was produced is declared A hand-run sweep is legitimate — SeedWithStride.md sanctions it for code-only targets — but must not silently read as a Fabric-pattern run. Same principle as VerifyGate recording which model reviewed the findings.
5 NO UPGRADE — STRIDE-only + populated grid The run quietly upgraded itself to STPA, which Start.md forbids.

Exit 8 (1, 2, 4, 5, 6, 7 taken; VerifyGate uses 3).

Also included

  • RenderReport grades 00-stride-seed.md core only when the contract names STRIDE. Grading it core unconditionally would fail every STPA-only run; leaving it merely supporting is how the miss happened. The contract decides — same principle as scope.requested deciding whether an unmodeled control action is a deferral or a shortfall.
  • Adds the §7b section. The workflow already mandates 00-stride-seed.md and the renderer had no slot for it, so the file was written and silently ignored.
  • stride-scorecard.json, so the report states method coverage from computed data rather than narration.
  • stpa stride [dir] [--warn-only], wired into stpa run immediately after ScopeGate — both are contract gates.

Verification

Six cases, all as expected:

case exit
no method contract recorded 8
STRIDE selected, never delivered 8
seed missing 4 of 5 sections 8
reconciliation one-directional 8
STRIDE-only silently upgraded to STPA 8
STPA-only — must not demand a STRIDE artifact 0

Plus --warn-only forces 0, and the renderer exits 5 / 0 with the seed absent / present.

On a real analysis it reports:

method contract honoured — STRIDE delivered: 26 threats · 23 mapped to UCAs ·
4 STRIDE-only · 1 out of scope · reconciled both directions.

One bug found by running it against real data: the threat-row matcher |\s*ST-\d+ also matched the reconciliation table's rows, which open with an id followed by prose in the same cell (| ST-1 plaintext :80 forward |), producing four false positives. It now requires the id to be its own cell. The first draft also compared bucket sums to detect undisposed rows, which double-counts a row that is both NEW and references a CA-n; it now asks the question directly and names the offending rows.

Notes

  • Branched from main, so this is independent of fix: compose gate read the wrong directory; report CSS escape emitted the wrong glyph #4. Both touch RenderReport.ts but in different regions (~289 / ~1219 here vs 1009 there), so they merge cleanly in either order.
  • No dependencies added.
  • A fresh stpa new scaffold does not record methods, so it fails this gate — deliberately, and exactly as it already fails ScopeGate for having no scope.requested. Both are intake's job to fill.

The intake asks which analysis to run and writes the answer to model.json as
`scope.methods`. Nothing read that field. Not ScopeGate, not VerifyGate, not
DiscoveryGate, not ComposeChains, not Prioritize, not the renderer:

    $ grep -c methods Tools/{ScopeGate,VerifyGate,DiscoveryGate,ComposeChains,Prioritize}.ts
    0 0 0 0 0

So "STPA + STRIDE" was an unenforced promise. A run could record it, deliver
STPA only, and pass every gate green.

That happened. A full-surface run recorded STPA + STRIDE, never executed the
STRIDE pass, and instead grew a hand-written nine-row table at the end of
05-constraints.md which the scope prose then described as "a STRIDE
per-element sweep was run". Every gate passed. The requester found it by
reading the finished report and asking where STRIDE was — which is the one
place a method gap should never be discovered first.

This is the Step-4 lesson again: the workflow said to do it, the workflow was
not enough, and it only stopped being skipped when RenderReport began exiting
5 without 04-scenarios.md. A method selection no gate checks is a control
action with no feedback channel.

StrideGate.ts checks five things and exits 8:

  1. CONTRACT   scope.methods is recorded at all. Absent fails rather than
                assuming STPA-only — that assumption is the hole.
  2. DELIVERY   STRIDE selected => 00-stride-seed.md exists with the five
                sections SeedWithStride.md specifies.
  3. RECONCILE  the cross-check runs in BOTH directions. STRIDE->STPA is the
                half people write; STPA->STRIDE names the emergent findings
                STRIDE could not produce, and is the half that gets dropped.
  4. PROVENANCE how the pass was produced is declared. A hand-run sweep is
                legitimate (SeedWithStride sanctions it for code targets) but
                must not silently read as a Fabric-pattern run — same reason
                VerifyGate records which model reviewed the findings.
  5. NO UPGRADE STRIDE-only + a populated grid = the run quietly upgraded
                itself to STPA, which Start.md forbids.

Also:
  - RenderReport grades 00-stride-seed.md `core` ONLY when the contract names
    STRIDE, so an STPA-only run is unaffected and a STRIDE run exits 5 without
    it. Belt and braces with the gate.
  - Adds the §7b section, so the artifact the workflow already mandates is
    actually displayed. It was previously written and silently ignored.
  - Writes stride-scorecard.json so the report states method coverage from
    computed data rather than narration.
  - `stpa stride [dir] [--warn-only]`, wired into `stpa run` after ScopeGate.

Exit 8 was free (1,2,4,5,6,7 taken; VerifyGate uses 3).

Verified against six cases: no contract (8), selected-not-delivered (8),
missing sections (8), one-directional reconciliation (8), silent upgrade (8),
STPA-only (0 — does not demand a STRIDE artifact). Plus --warn-only forces 0,
and the renderer exits 5/0 with the seed absent/present.

One bug found by running it: the threat-row matcher `|\s*ST-\d+` also matched
the reconciliation table's rows, which open with an id followed by prose in
the same cell, and reported four false positives. It now requires the id to
be its own cell.
@MyAlterLego

Copy link
Copy Markdown
Contributor Author

Closing this — the premise it was built on has been revised, and the reasoning is worth recording rather than losing.

This gate enforces that a method selected at intake is actually delivered. It was written after a run recorded STPA + STRIDE, delivered STPA only, and passed every gate green. The gate itself works (six failure cases verified, exit 8; renderer exits 5 independently).

But the fix it implements assumes STRIDE belongs inside the STPA skill as a secondary pass. On review that assumption is wrong, for a reason the gate can't fix:

A STRIDE pass run by the same model that authored the STPA analysis produces a circular cross-check. The value of two methods is that they fail differently, and that property comes from independence of the reasoner, not from the vocabulary. When one model enumerates both sides, "22 of 26 threats map to existing UCA cells" is evidence of internal consistency, not of coverage. Running the Fabric create_stride_threat_model pattern would standardise the schema but not change this, because the same model still executes the pattern.

So a real dual-method assessment needs both halves first-class — each returning full findings, each independently adversarially validated, then synthesised. That is a different deliverable from a pure STPA run, and bolting a weaker half onto one devalues both. It belongs in its own skill.

The decision therefore:

  • STRIDE is removed from this skill entirely; it reverts to pure STPA.
  • This gate is parked for that future dual-method skill, where the method contract it enforces is genuinely load-bearing (with more than one method, there is something real to check).

The branch feat/method-contract-gate is left in place so it can be revived rather than rewritten.

#4 is unaffected and should still merge — those are two genuine bugs (a compose gate that silently checked the wrong directory and reported a false UNRATED COMPOSITION, and a legacy-octal CSS escape that crashed the renderer under Node and emitted the wrong glyph under Bun).

One thing worth carrying into the future skill regardless of STRIDE: scope.methods is currently read by nothing. Verified on main:

$ grep -c methods Tools/{ScopeGate,VerifyGate,DiscoveryGate,ComposeChains,Prioritize}.ts
0 0 0 0 0

An intake field no gate checks is an unenforced promise — the same defect class this toolkit exists to find.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants