test: add the toolkit join parity harness - #1113
Open
teallarson wants to merge 6 commits into
Open
Conversation
Build the oracle S5 and S6 are judged by: does joining the extracted layers (enrichment + curation) against a catalog snapshot reproduce the toolkit JSON we ship today? - capture-catalog-snapshot.ts pulls a raw /v1/tool_metadata snapshot (paginated at 1000), records total_count, and refuses to write a truncated fetch. The snapshot is gitignored (~10 MB, never committed). - verify-toolkit-join.ts joins a snapshot with the enrichment and curation layers through the real merger (no LLM, committed toolkit as previousToolkit) and reports the first structural difference per file by JSON path. Comparison is structural, not raw-byte: generatedAt is volatile and summary's key position varies across generator versions. Until S3/S5 land the enrichment/ and curation/ directories, the layers are read straight out of the committed toolkit JSON (the default). Proven offline: projecting every committed tool back to its pre-merge shape and re-joining reproduces all 117 toolkits with zero differences; an injected one-character enrichment change and a missing catalog item each fail with a named JSON path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Aug 5, 2026
These scripts live in scripts/, which nothing type-checks today — #1106 widens the generator project to cover it. Under that project's exactOptionalPropertyTypes and noUncheckedIndexedAccess they do not compile, so #1106 landing would break this PR. - IMetadataSource was imported from src/sources/interfaces, which does not export it; it lives in src/sources/internal, where all eight other consumers import it from. It is an 'import type', so it erased at runtime and no test could catch it. - Bind the indexed argv reads before use: a truthiness check on argv[i + 1] does not narrow a later, separate read of the same index. - Declare the two optional properties that legitimately receive an explicit undefined as '| undefined' rather than spreading conditionally at each call site. No behaviour change; the harness still reproduces all 117 toolkits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
teallarson
marked this pull request as ready for review
August 6, 2026 21:22
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e9a38de. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The generator is being split into upstream data, enrichment, and curation layers. This adds a deterministic parity gate that answers one question: does joining those layers still reproduce the toolkit JSON we ship today?
It turns output preservation from a manual review task into a mechanical check, protecting the pipeline as its sources are extracted and reorganized.
How it fits
The harness sits alongside generation rather than replacing it: it uses the production merger, reports the first structural difference by JSON path, and makes future pipeline slices independently verifiable.
Changes
total_countand rejects truncated fetches.Verification
generatedAtare handled structurally to avoid false positives.Independent of #1112 at merge time, but intended to validate it and the subsequent enrichment/source-extraction slices.
Note
Low Risk
New scripts and tests only; no changes to the production generator merge or shipped data paths.
Overview
Adds a join parity gate for the upcoming catalog / enrichment / curation split: mechanically check that those layers still produce the same shipped toolkit JSON.
capture-catalog-snapshot.tspaginates Engine/v1/tool_metadata(aligned withEngineApiSource), recordstotal_count, and refuses to write truncated snapshots..gitignorekeeps the ~10 MB local snapshot out of git.verify-toolkit-join.tsloads a snapshot throughparseToolMetadataResponse, runs the realmergeToolkitpath (design-system metadata + provider resolver), overlays frozen enrichment (codeExample,secretsInfo,summary), and compares todata/toolkitswith a structural diff (ignoresgeneratedAt, treats optional null/absent keys as equivalent). Non-zero exit on mismatch or missing catalog tools. Until separate layer dirs exist, enrichment/curation default to the committed toolkit files.verify-toolkit-join.test.tscovers the diff helpers, catalog reshape, full round-trip over all committed toolkits, and failure cases (enrichment drift, missing catalog entries).Reviewed by Cursor Bugbot for commit f18ce96. Bugbot is set up for automated code reviews on this repo. Configure here.