feat: add Demo Manifest build pipeline#24
Draft
MartinKanera wants to merge 14 commits into
Draft
Conversation
Golden-fixture test at the transform seam for the Demo Manifest build script (joint-mcp#27), written test-first. Covers variant canonicalization (react-ts -> react, vue-ts -> vue), edition detection incl. the CDN/no-dependency title fallback, README fallback from variant to demo root, keyword extraction, and Uses extraction (Joint API symbols via namespace member resolution).
Pure transform + walker producing one searchable Manifest per demo variant (joint-mcp#27): title/summary from README, canonical Variant (react-* -> react), joint packages, Edition, used Joint API symbols extracted from imports, file list, and the demo_id matching the R2 versioned_demos layout so get_demo_code keeps working unchanged. Makes the golden-fixture test green.
CLI for joint-mcp#27: walks all demos and writes one Manifest per variant plus index.json into .manifests/version-X.Y/. The --version flag is required because the repo is unversioned — the value labels the Demo Snapshot being synced (forward-only from 4.3).
Cover usage, the required --version flag, output layout, and the intended R2 destination (manifests/ prefix next to versioned_demos/) for the Demo Manifest script (joint-mcp#27).
Classify legacy Rappid packages (@clientio/rappid, rappid) as commercial, and make the variant file walker skip symlinks and dot-directories via lstat (dot-files like .npmrc stay listed). Review findings from joint-mcp#38; rebuilt 4.3 output verified byte-identical.
Keywords now come solely from a root-level demo-keywords.json overlay (README extraction removed; missing demos warn and omit the line, comma keywords warn and are skipped). Uses drops tooling and unused imports, canonicalizes aliased and star imports, and collapses member chains after the first capitalized segment. Summaries lose the repeated 'also available online' sentence. The Files section becomes Source files and hides lockfiles and binary assets (display-only; the Demo Snapshot and get_demo_code are unaffected). One golden refresh, per issue #39, so R2 never serves the v1 noise.
parseArgs strict mode threw a raw stack trace on unknown flags or positionals; wrap it so mistakes print the usage line and exit 1. The required --version guard is unchanged. Folded in from #38 via #39.
Bootstrap demo-keywords.json for all demos: category and feature tags harvested from each demo's jointjs.com/demos page (slugs resolved via README links), inference from README and source where no page exists, plus diagram-type synonyms per the authoring rule in the manifests README. Sole keyword source for Demo Manifests from now on (#39).
Review follow-ups: replace a U+2011 non-breaking hyphen that broke exact-match retrieval, restore sankey-diagram's two dropped site tags, drop the fta-js package token, swap generic-UML spillover synonyms for use-case vocabulary, name the online-note pattern, and comment the lazy per-demo keyword resolution. No manifest bytes change except the five corrected Keywords lines.
The StackBlitz badge in 83 variant READMEs is multi-line HTML, and per-line tag stripping left its attribute lines in every affected Manifest summary. Clean the summary as one joined block so tags strip across line boundaries, and require tags to start with a letter or '/' so prose '<' is never swallowed. Nothing from the badge is kept: the StackBlitz URL is derivable from demo_id, like the jointjs.com live URL. Addendum to #39, recorded there.
Sibling variant manifests were near-clones competing for AutoRAG
top-k slots and hiding their siblings (joint-mcp#39 v3 delta). One
document per demo now carries per-variant sections (demo_id, Packages,
Uses, Source files) under a shared header sourced from the demo-root
README (title, summary, Edition, Keywords). React sections not built
on @joint/react get an advisory steering to the declarative API. The
index slims to a bare {demo, variant_dir, variant} array at
manifests-index/version-X.Y.json, outside the AutoRAG-indexed prefix,
enabling an exact folder $eq version filter; build stats move to
stdout.
Manifest v3 sources summaries from demo-root READMEs, and framework-name tokens there are a retrieval hazard — every demo smells react-ish to the embedder (joint-mcp#39). Remove the 'can be easily integrated with TypeScript, React, ...' boilerplate from flowchart, kanban, and mind-map.
Update the manifests script README for v3 (joint-mcp#39): flat per-demo output with per-variant sections, the slim runtime index outside the AutoRAG-indexed prefix, demo-root README title/summary sourcing, and the imperative-react advisory.
The documented rclone command was hand-typed per sync and already caused a live defect: unanchored excludes stripped every variant's package.json from the Demo Snapshot (joint-mcp#30). Encode the structural filter ruleset and both v3 manifest prefixes behind npm run sync / sync:prod with preflight guards (build output present, clean tree) so a stale or partial upload fails closed. Every destination is version-scoped, so other versions' objects in the bucket are never deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Demo Manifest build pipeline: one searchable markdown document per demo (187 for snapshot 4.3, each with one section per variant) plus a slim runtime index, so the JointJS MCP demo search (clientIO/joint-mcp) can rank curated catalog entries instead of raw source chunks. Includes the v2 quality pass and the v3 per-demo format bump, so the first R2 sync never indexes noisy v1 output or per-variant near-clones.
Why
Demo READMEs are ~80% shared boilerplate with no packages/edition/variant metadata — poor retrieval documents. Manifests carry that signal deterministically (no LLM at build time), with vocabulary shared with the jointjs.com/demos taxonomy. Sibling variant manifests were near-clones (kanban js vs ts) competing for top-k slots and hiding each other — hence one document per demo with per-variant sections.
Key changes
.github/scripts/manifests/— pure transform + walker + CLI (npm run manifests:build -- --version X.Y), golden-fixture tested (npm run manifests:test, byte-exact)manifests/version-X.Y/{demo}.md(shared title/summary from the demo-root README, Edition, Keywords; per variant its demo_id, Packages, Uses, Source files) + bare-array index{demo, variant_dir, variant}atmanifests-index/version-X.Y.json, kept outside the AutoRAG-indexed prefix@joint/react*carry a deterministic note steering agents to the declarative API (6 sections at 4.3)demo-keywords.json— curated per-demo keyword overlay (187 entries, harvested from jointjs.com/demos tags + synonyms); sole keyword source, missing entries degrade to a build warning### Source filesview (lockfiles/binaries excluded — display-only, the Demo Snapshot upload is unaffected)Testing
npm run manifests:test— 9/9, byte-exact golden fixtures (incl. advisory positive/negative cases)Notes
docs/superpowers/specs/); a companion joint-mcp PR carries the glossary/docs.Closes clientIO/joint-mcp#27
Closes clientIO/joint-mcp#39
🤖 Generated with Claude Code
https://claude.ai/code/session_01Lf1wbLhUadcXfkJB8ByoHT