Change blueprint format to git and enable receiving blueprint updates

AGENTS.md

@@ -14,7 +14,7 @@ The project structure is:
 * packages/workshop-backend: The Gadgets Workshop server.
     * Runs on Cloudflare Workers.
     * This is the **kernel**: it defines the architecture and is held to a higher bar than UI/gatekeeper code. Reviewers read *every line* of `workshop-backend` and of API changes in `workshop-shared`, so keep diffs here small and elegant. Concretely: doc-comment **every** exported member of the `workshop-shared` public API (types, consts, and functions — not just interfaces); never introduce a hand-written interface that mirrors an RPC interface plus an `as unknown as` cast (derive from the real type instead, or rethink the design); and prefer reusing existing mechanisms over adding parallel ones. Capability-based security note: a resource becomes "ambient" (auto-injected) only by user/admin configuration — a gatekeeper must never assert its own ambience. When a change to this package is large, split it by concern into separate PRs (and at minimum group commits so `workshop-backend`/`workshop-shared` can be reviewed apart from UI), since fewer kernel lines = easier review.
-    * Installs the **bundled blueprints** from `@gadgets/bundled-blueprints` on a deployment's first `/api` request, and offers them as the deployment's standard output formats. `scripts/build-bundled-blueprints.ts` generates the gitignored `src/generated/bundled-blueprints.ts` they are installed from (`BUNDLED_BLUEPRINTS_DIR` overrides the source directory), so `build` and `test` both run the generator first; `pnpm import:bundled-blueprint` updates or adds a blueprint from a Workshop export.
+    * Installs the **bundled blueprints** from `@gadgets/bundled-blueprints` on a deployment's first `/api` request, and offers them as the deployment's standard output formats. `scripts/build-bundled-blueprints.ts` generates the gitignored `src/generated/bundled-blueprints.ts` they are installed from, which holds each blueprint's manifest and built files (`BUNDLED_BLUEPRINTS_DIR` overrides the source directory), so `build` and `test` both run the generator first; `pnpm import:bundled-blueprint` updates or adds a blueprint from a Workshop export, of either `.gadget` version. Installing one (`src/bundled-blueprints.ts`) stores it as any published blueprint is stored: a git commit of its files, as a packfile at `<blueprintId>/<commitId>` in R2, named by `metadata.commitId` in KV (see `docs/blueprints.md`). The commit is a *snapshot release* (`buildSnapshotRelease` in `src/blueprint-release.ts`): parentless, with a fixed author, date and message, so it depends on the files alone. Every deployment therefore derives the same commit from the same files, and the installer never reads what was installed before. The cost is that a bundled blueprint's versions are not chained to one another as a published blueprint's releases are. A gadget made from one still takes its updates, merged against the release the gadget last took.
     * Agent turns emit the Workers trace spans Cloudflare's Agents dashboard reads (`src/agent-tracing.ts`): `invoke_agent` per turn, `chat` per model request, `execute_tool` per tool call, and `tool_approval` when an agent's action waits for a manual decision and when it is approved or denied. They carry token counts, including cache reads and writes, but never prompts, responses or tool data. A span belongs to the invocation that opened it, and once a Durable Object invocation has returned, the next event the object receives ends its trace. So a turn keeps all its spans only while the request that started it (usually the browser session) stays open.
 * packages/bundled-blueprints: The blueprints the deployment ships with, `blueprints/<name>/` holds each blueprint's `blueprint.json` plus its gadget code under `files/`, which may be TypeScript.
 * packages/workshop-shared: Shared API definitions between client and server.

docs/integration-testing.md

@@ -79,6 +79,20 @@ One corollary that is easy to get wrong: the "nothing escaped to the internet" a
 running, so it would inspect and clear state they are still using — and could discard an escape a
 sibling was about to be blamed for.
 
+### A redeploy keeps storage, so an upgrade can be tested
+
+`server.update()` is the opposite of `reset()`: it reloads the Workers from new configs and keeps
+every Durable Object, the KV namespaces, the R2 bucket and the server URL. That makes it a
+deployment of a new version over an old one, which is what `Harness.redeployWorkshop()` uses it for.
+It costs about 1.5 s, and it does break every open RPC session, so a test that redeploys starts a
+harness of its own rather than doing it under its siblings.
+
+What changes between the two builds has to be something a config can express, because the Workshop
+is built once, before any test file runs. A build-time input qualifies if wrangler's bundler can
+swap it: `bundleBlueprints()` points the `alias` for the Workshop's generated bundled-blueprints
+module at `fixtures/bundled-blueprints.ts`, and hands that fixture its list through a `define`. The
+installer, and the check that decides whether to run it, are the Workshop's own.
+
 ### wrangler and miniflare versions are coupled
 
 Nothing pins `workerd` directly: `wrangler` and `miniflare` each depend on an exact `workerd`. The

packages/bundled-blueprints/README.md

@@ -1,15 +1,15 @@
 # Bundled blueprints
 
 This package holds the blueprints that ship with this repo, the gadget libraries they import, and
-the build that turns a blueprint directory into the archives the Workshop backend installs. A fresh
+the build that turns a blueprint directory into the files the Workshop backend installs. A fresh
 deployment installs the blueprints into BLUEPRINTS KV and BLUEPRINT_CONTENT R2 on its first `/api`
 request, after which they are ordinary blueprints, and promotes them as its standard output formats.
 "Bundled" is what this package holds; "format" is a curation state an admin controls at runtime --
 a bundled blueprint can be taken out of the formats menu, and an unbundled one promoted into it.
 
 Nothing here is deployed on its own. The Workshop backend's `scripts/build-bundled-blueprints.ts`
 imports `src/` to generate the gitignored `src/generated/bundled-blueprints.ts` it compiles the
-archives into, and `pnpm import:bundled-blueprint` (the same package) writes into `blueprints/`.
+blueprints into, and `pnpm import:bundled-blueprint` (the same package) writes into `blueprints/`.
 
 ## Layout
 
@@ -21,26 +21,32 @@ blueprints/<name>/     one bundled blueprint, committed as reviewable source
     client.ts          import { el } from "@gadgets/bundled-blueprints/libraries/ui/client"
     server.ts          import { MutationQueue } from ".../libraries/sync/server"
     lib/protocol.ts    the document, operation and RPC types both sides share
-  __tests__/           vitest, repo-only; never part of the archive
+  __tests__/           vitest, repo-only; never part of the blueprint
 libraries/<name>/      a gadget library: client.ts, server.ts, src/**, __tests__/** (see libraries/README.md)
-src/                   the build: archive codec and source reader (files.ts), manifest parser
+src/                   the build: source and archive reader (files.ts), manifest parser
                        (manifest.ts), module generator (generate.ts); index.ts is what the backend imports
 __tests__/             the build's own tests
 ```
 
 `files/` is the gadget's code. `blueprint.json` contains its install ID, presentation, provenance,
-bindings, blueprint `version`, and bundled `revision`. The build converts these files into the same
-gzip-compressed Yjs `.gadget` representation used by uploaded blueprints and embeds it in the
-generated Worker module. No binary archive is committed.
+bindings, blueprint `version`, and bundled `revision`. The build embeds the manifest and the files,
+as built, in the generated Worker module. No binary archive is committed, and none is built.
+
+Installing a blueprint makes a git commit of its files, as publishing one from a Workshop does, and
+stores it as a packfile. The commit has no parent and a fixed author, date and message, so it
+depends on the files and nothing else: every deployment that installs the same files installs the
+same commit, and a blueprint whose files have not changed is the same commit however often it is
+installed. A bundled blueprint's versions are not chained to one another the way a published
+blueprint's releases are: each is a commit on its own.
 
 ### Libraries
 
 A blueprint may import the shared **gadget libraries** in `libraries/` (see [its
 README](libraries/README.md)) by this package's name and the subpath its `package.json` exports:
 `@gadgets/bundled-blueprints/libraries/<name>/client` from its client, `.../<name>/server` from its
 server. The build resolves each to the library's entry and inlines what the entry uses into the
-shipped `client.js` / `server.js`, as it does a `lib/` module, so the archive stays self-contained
-and a gadget created from the blueprint carries its own copy of the library as of its
+shipped `client.js` / `server.js`, as it does a `lib/` module, so the blueprint stays
+self-contained and a gadget created from it carries its own copy of the library as of its
 instantiation. A client may not import a library's server side (it would drag a Durable Object into
 the iframe), and a library, reached by that subpath, is the one thing an import may reach outside
 `files/` for. The importer has to be TypeScript: a JavaScript module ships as written, with nothing
@@ -52,8 +58,8 @@ their own domain code in `files/`.
 
 `files/` may be written in TypeScript: `client.ts` and `server.ts` are the entry points, and
 `lib/**/*.ts` holds the modules they import (by on-disk name, `./lib/protocol.ts`). The build bundles
-each entry with its imports into the `client.js` / `server.js` the archive ships -- readable rather
-than minified -- so the installed gadget, and the agent that later edits it, see exactly one
+each entry with its imports into the `client.js` / `server.js` the blueprint ships -- readable
+rather than minified -- so the installed gadget, and the agent that later edits it, see exactly one
 JavaScript file per side, the same as for a blueprint written in plain JavaScript. Those `lib/`
 modules are build input only and are not stored, and `.d.ts` files are dropped. The bundled Docs,
 Sheets and Slides blueprints are written this way, each with a `lib/protocol.ts` holding the types
@@ -135,7 +141,7 @@ subpath -- and stay inside these limits, which the build does not enforce:
   the globals its runtime supplies, and the type check is only as honest as that isolation.
 - A tree under `BUNDLED_BLUEPRINTS_DIR` is bundled but not type-checked; the programs above cover
   this package's `blueprints/` and `libraries/` only. Type-check such a tree in its own workspace.
-- The comment naming each inlined library module is rewritten line by line, so the archive is the
+- The comment naming each inlined library module is rewritten line by line, so the bundle is the
   same wherever it is built; a template literal whose own line spells exactly such a path would be
   rewritten with it.
 
@@ -146,15 +152,16 @@ and run under `pnpm test` (jsdom by default; a pure module's or a server's test
 named `server.test.ts` or `<topic>.server.test.ts`, which is what puts it under the Workers types
 rather than the DOM's; at run time it gets `cloudflare:workers` as a stub of its base classes
 (`__tests__/stubs/cloudflare-workers.ts`), so a Durable Object can be constructed over in-memory
-storage. The archives the build produces are still installed and inspected inside workerd by the
+storage. The blueprints the build produces are still installed and inspected inside workerd by the
 Workshop backend's suite. Only `blueprint.json` and `files/` are read by the build, so `__tests__/`
-(and anything else beside them) is repo-only and never part of the archive.
+(and anything else beside them) is repo-only and never part of the blueprint.
 
 `blueprintId` is the install key. Never change it after deployment: the new ID would install a
 second blueprint while the old one remained. (TODO: the bundled IDs keep the historical `format.`
 prefix -- `format.document`, `format.spreadsheet`, `format.slides` -- for that reason; renaming them
-needs an install-time migration keyed on the old id.) `version` is the blueprint's published content version
-and R2 key. The build fingerprints the generated archive, so direct edits under `files/` reinstall
+needs an install-time migration keyed on the old id.) `version` is the blueprint's published content
+version, a counter for display: the content itself is stored under the commit of its files. The
+build fingerprints the files and the rest of the manifest, so direct edits under `files/` reinstall
 automatically. `revision` remains an explicit reinstall trigger and is bumped by the importer.
 
 ## Editing presentation
@@ -176,6 +183,12 @@ updates archive-owned metadata (`created`, `version`, `lastUpdated`, and `bindin
 and bindings. An export the build rejects is refused before it replaces
 anything. Review the resulting source diff normally.
 
+An export is a `.gadget` archive in one of two versions, and the importer reads both. Version 1
+holds a snapshot of the blueprint's files. Version 2, which a Workshop writes for anything published
+since blueprints became git commits, holds a git packfile, and the importer has `git` unpack it.
+Only the files are taken from either: the bundled blueprint's commit is made from them at install,
+not copied from the export.
+
 An export contains the bundled JavaScript, so importing one over a TypeScript blueprint replaces its
 sources with the built `client.js` / `server.js`. Edit a TypeScript blueprint in the repo instead,
 and use the Workshop only to try the result.
@@ -216,7 +229,9 @@ be, so its editor and `tsc` resolve the same `exports`; or it can stay JavaScrip
 
 Directories using the previous `<name>.gadget` plus `<name>.json` layout remain supported, so an
 existing deployment can update this repo without coordinating a format conversion. Importing a new
-export into one of those entries migrates that pair to the extracted layout automatically.
+export into one of those entries migrates that pair to the extracted layout automatically. That is
+also the only way to update one: the layout holds version 1 archives, and the build refuses a
+version 2 export dropped in a `<name>.gadget`'s place.
 
 Administrators can also publish and promote ordinary blueprints at runtime instead of rebuilding a
 deployment.

packages/bundled-blueprints/__tests__/archives.ts

@@ -0,0 +1,39 @@
+// Writes `.gadget` archives for tests of the code that reads them. Nothing in `src/` writes one:
+// the build emits a blueprint's files, and the archives the importer reads come from a Workshop.
+// Shared with the backend scripts' suite (`scripts/build-bundled-blueprints.test.ts`).
+
+import { gzipSync } from "node:zlib";
+import * as Y from "yjs";
+
+const MAGIC = 0xec2e2d3a2300e317n;
+const PREFIX_BYTES = 24;
+
+/** A `.gadget` archive of the given version, holding `metadata` and `content` as they are. */
+export function serializeArchive(
+  version: number,
+  metadata: Record<string, unknown>,
+  content: Uint8Array,
+): Uint8Array {
+  const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata));
+  const out = new Uint8Array(PREFIX_BYTES + metadataBytes.byteLength + content.byteLength);
+  const view = new DataView(out.buffer);
+  view.setBigUint64(0, MAGIC);
+  view.setUint32(8, version);
+  view.setUint32(12, metadataBytes.byteLength);
+  view.setBigUint64(16, BigInt(content.byteLength));
+  out.set(metadataBytes, PREFIX_BYTES);
+  out.set(content, PREFIX_BYTES + metadataBytes.byteLength);
+  return out;
+}
+
+/** The content of a version 1 archive: a gzip-compressed Yjs snapshot of `files`. */
+export function buildSnapshotContent(files: ReadonlyMap<string, string>): Uint8Array {
+  const doc = new Y.Doc();
+  const root = doc.getMap();
+  for (const [filename, source] of files) {
+    const text = new Y.Text();
+    root.set(filename, text);
+    text.insert(0, source);
+  }
+  return gzipSync(Y.encodeStateAsUpdateV2(doc));
+}