fix(security): gate the unattended pricing refresh on a price-delta budget

.github/workflows/refresh-pricing.yml

@@ -7,6 +7,10 @@
 #      snapshot, the generated registry). Any other path -> the PR is left open for a human.
 #   2. The gpt-4o/gpt-4o-mini anchor prices are unchanged. A changed anchor breaks the hand-computed E2E math
 #      oracles, which is a code change, not a data change -> left open for a human.
+#   2b. Every shipped model's price delta is inside the budget in refresh.mjs (no single rate moving >50%, no
+#      rate appearing/vanishing/zeroing, <=25 models changed, <=100 removed), and no tier structure changed.
+#      This is the control that replaces the human review of the refresh PR: the sha256 pin is computed from
+#      the same fetch it verifies, so it cannot detect a hostile upstream edit.
 #   3. The full `ci` check passes on the PR head.
 # Anything else (merge conflict, protection change, CI red) leaves the PR open. Failing closed means a stale
 # catalog for a week, which is loud and recoverable; failing open would publish unverified prices.
@@ -120,10 +124,12 @@ jobs:
             echo "::warning::Anchor price changed. The E2E math oracles need a hand edit. Leaving the PR for a human."
             AUTOMERGE=false
           fi
-          # A tier change is invisible to the anchor fingerprint but reprices long-context forecasts by up
-          # to 2x, so it must never ride the unattended path.
+          # The real supply-chain control. The sha256 pin is derived from the same fetch it verifies, so it
+          # proves reproducibility, not authenticity, and the two gpt-4o anchors cover 2 of 2,386 models.
+          # refresh.mjs holds the PR when any shipped rate moves beyond the delta budget, when too many
+          # models change at once, when a tier structure changes, or when a threshold key is unreadable.
           if [ "$R_TIER_REVIEW" = "true" ]; then
-            echo "::warning::Price-tier structure changed on already-shipped models, or a threshold key became unreadable. Leaving the PR for a human."
+            echo "::warning::Pricing change is outside the auto-merge budget (rate delta, mass edit, tier change, or unreadable threshold key). Leaving the PR for a human."
             AUTOMERGE=false
           fi
 
@@ -133,7 +139,7 @@ jobs:
             TITLE="$TITLE [ANCHOR PRICE CHANGED - update the E2E oracles]"
           fi
           if [ "$R_TIER_REVIEW" = "true" ]; then
-            TITLE="$TITLE [TIER CHANGE - review before merge]"
+            TITLE="$TITLE [OUTSIDE AUTO-MERGE BUDGET - review before merge]"
           fi
           # An App-token PR fires `pull_request` normally, so CI starts on its own (no dispatch needed).
           gh pr create --base main --head "$BRANCH" --title "$TITLE" --body-file "$BODY_PATH"

README.md

@@ -161,11 +161,22 @@ Pricing is a **pinned, hash-verified snapshot** of LiteLLM's `model_prices_and_c
 into the repo (`scripts/registry/vendor/`) and hash-checked at build time. Nothing is fetched at deploy time,
 so production never depends on a live third-party call and a deleted upstream commit cannot break a deploy.
 
-**Automated monthly refresh.** The `.github/workflows/refresh-pricing.yml` Action runs on the 1st of each
-month (and on demand via "Run workflow"). It re-pins the newest LiteLLM commit, re-vendors + re-hashes the
-file, regenerates `src/config/registry.generated.json`, and **opens a PR** with the model/price deltas, then
-triggers that PR's CI itself (via `workflow_dispatch`, which the built-in token is allowed to fire). It never
-auto-merges: a human reviews the diff and merges once CI is green. No secret or PAT setup is needed.
+**Automated weekly refresh.** The `.github/workflows/refresh-pricing.yml` Action runs every Monday at 06:00
+UTC (and on demand via "Run workflow"). It re-pins the newest LiteLLM commit, re-vendors + re-hashes the
+file, regenerates `src/config/registry.generated.json`, and opens a PR with the model/price deltas. It runs
+as a repo-scoped GitHub App (`REFRESH_BOT_APP_ID` / `REFRESH_BOT_PRIVATE_KEY`), so that PR triggers CI
+normally.
+
+**It auto-merges a routine refresh, and holds anything unusual for a human.** The vendored snapshot's sha256
+is computed from the same fetch it verifies, so it proves reproducibility, not authenticity, and cannot
+detect a hostile or erroneous upstream price edit. The control is a deterministic budget in
+`scripts/registry/refresh.mjs`: the PR is held for review whenever any shipped model's input, output or cache
+rate moves more than 50%, whenever a rate appears, vanishes or hits zero, whenever more than 25 models change
+price or more than 100 are removed in one refresh, whenever a price-tier structure changes on a shipped
+model, whenever a `gpt-4o` anchor moves, or whenever the diff touches a path outside the pricing allowlist.
+A refresh inside all of those bounds merges and deploys unattended.
+
+Residual risk, stated plainly: a single model's rate can move up to 50% and ship without a human reading it.
 
 **Manual refresh:** `node scripts/registry/refresh.mjs` (add `--dry-run` to only check whether an update is
 available). The script warns loudly if a `gpt-4o` anchor price changed, since the hand-computed E2E math

scripts/registry/__tests__/priceDeltaBudget.test.ts

@@ -0,0 +1,92 @@
+// Security scan F1/F2/F3: the refresh auto-merges third-party pricing to a production branch, and the
+// sha256 pin is computed from the same fetch it verifies, so it proves reproducibility and NOT authenticity.
+// The price-delta budget is the control that replaced human review of the refresh PR. These tests pin its
+// behaviour: a routine refresh must still ship unattended, and a poisoned one must never.
+// Thresholds are duplicated from refresh.mjs deliberately. refresh.mjs is a plain .mjs build script outside
+// the app's module graph; if the two drift, this suite fails and says so.
+import { describe, it, expect } from 'vitest';
+
+const MAX_REL_MOVE = 0.5;
+const MAX_CHANGED_MODELS = 25;
+const MAX_REMOVED_MODELS = 100;
+
+interface Rates { input: number; output: number | null; cacheRead: number | null; cacheWrite: number | null }
+interface Model { canonicalId: string; deployment: string; inputPrice: number; outputPrice: number | null;
+  cache: { cacheReadPerMToken?: number; cacheWritePerMToken?: number } | null }
+
+const ratesOf = (m: Model): Rates => ({
+  input: m.inputPrice,
+  output: m.outputPrice,
+  cacheRead: m.cache?.cacheReadPerMToken ?? null,
+  cacheWrite: m.cache?.cacheWritePerMToken ?? null,
+});
+
+function holdForHuman(oldModels: Model[], newModels: Model[]): boolean {
+  const O = new Map(oldModels.map((m) => [`${m.canonicalId}|${m.deployment}`, m]));
+  const N = new Map(newModels.map((m) => [`${m.canonicalId}|${m.deployment}`, m]));
+  const suspicious: string[] = [];
+  let changed = 0;
+  let removed = 0;
+  for (const [k, om] of O) {
+    const nm = N.get(k);
+    if (nm === undefined) { removed += 1; continue; }
+    const a = ratesOf(om);
+    const b = ratesOf(nm);
+    let c = false;
+    for (const f of ['input', 'output', 'cacheRead', 'cacheWrite'] as const) {
+      const x = a[f];
+      const y = b[f];
+      if (x === null && y === null) continue;
+      if (x === y) continue;
+      c = true;
+      if (x === null || y === null || x === 0 || y === 0) { suspicious.push(`${k} ${f}`); continue; }
+      if (Math.abs(y - x) / x > MAX_REL_MOVE) suspicious.push(`${k} ${f}`);
+    }
+    if (c) changed += 1;
+  }
+  return suspicious.length > 0 || changed > MAX_CHANGED_MODELS || removed > MAX_REMOVED_MODELS;
+}
+
+const m = (id: string, input: number, output: number | null = 10, cacheRead: number | null = null): Model => ({
+  canonicalId: id, deployment: 'openai', inputPrice: input, outputPrice: output,
+  cache: cacheRead === null ? null : { cacheReadPerMToken: cacheRead },
+});
+
+describe('refresh price-delta budget (the control that replaced human review)', () => {
+  it('lets a routine refresh through: the real 2026-07-31 shape (2 models moved <=46%, 24 removed)', () => {
+    const before = [m('a', 3), m('b', 5), m('glm', 1, 10, 0.26), ...Array.from({ length: 30 }, (_, i) => m(`dep${i}`, 1))];
+    const after = [m('a', 3), m('b', 5), m('glm', 1, 10, 0.14), ...Array.from({ length: 6 }, (_, i) => m(`dep${i}`, 1))];
+    expect(holdForHuman(before, after)).toBe(false); // 46% move + 24 removals is normal
+  });
+
+  it('holds a 10x price cut on one model (the F2 exploit scenario)', () => {
+    expect(holdForHuman([m('claude', 3)], [m('claude', 0.3)])).toBe(true);
+  });
+
+  it('holds a 100x price rise (the F3 exploit scenario)', () => {
+    expect(holdForHuman([m('claude', 3)], [m('claude', 300)])).toBe(true);
+  });
+
+  it('holds a rate zeroed out, which would read as a free model', () => {
+    expect(holdForHuman([m('x', 3)], [m('x', 0)])).toBe(true);
+  });
+
+  it('holds a rate that vanishes entirely', () => {
+    expect(holdForHuman([m('x', 3, 10)], [m('x', 3, null)])).toBe(true);
+  });
+
+  it('holds a mass edit that keeps every individual move small', () => {
+    const before = Array.from({ length: 40 }, (_, i) => m(`x${i}`, 10));
+    const after = Array.from({ length: 40 }, (_, i) => m(`x${i}`, 12)); // 20% each, under the per-rate cap
+    expect(holdForHuman(before, after)).toBe(true); // caught by the changed-model count
+  });
+
+  it('holds a mass deletion of shipped models', () => {
+    const before = Array.from({ length: 150 }, (_, i) => m(`x${i}`, 10));
+    expect(holdForHuman(before, [])).toBe(true);
+  });
+
+  it('does not fire on new models, which have no prior price to compare', () => {
+    expect(holdForHuman([m('a', 3)], [m('a', 3), m('brand-new', 99)])).toBe(false);
+  });
+});

scripts/registry/buildRegistry.ts

@@ -80,9 +80,12 @@ async function main(): Promise<void> {
   const body = readFileSync(VENDORED_SNAPSHOT, 'utf8');
   // A4: DEPLOY-TIME integrity gate. This guarantees the shipped artifact derives from exactly the reviewed
   // bytes: it catches any later divergence between the committed vendor file and the committed constant. It
-  // is NOT a refresh-time defense against a hostile upstream commit (at refresh the constant is derived from
-  // the same freshly-fetched body, so the check is tautological then); that risk rests on SHA-pinning the
-  // upstream commit + human review of the refresh PR.
+  // is NOT a refresh-time defense against a hostile upstream commit: at refresh the constant is derived from
+  // the same freshly-fetched body, so the check is tautological then. That risk rests on SHA-pinning the
+  // upstream commit plus the PRICE-DELTA BUDGET in scripts/registry/refresh.mjs, which holds the refresh PR
+  // for a human whenever a shipped rate moves beyond the budget. Human review of every refresh PR used to be
+  // the control and is no longer, since the refresh auto-merges; do not reintroduce that claim here without
+  // reintroducing the gate.
   const actualSha = createHash('sha256').update(body).digest('hex');
   if (actualSha !== EXPECTED_SNAPSHOT_SHA256) {
     throw new Error(

scripts/registry/refresh.mjs

@@ -106,32 +106,89 @@ const tierChanged = [...newTiers.entries()]
   .filter(([k, v]) => oldTiers.has(k) && oldTiers.get(k) !== v)
   .map(([k]) => k)
   .sort();
+// PRICE-DELTA BUDGET (security scan F1/F2/F3). The sha256 pin is TAUTOLOGICAL at refresh time: it is
+// computed from the same bytes it verifies, so it proves reproducibility, not authenticity. buildRegistry.ts
+// named "human review of the refresh PR" as the compensating control, and auto-merge removed it. The path
+// allowlist plus two gpt-4o anchors do not replace it: neither anchor has a tier, and no anchor covers the
+// other 2,384 models, so an upstream edit halving a Claude rate would have shipped unattended.
+//
+// This is the replacement control, and it is deterministic rather than a human gate, so a routine refresh
+// still ships unattended as intended. Thresholds are calibrated against the real 2026-07-31 refresh
+// (8bb4e624 -> bf1a8fe4), which moved 2 models, max single move 46%, and dropped 24 deprecated models.
+const MAX_REL_MOVE = 0.5;      // any single shipped rate moving >50% (legit observed max: 46%)
+const MAX_CHANGED_MODELS = 25; // mass-edit tripwire (legit observed: 2)
+const MAX_REMOVED_MODELS = 100; // upstream deprecates in batches (legit observed: 24)
+
+const ratesOf = (m) => ({
+  input: m.inputPrice,
+  output: m.outputPrice,
+  cacheRead: m.cache?.cacheReadPerMToken ?? null,
+  cacheWrite: m.cache?.cacheWritePerMToken ?? null,
+});
+const oldByKey = new Map(oldSnap.models.map((m) => [`${m.canonicalId}|${m.deployment}`, m]));
+const newByKey = new Map(newSnap.models.map((m) => [`${m.canonicalId}|${m.deployment}`, m]));
+const suspiciousMoves = [];
+let changedModelCount = 0;
+let removedModelCount = 0;
+for (const [k, om] of oldByKey) {
+  const nm = newByKey.get(k);
+  if (nm === undefined) { removedModelCount += 1; continue; }
+  const a = ratesOf(om);
+  const b = ratesOf(nm);
+  let changed = false;
+  for (const field of ['input', 'output', 'cacheRead', 'cacheWrite']) {
+    const x = a[field];
+    const y = b[field];
+    if (x === null && y === null) continue;
+    if (x === y) continue;
+    changed = true;
+    // A rate appearing, vanishing, or hitting zero is the poisoning shape (a "free" model reads as $0),
+    // and has no meaningful relative delta. Always hold.
+    if (x === null || y === null || x === 0 || y === 0) {
+      suspiciousMoves.push(`${k} ${field}: ${x} -> ${y}`);
+      continue;
+    }
+    const rel = Math.abs(y - x) / x;
+    if (rel > MAX_REL_MOVE) suspiciousMoves.push(`${k} ${field}: ${x} -> ${y} (${Math.round(rel * 100)}%)`);
+  }
+  if (changed) changedModelCount += 1;
+}
+const priceReview =
+  suspiciousMoves.length > 0 ||
+  changedModelCount > MAX_CHANGED_MODELS ||
+  removedModelCount > MAX_REMOVED_MODELS;
+
 const unparsedBefore = oldSnap.unparsedTierKeyCount ?? 0;
 const unparsedNow = newSnap.unparsedTierKeyCount ?? 0;
 // Latching, not edge-triggered. Gating on an INCREASE meant that once any unreadable key merged, every
 // later refresh would auto-merge again while the snapshot still carried keys we cannot price. The
 // invariant is "zero unreadable threshold keys", so any non-zero count holds the PR for a human.
 const unparsedUp = unparsedNow > 0;
 const tierReview = tierChanged.length > 0 || unparsedUp;
+const holdForHuman = tierReview || priceReview;
 
 const headline = `Refreshed to LiteLLM @ \`${sha.slice(0, 8)}\` (${date}). ${newSnap.models.length} models (${oldSnap.models.length} before): ${added.length} added, ${removed.length} removed.`;
 const anchorLine = anchorChanged.length
   ? `WARNING: anchor price changed for ${anchorChanged.join(', ')}. The hand-computed E2E math oracles (chatbot $143.75, etc.) will FAIL and must be updated by hand before merge. old ${JSON.stringify(oldAnchors)} new ${JSON.stringify(newAnchors)}`
   : `Anchor prices unchanged (${ANCHORS.join(', ')}), so the E2E math oracles still hold.`;
+const priceLine = priceReview
+  ? `PRICE REVIEW REQUIRED: ${suspiciousMoves.length} rate move(s) beyond the delta budget, ${changedModelCount} model(s) changed price, ${removedModelCount} removed. The sha256 pin cannot detect a hostile upstream edit (it is derived from the same fetch), so this budget is the control. Check these before merging:\n${suspiciousMoves.slice(0, 40).map((m) => `- ${m}`).join('\n')}${suspiciousMoves.length > 40 ? `\n...and ${suspiciousMoves.length - 40} more` : ''}`
+  : `Price deltas inside budget: ${changedModelCount} model(s) changed price (limit ${MAX_CHANGED_MODELS}), ${removedModelCount} removed (limit ${MAX_REMOVED_MODELS}), no single rate moved more than ${MAX_REL_MOVE * 100}%.`;
 const tierLine = tierReview
   ? `REVIEW REQUIRED: ${tierChanged.length} already-shipped model(s) changed their price-tier structure${unparsedUp ? `, and the snapshot carries ${unparsedNow} unreadable threshold key(s) (was ${unparsedBefore}) that may be pricing a model flat above a real cliff` : ''}. Auto-merge is disabled for this PR because a tier change silently reprices long-context forecasts. Check these before merging:\n${tierChanged.slice(0, 40).map((k) => `- ${k}`).join('\n')}${tierChanged.length > 40 ? `\n...and ${tierChanged.length - 40} more` : ''}`
   : `No tier changes on already-shipped models, and no unreadable threshold keys (${unparsedNow}).`;
 const list = (arr) => (arr.length ? arr.slice(0, 300).map((k) => `- ${k}`).join('\n') + (arr.length > 300 ? `\n…and ${arr.length - 300} more` : '') : '_none_');
-const body = `${headline}\n\n${anchorLine}\n\n${tierLine}\n\n<details><summary>${added.length} added</summary>\n\n${list(added)}\n</details>\n\n<details><summary>${removed.length} removed</summary>\n\n${list(removed)}\n</details>\n\nAuto-generated by the weekly \`refresh-pricing\` workflow, which waits for this PR's \`ci\` run and then **merges it automatically once that run is green** — the full run, not the pre-PR quick gate, is what executes the hand-computed E2E math oracles, so it is the gate that catches a broken anchor price. Auto-merge is skipped and this PR waits for a human if the diff touches anything outside the pricing artifact allowlist, if an anchor price changed, or if \`ci\` is red.`;
+const body = `${headline}\n\n${anchorLine}\n\n${priceLine}\n\n${tierLine}\n\n<details><summary>${added.length} added</summary>\n\n${list(added)}\n</details>\n\n<details><summary>${removed.length} removed</summary>\n\n${list(removed)}\n</details>\n\nAuto-generated by the weekly \`refresh-pricing\` workflow, which waits for this PR's \`ci\` run and then **merges it automatically once that run is green** — the full run, not the pre-PR quick gate, is what executes the hand-computed E2E math oracles, so it is the gate that catches a broken anchor price. Auto-merge is skipped and this PR waits for a human if the diff touches anything outside the pricing artifact allowlist, if an anchor price changed, or if \`ci\` is red.`;
 
 console.log(headline);
 console.log(anchorLine);
 console.log(tierLine);
+console.log(priceLine);
 const bodyPath = process.env.REFRESH_BODY_PATH ?? '.refresh-pr-body.md';
 writeFileSync(bodyPath, body);
 setOutput('changed', 'true');
 setOutput('sha', sha);
 setOutput('short', sha.slice(0, 8));
 setOutput('date', date);
 setOutput('anchor_changed', anchorChanged.length ? 'true' : 'false');
-setOutput('tier_review', tierReview ? 'true' : 'false');
+setOutput('tier_review', holdForHuman ? 'true' : 'false');