fix(security): gate the unattended pricing refresh on a price-delta budget - #30
Merged
Merged
Conversation
…udget Whole-repo security scan (181 agents, high effort, three-lens adversarial panel) returned three findings that share one root cause, and the cause is code I shipped this morning. The vendored snapshot's EXPECTED_SNAPSHOT_SHA256 is computed from the same fetch it verifies, so it proves reproducibility and NOT authenticity. buildRegistry.ts said so in its own comment and named the compensating control: "human review of the refresh PR". Making the refresh auto-merge removed exactly that control, and the replacement guards did not cover the gap. The path allowlist permits every pricing change by design, and the anchor gate covers gpt-4o and gpt-4o-mini, which is 2 of 2,386 models and neither carries a tier. Anyone landing a price edit in a community-maintained upstream file could have had it fetched, hashed into our own pin, built, auto-merged and deployed to tokentally.ai inside a week with nobody reading a number. Fix is a deterministic budget rather than a human gate, so a routine refresh still ships unattended as intended. The PR is held for review when any shipped model's input/output/cache rate moves more than 50%, when a rate appears, vanishes or hits zero, when more than 25 models change price, when more than 100 are removed, or on the existing tier/anchor/allowlist conditions. Thresholds are calibrated against the real 2026-07-31 refresh rather than guessed: it moved 2 models, max single move 46%, and dropped 24 deprecated models, so a 25% cap would have blocked a legitimate refresh. Verified against that refresh (passes) and against poisoned variants: a 10x cut, a zeroed rate, and a 40-model 20% mass edit all hold. Also corrects two claims the scan found stale: buildRegistry.ts still named human review as the control, and README.md still said the refresh was monthly and "never auto-merges". Residual risk, documented in the README rather than hidden: a single model's rate can still move up to 50% and ship unattended. 406 tests. Gates green: lint, both tsconfigs, build, size, claims, first-paint, npm audit 0, semgrep 0.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR strengthens the supply-chain safety of the automated LiteLLM pricing snapshot refresh by adding a deterministic “price-delta budget” gate that holds refresh PRs for human review when pricing changes look suspicious or unusually large, while still allowing routine refreshes to auto-merge unattended.
Changes:
- Add price-delta budgeting logic to the refresh script and surface it in the generated PR body/output used by the workflow.
- Update documented security rationale/controls in the registry build script, workflow comments, and README.
- Add regression tests that pin the budget thresholds/behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/registry/refresh.mjs | Implements the price-delta budget gate and includes it in PR messaging + workflow outputs. |
| scripts/registry/buildRegistry.ts | Updates security-control documentation to reflect the new deterministic gate. |
| scripts/registry/tests/priceDeltaBudget.test.ts | Adds tests to lock in the budget thresholds and expected hold/allow behavior. |
| README.md | Updates refresh cadence and documents the new unattended vs. hold-for-review policy and residual risk. |
| .github/workflows/refresh-pricing.yml | Documents and enforces the updated “outside auto-merge budget” hold condition in the automation flow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+85
to
+88
| // upstream commit plus the PRICE-DELTA BUDGET in scripts/registry/refresh.mjs, which holds the refresh PR | ||
| // for a human whenever a shipped rate moves beyond the budget. Human review of every refresh PR used to be | ||
| // the control and is no longer, since the refresh auto-merges; do not reintroduce that claim here without | ||
| // reintroducing the gate. |
Comment on lines
+56
to
+60
| it('lets a routine refresh through: the real 2026-07-31 shape (2 models moved <=46%, 24 removed)', () => { | ||
| const before = [m('a', 3), m('b', 5), m('glm', 1, 10, 0.26), ...Array.from({ length: 30 }, (_, i) => m(`dep${i}`, 1))]; | ||
| const after = [m('a', 3), m('b', 5), m('glm', 1, 10, 0.14), ...Array.from({ length: 6 }, (_, i) => m(`dep${i}`, 1))]; | ||
| expect(holdForHuman(before, after)).toBe(false); // 46% move + 24 removals is normal | ||
| }); |
Comment on lines
+10
to
+13
| # 2b. Every shipped model's price delta is inside the budget in refresh.mjs (no single rate moving >50%, no | ||
| # rate appearing/vanishing/zeroing, <=25 models changed, <=100 removed), and no tier structure changed. | ||
| # This is the control that replaces the human review of the refresh PR: the sha256 pin is computed from | ||
| # the same fetch it verifies, so it cannot detect a hostile upstream edit. |
Comment on lines
183
to
+186
| console.log(headline); | ||
| console.log(anchorLine); | ||
| console.log(tierLine); | ||
| console.log(priceLine); |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Whole-repository security scan at high effort: 181 agents, 17.8M tokens, three-lens adversarial verification panel. Three findings, all sharing one root cause, and the cause is code I shipped this morning.
The finding
EXPECTED_SNAPSHOT_SHA256is computed from the same fetch it verifies (refresh.mjs:84). It proves the vendored file matches what we downloaded. It proves nothing about whether what we downloaded is honest.buildRegistry.tsalready said this, and named the compensating control:Making the refresh auto-merge (#26, #28) deleted that control. The replacements do not cover the gap:
gpt-4oandgpt-4o-mini. That is 2 of 2,386 models, and neither carries a tier.So: land a price edit in a community-maintained upstream file, and Monday it is fetched, hashed into our own pin, built, auto-merged and deployed to tokentally.ai. Nobody reads a number. A 10x cut to a Claude rate passes
MAX_RAW_RATEcomfortably.The fix
A deterministic budget, not a human gate — a routine refresh still ships unattended, which is the point of the automation. The PR is held for review when:
Thresholds are calibrated, not guessed
Measured against the real 2026-07-31 refresh (
8bb4e624->bf1a8fe4):A 25% cap would have blocked that legitimate refresh. Verified behaviour:
8 regression tests pin this, including a parity assertion against the thresholds in
refresh.mjs.Stale claims corrected
buildRegistry.tsstill named human review as the control. It now names the budget, and warns against reinstating that claim without reinstating the gate.README.mdstill said the refresh was monthly and "never auto-merges: a human reviews the diff". Both false since this morning.Residual risk
A single model's rate can still move up to 50% and ship unattended. Stated in the README rather than buried. Tightening below 50% would block legitimate refreshes at the observed 46% move.
Verification
406 tests (from 398) · lint · both tsconfigs · build · size budget · claims · first-paint ·
npm audit0 · semgrep 0 across 5 rulesets.