ci (release): Rework the SBOM output verification step - #383
Draft
turbobobbytraykov wants to merge 1 commit into
Draft
ci (release): Rework the SBOM output verification step#383turbobobbytraykov wants to merge 1 commit into
turbobobbytraykov wants to merge 1 commit into
Conversation
Restores the 'Verify SBOM output' step and the two SBOM_* budget variables it reads, unchanged, so the rework has a baseline to diff against rather than starting from a deleted step. The check is not in a state to merge. It asserts SBOM quality through a hand-maintained blocklist of build-only package names, a component ceiling and a declared-license floor - three proxies that each fail in a different direction: the blocklist only catches names somebody thought to add, the ceiling is a single number covering two ecosystems, and the license floor is set below the measured value rather than at it. Draft until that is replaced with something that checks the property we actually care about.
20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Warning
Draft — this needs reworking completely. Do not review it as a proposal; review it as a problem statement.
Reinstates the
Verify SBOM outputstep and the twoSBOM_*budget variables it reads, unchanged, on a branch of its own. It was removed from #371 in4c748a7so the rest of the release workflow is not held behind this rework. Restoring it verbatim here gives the rework a baseline to diff against rather than a deleted step to reconstruct from memory.Base branch is
btraykov/release-workflow-refactoring, notmaster, because the step only makes sense on top of thesbomjob that #371 introduces.Motivation / Context
The
sbomjob generates SPDX 2.2 and 3.0 documents and attests them against the package digest. An SBOM that is generated from the wrong dependency graph — the whole dev tree instead of the shipped set — is worse than no SBOM, because it is signed, attested, and wrong. The alpha.1 run produced 583 components for a package that ships around 60. Something has to fail the release when that happens again.The step in this PR is that something, and it does catch the alpha.1 failure. But it establishes the property it wants through three proxies, and each fails in a different direction:
webpack,xunit,prettierand the dozen-odd others somebody thought to list. A new devDependency, a renamed one, or a transitive build-only package nobody enumerated passes silently. The list needs updating every time the toolchain changes, and nothing makes that failure visible.The underlying issue is that all three try to infer "was this generated from a production-only graph?" from the shape of the output. That question has a direct answer — the restore and install that fed the generator either excluded dev dependencies or did not — and asserting it directly would be both stronger and maintenance-free.
What the rework should probably look like
Not committing to a design in the PR description, but the directions worth evaluating:
sbomjob already runsdotnet restoreon the library project alone andnpm ci --omit=dev --ignore-scriptsfollowed by deleting the lockfile. Those are the invariants that matter. Verifying them directly — that no otherproject.assets.jsonwas written, that the lockfile is gone before the generator runs — replaces all three proxies with checks that cannot silently rot.artifacts. Its dependency group and its bundled static assets are the shipped set by definition, so the SBOM's package list can be diffed against them instead of against a number somebody picked.eng/bundle-budgets.jsonhandles bundle sizes.### SBOMblock with component and license counts is the one part of this step that is unambiguously worth keeping regardless of how the assertions change.Type of Change (check all that apply):
Component(s) / Area(s) Affected:
Release workflow,
sbomjob only. No product code.How Has This Been Tested?
Not tested, deliberately. The content is byte-identical to what was on
btraykov/release-workflow-refactoringatcde2b0a—git diffagainst the base is 57 additions and 0 deletions, and the YAML was parsed to confirmVerify SBOM outputsits back betweenGenerate SBOMsandReverify package before attestation. The step's behaviour was exercised by the0.1.2-alpha.*release runs; the objection here is to what it asserts, not to whether it runs.Test Configuration:
windows-latestrunnerChecklist:
README.MDCHANGELOG.MDupdates for newly added functionalityMerge order
Blocked on #371. Rebase onto
masterand retarget once that merges.