Skip to content

test: assert every data-refresh script is scheduled or recorded as exempt - #1358

Merged
mrbobbytables merged 1 commit into
mainfrom
quality/test-refresh-script-wiring
Oct 12, 2026
Merged

mrbobbytables merged 1 commit into
mainfrom
quality/test-refresh-script-wiring

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds tests/refresh-script-wiring.test.mjs. The diff is one new file under tests/ — no behaviour, CI definition, or data file changes.

tests/workflow-scripts.test.mjs already asserts that every validate:*/check:* gate is run by CI or recorded in GATES_NOT_RUN_BY_CI with a written reason. Nothing made the same demand of the scripts on the other side of that contract — the collect:*, fetch:*, import:* and generate:* scripts that write the datasets those gates then validate.

tests/automation-workflow-ordering.test.mjs names the same four namespaces, but only constrains the order of steps in a job that already runs one of them, so a refresh script nothing schedules never appears in any job and that file has nothing to order. The failure that leaves open is silent: a collector stops being scheduled, its dataset stops being refreshed, and its validator keeps passing — because stale data is still valid data.

What the new file asserts

  • Every collect:*/fetch:*/import:*/generate:* script is scheduled, or recorded in REFRESH_NOT_RUN_BY_CI with a reason.
  • Resolution follows npm run chains through package.json, so generate:members counts as wired via import:architectures rather than forcing a false exemption. A dedicated test pins that this transitive step is load-bearing — it fails if generate:members ever becomes directly reachable, so the guard cannot quietly stop proving anything.
  • Staleness guards: an exemption fails if it names a script package.json no longer defines, carries no reason, names a script outside the refresh namespaces, or names a script that is in fact scheduled.
  • Vacuity guards: the detector must still recognise the known refresh scripts, and must still reach the five scheduled collectors.

Exemptions recorded

collect:case-studies and collect:launch-metrics are unscheduled on purpose. That decision previously lived only in prose (scripts/collect-case-studies.mjs, docs/resources/case-studies.md, CONTRIBUTING.md, ROADMAP.md); it is now on the record where a test can tell it apart from an accident.

Verification

  • node --test tests/refresh-script-wiring.test.mjs — 8 passed, 0 failed.
  • Mutation-checked twice, to confirm the guard is not vacuous:
    • renaming an exemption key to a script that does not exist → 3 tests fail;
    • removing npm run collect:radar-reports from refresh-radar-reports.yml → 2 tests fail, including the main wiring assertion.
  • npm run test:unit:coverage:check — exit 0 with the new file included (src files 100.00% lines / 100.00% regions).
  • npx prettier --check — clean.

Related Issue

Closes #1357


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

tests/workflow-scripts.test.mjs demands that every validate:*/check:* gate
be run by some workflow or recorded in GATES_NOT_RUN_BY_CI with a reason.
Nothing made the same demand of the collect:*/fetch:*/import:*/generate:*
scripts that write the datasets those gates then validate.

tests/automation-workflow-ordering.test.mjs names the same four namespaces,
but only constrains the order of steps inside a workflow that already runs
one of them, so a refresh script no workflow runs at all never appears in
any job and that file has nothing to order.

The failure that leaves open is silent: a collector stops being scheduled,
its dataset stops being refreshed, and its validator keeps passing, because
stale data is still valid data.

This adds the guard, following the GATES_NOT_RUN_BY_CI convention. It
resolves npm run chains through package.json so generate:members counts as
wired via import:architectures rather than forcing a false exemption, and
records the two deliberately manual collectors -- collect:case-studies and
collect:launch-metrics -- in REFRESH_NOT_RUN_BY_CI with the reasons that
until now lived only in prose.

Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit d13bd02 Oct 12, 2026
8 checks passed
@mrbobbytables
mrbobbytables deleted the quality/test-refresh-script-wiring branch October 16, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] No test asserts that data-refresh scripts are run by a workflow

1 participant