|
| 1 | +#!/usr/bin/env bash |
| 2 | +# test_job_gates.sh — hermetic structural tests for the JOB-LEVEL `if:` conditions in |
| 3 | +# .github/workflows/pr-risk.yml. No network, no Actions: this reads the workflow file as text. |
| 4 | +# |
| 5 | +# WHY A SCRIPT SUITE CANNOT COVER THIS. Every other pr-risk suite tests a script, and a job `if:` |
| 6 | +# is not in one — it exists only in the workflow, is evaluated by the runner before any step is |
| 7 | +# dispatched, and decides whether a script is invoked at all. `resolve-enabled.sh` can be green on |
| 8 | +# every input it takes and still have its answer consumed by the wrong job, because which jobs |
| 9 | +# read `enabled` is not a fact any script can see. So the gates get the same treatment |
| 10 | +# `test_pin_contract.sh` gives the `workflows_ref` guard: a structural read over the YAML text. |
| 11 | +# |
| 12 | +# THE TWO FAILURE MODES THIS FILE PINS, both of which ship green through every script suite: |
| 13 | +# |
| 14 | +# * THE DISPATCH PREVIEW PUBLISHES NOTHING. `enabled` governs the AUTOMATIC `pull_request` |
| 15 | +# stream only — the `grade` job is exempt on `workflow_dispatch` precisely so a repo enrolled |
| 16 | +# OFF can grade one PR and see what the grader would say. Re-gating `publish-check` on |
| 17 | +# `needs.gate.outputs.enabled` silently removes the Check Run from that preview: the run |
| 18 | +# grades, labels and comments, and the one immutable commit-attached artifact the preview |
| 19 | +# exists to demonstrate is the single thing missing. Nothing fails; the reader just concludes |
| 20 | +# the feature does not work. |
| 21 | +# |
| 22 | +# * THE SKIPPED-NEEDS RUNNER SPIN. The tempting minimal fix is `inputs.check_run && |
| 23 | +# !cancelled()`. But `!cancelled()` is a status function, and ANY status function in an `if:` |
| 24 | +# replaces the implicit "all needs succeeded" check — so on a disabled `pull_request` run, |
| 25 | +# where `grade` was merely SKIPPED, `publish-check` RUNS: it boots a runner, finds an empty |
| 26 | +# `surfaces`, publishes nothing and exits 0. Green, invisible, and one wasted runner per |
| 27 | +# disabled PR across every enrolled repo — exactly the cost the cheap `gate` job exists to |
| 28 | +# avoid. Hence the `needs.grade.result` test, which is what makes the `!cancelled()` safe. |
| 29 | +# |
| 30 | +# WHY `!= 'skipped'` AND NOT `== 'success'`, restated here because the two read alike: the publish |
| 31 | +# step already tolerates a partial render (an empty `surfaces` is "nothing to publish", not an |
| 32 | +# error, and per-target failures are warnings). A `success()`-shaped gate would throw away the |
| 33 | +# targets that DID render whenever grade failed partway. |
| 34 | +# |
| 35 | +# The grade side is pinned too, as the other half of the same contract: PR #116's dispatch |
| 36 | +# exemption is what makes the publish-check gate correct, so a change that drops it must fail |
| 37 | +# here rather than leave this file asserting a preview that no longer grades. |
| 38 | +# |
| 39 | +# THE EXPRESSIONS ARE PINNED BY EQUALITY, and the named needles below are RESTATEMENTS, not the |
| 40 | +# defence — the same division `test_pin_contract.sh` settled on, for the same reason. A needle |
| 41 | +# proves a term is PRESENT and nothing about how it is JOINED, so a scan built only from needles |
| 42 | +# passes on gates that mean the opposite of what it claims to pin: `needs.grade.result != |
| 43 | +# 'skipped' || inputs.check_run || !cancelled()` contains every positive needle and none of the |
| 44 | +# forbidden one, yet `!cancelled()` alone then boots a `checks: write` runner on every |
| 45 | +# non-cancelled run — the skipped-needs spin above. `!inputs.check_run` likewise satisfies an |
| 46 | +# `inputs.check_run` needle while inverting the opt-in. Only the equality sees the operators. The |
| 47 | +# needles stay because they name WHICH invariant broke instead of only printing two strings. |
| 48 | +# |
| 49 | +# Editing a gate on purpose is therefore a deliberate two-place edit — the workflow and the |
| 50 | +# expected string here — whose diff a reviewer sees. That is the property, not an obstacle. |
| 51 | +# |
| 52 | +# THE GATE IS NOT THE ONLY PLACE `enabled` COULD COME BACK. A step-level `if:` on `Create the |
| 53 | +# Check Run(s)` reproduces the dispatch-preview-publishes-nothing mode exactly, with a job gate |
| 54 | +# that still reads correctly. So the forbidden `needs.gate.outputs.enabled` is scanned for over |
| 55 | +# the WHOLE publish-check job body, not just its `if:`. And the gate's safety rests on `grade` |
| 56 | +# being in `needs:` — drop it and `needs.grade.result` resolves to the empty string, `'' != |
| 57 | +# 'skipped'` is true, and the job both stops waiting for grade and runs on every non-cancelled |
| 58 | +# run. That membership is asserted too. |
| 59 | +# |
| 60 | +# Extraction is by anchor, so every scan self-checks that it matched something — a stale anchor |
| 61 | +# fails loudly rather than printing a vacuous `ok` over an empty string. |
| 62 | +# |
| 63 | +# bash tests/test_job_gates.sh # exit 0 = all green |
| 64 | +set -uo pipefail |
| 65 | + |
| 66 | +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
| 67 | +WF="$SELF_DIR/../../../.github/workflows/pr-risk.yml" |
| 68 | +[ -f "$WF" ] || { echo "FATAL: $WF not found" >&2; exit 1; } |
| 69 | + |
| 70 | +PASS=0; FAIL=0 |
| 71 | +ok() { PASS=$((PASS+1)); printf 'ok %s\n' "$1"; } |
| 72 | +bad() { FAIL=$((FAIL+1)); printf 'FAIL %s\n got: %s\n' "$1" "${2:-}"; } |
| 73 | +eq() { if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (expected '$2')" "$3"; fi } |
| 74 | +# `grep -F`, not `case` globbing: these needles are workflow-expression syntax (`!cancelled()`, |
| 75 | +# `needs.gate.outputs.enabled`), whose parens and dots a glob or a regex would reinterpret into |
| 76 | +# something laxer. |
| 77 | +has() { if printf '%s\n' "$2" | grep -qF -- "$3"; then ok "$1"; else bad "$1" "not found: $3"; fi } |
| 78 | +no() { if printf '%s\n' "$2" | grep -qF -- "$3"; then bad "$1" "present: $3"; else ok "$1"; fi } |
| 79 | + |
| 80 | +# --- exactly one of each job, which is what makes the extraction below meaningful -------------- |
| 81 | +# The slicer keys on a 2-space-indented job key. A second job of the same name is not valid YAML, |
| 82 | +# but a RENAME that leaves the old key behind, or a copy-pasted block, would have it silently |
| 83 | +# read the FIRST copy while the runner obeys the last. Count first. |
| 84 | +count_job() { grep -cE "^ $1:[[:space:]]*(#.*)?\$" "$WF"; } |
| 85 | +eq "exactly one grade: job" 1 "$(count_job grade)" |
| 86 | +eq "exactly one publish-check: job" 1 "$(count_job 'publish-check')" |
| 87 | + |
| 88 | +# --- the whole body of a named job, WHOLE-LINE comments stripped ------------------------------- |
| 89 | +# From the job's own 2-space key to the NEXT 2-space key (or EOF). Bounding on the next job — not |
| 90 | +# on this job's `name:`, as an earlier draft did — is what keeps a rename or a dropped `name:` |
| 91 | +# from running the slice on into the following job, where a needle satisfied by somebody else's |
| 92 | +# gate reads as green. Whole-line comments go because the rationale above `publish-check`'s gate |
| 93 | +# QUOTES the very anti-pattern this file forbids, so a comment-blind scan would fail on the prose |
| 94 | +# explaining why the code is right. |
| 95 | +# |
| 96 | +# The comment strip must come BEFORE the terminator, not after: a 2-space `# ...` line is not a |
| 97 | +# job key, and this workflow separates its jobs with sixteen-line comment blocks at exactly that |
| 98 | +# indent. Terminating on one would end the slice early — harmless between jobs, but a truncated |
| 99 | +# body is a VACUOUS pass for the whole-job `enabled` scan below, which is the half that has no |
| 100 | +# equality behind it. |
| 101 | +job_body() { |
| 102 | + awk -v job="$1" ' |
| 103 | + $0 ~ "^ " job ":[[:space:]]*(#.*)?$" { injob = 1; next } |
| 104 | + !injob { next } |
| 105 | + /^[[:space:]]*#/ { next } |
| 106 | + /^ [^[:space:]]/ { injob = 0; next } |
| 107 | + { print } |
| 108 | + ' "$WF" |
| 109 | +} |
| 110 | + |
| 111 | +# --- that job's `if:` expression, folded to one line, the key and block indicator stripped ----- |
| 112 | +# Keep the `if:` line and its continuations, then squash all whitespace to single spaces. The |
| 113 | +# squash is load-bearing, not cosmetic: a `>-` scalar splits the expression across lines, so a |
| 114 | +# multi-token needle would never match the raw text and every negative assertion below would pass |
| 115 | +# vacuously. A BLANK line is a continuation, not a terminator — a blank inside a `>-` scalar is |
| 116 | +# legal YAML, and treating it as the end would silently truncate the expression to a prefix and |
| 117 | +# let the equality and the `no` needles pass over the half they never saw. Only a real 4-space |
| 118 | +# key ends the block. |
| 119 | +job_if() { |
| 120 | + job_body "$1" | awk ' |
| 121 | + /^ if:/ { inif = 1; print; next } |
| 122 | + !inif { next } |
| 123 | + /^[[:space:]]*$/ { next } |
| 124 | + /^ / { print; next } |
| 125 | + { inif = 0 } |
| 126 | + ' | tr '\n' ' ' | tr -s '[:space:]' ' ' \ |
| 127 | + | sed -E 's/^[[:space:]]*if:[[:space:]]*(>-|>|\|-|\|)?[[:space:]]*//; s/[[:space:]]+$//' |
| 128 | +} |
| 129 | + |
| 130 | +# --- that job's `needs:`, normalized to a bare comma list -------------------------------------- |
| 131 | +# Flow style (`needs: [gate, grade]`) and block style (`- gate` on following lines) both reduce to |
| 132 | +# `gate,grade`, so a pure reformat of the key does not turn into a spurious red here. |
| 133 | +job_needs() { |
| 134 | + job_body "$1" | awk ' |
| 135 | + /^ needs:/ { sub(/^ needs:[[:space:]]*/, ""); v = $0; inneeds = 1; next } |
| 136 | + inneeds && /^ *- / { sub(/^[[:space:]]*-[[:space:]]*/, ""); v = v (v == "" ? "" : ",") $0; next } |
| 137 | + inneeds { exit } |
| 138 | + END { gsub(/[][[:space:]]/, "", v); print v } |
| 139 | + ' |
| 140 | +} |
| 141 | +needs_has() { case ",$2," in *",$3,"*) ok "$1" ;; *) bad "$1" "needs: ${2:-<none>}" ;; esac } |
| 142 | + |
| 143 | +GRADE_IF="$(job_if grade)" |
| 144 | +PUBLISH_IF="$(job_if 'publish-check')" |
| 145 | +PUBLISH_BODY="$(job_body 'publish-check')" |
| 146 | + |
| 147 | +# Coverage self-checks. Without these, a slicer whose anchors went stale returns "" and every |
| 148 | +# `has` below reports a real failure while every `no` reports a fake success — the worse half. |
| 149 | +if [ -n "$GRADE_IF" ]; then ok "the grade if: was extracted ($GRADE_IF)" |
| 150 | +else bad "the grade if: was extracted" "empty — the slicer's anchors are stale"; fi |
| 151 | +if [ -n "$PUBLISH_IF" ]; then ok "the publish-check if: was extracted ($PUBLISH_IF)" |
| 152 | +else bad "the publish-check if: was extracted" "empty — the slicer's anchors are stale"; fi |
| 153 | +if [ -n "$PUBLISH_BODY" ]; then ok "the publish-check job body was extracted" |
| 154 | +else bad "the publish-check job body was extracted" "empty — the slicer's anchors are stale"; fi |
| 155 | +# The slice must stop at the job it belongs to. Assert it on `grade`, the one with a successor: |
| 156 | +# publish-check is last, so its own slice running long has nothing to run into and would prove |
| 157 | +# nothing. `needs.grade.result` appears only in publish-check, so seeing it inside grade's body |
| 158 | +# means the two jobs were concatenated — the exact failure a `name:`-bounded slicer had. |
| 159 | +no "the grade slice stops before the next job" "$(job_body grade)" "needs.grade.result" |
| 160 | +no "grade's if: did not absorb publish-check's" "$GRADE_IF" "inputs.check_run" |
| 161 | + |
| 162 | +# --- THE PINS. The expressions verbatim; everything below them only names what broke ----------- |
| 163 | +eq "the grade gate is exactly what it should be" \ |
| 164 | + "needs.gate.outputs.enabled == 'true' || github.event_name == 'workflow_dispatch'" \ |
| 165 | + "$GRADE_IF" |
| 166 | +eq "the publish-check gate is exactly what it should be" \ |
| 167 | + "needs.grade.result != 'skipped' && inputs.check_run && !cancelled()" \ |
| 168 | + "$PUBLISH_IF" |
| 169 | + |
| 170 | +# --- grade: `enabled` OR a manual dispatch ---------------------------------------------------- |
| 171 | +# PR #116's exemption. Dropping the dispatch arm would make `enabled: false` a lockout rather than |
| 172 | +# a mute and delete the staged rollout the input exists for; dropping the `enabled` arm would make |
| 173 | +# every enrolled repo grade whatever the switch says. |
| 174 | +has "grade is gated on the resolved enabled flag" "$GRADE_IF" "needs.gate.outputs.enabled == 'true'" |
| 175 | +has "grade exempts a manual workflow_dispatch" "$GRADE_IF" "github.event_name == 'workflow_dispatch'" |
| 176 | +needs_has "grade waits on gate" "$(job_needs grade)" "gate" |
| 177 | + |
| 178 | +# --- publish-check: follows GRADE, never `enabled` -------------------------------------------- |
| 179 | +# `grade` in `needs:` is what gives `needs.grade.result` a value at all. Without it the term |
| 180 | +# resolves to the empty string, `'' != 'skipped'` is TRUE, and the gate silently degenerates into |
| 181 | +# the `!cancelled()`-only form this file exists to forbid — with every needle below still green. |
| 182 | +needs_has "publish-check waits on grade" "$(job_needs 'publish-check')" "grade" |
| 183 | +has "publish-check is gated on grade having run" "$PUBLISH_IF" "needs.grade.result != 'skipped'" |
| 184 | +has "publish-check still honours the opt-in input" "$PUBLISH_IF" "inputs.check_run" |
| 185 | +no "publish-check does not INVERT the opt-in" "$PUBLISH_IF" "!inputs.check_run" |
| 186 | +has "publish-check still runs on a failed grade" "$PUBLISH_IF" '!cancelled()' |
| 187 | +has "publish-check joins its terms with AND" "$PUBLISH_IF" "&&" |
| 188 | +no "publish-check joins no term with OR" "$PUBLISH_IF" "||" |
| 189 | + |
| 190 | +# THE REGRESSION THIS FILE EXISTS FOR. `needs.gate.outputs.enabled` anywhere in this job is the |
| 191 | +# dispatch-preview-publishes-nothing failure mode: it re-imposes the automatic stream's switch on |
| 192 | +# a manual run that grades, labels and comments regardless. Scanned over the WHOLE job, because a |
| 193 | +# step-level `if:` on the publishing step reproduces it exactly while the job gate still reads |
| 194 | +# right — and the job gate is the only thing an `if:`-only scan can see. |
| 195 | +no "publish-check is NOT gated on enabled" "$PUBLISH_IF" "needs.gate.outputs.enabled" |
| 196 | +no "and no step inside publish-check re-imposes it either" "$PUBLISH_BODY" "needs.gate.outputs.enabled" |
| 197 | + |
| 198 | +# THE OTHER HALF, and it is a CONJUNCTION rather than a needle: `inputs.check_run && !cancelled()` |
| 199 | +# is a legitimate SUBSTRING of the correct gate, so its presence proves nothing on its own. What |
| 200 | +# must never happen is a status function standing WITHOUT a `needs.grade.result` test — the |
| 201 | +# skipped-needs runner spin, which no script suite and no green workflow run would ever reveal. |
| 202 | +if printf '%s\n' "$PUBLISH_IF" | grep -qE -- '!?(cancelled|always|success|failure)\(\)' \ |
| 203 | + && ! printf '%s\n' "$PUBLISH_IF" | grep -qF -- 'needs.grade.result'; then |
| 204 | + bad "publish-check never pairs a status function with no needs.grade.result test" "$PUBLISH_IF" |
| 205 | +else |
| 206 | + ok "publish-check never pairs a status function with no needs.grade.result test" |
| 207 | +fi |
| 208 | + |
| 209 | +printf '\n%s passed, %s failed\n' "$PASS" "$FAIL" |
| 210 | +[ "$FAIL" -eq 0 ] |
0 commit comments