tag_deploy workflows hardcode centos7/centos8 RPM builds; move the OS matrix into release_rpms.yml (el8/el9/el10)

[silug]

Summary

#80 fixed release_rpms.yml to pull builders from ghcr.io/simp/simp-<os>-build with build_container_os defaulting to el8. That rolled out to ~70 repos — but RPM builds are still failing fleet-wide, because the workflows that dispatch release_rpms.yml override the default with a hardcoded centos7/centos8 matrix.

Every tag push therefore fires two builds, both explicitly requesting retired container OSes, and both die on:

NameError: uninitialized constant JSON::Fragment

#80's new default is never reached.

Root cause

All four caller templates hardcode the same matrix:

Template	matrix lines	dispatch line
modules/profile/files/pupmod/_github/workflows/tag_deploy.yml	136–139	160
modules/profile/files/_github/workflows/tag_deploy_github-rpms.yml	110–113	134
modules/profile/files/_github/workflows/tag_deploy_github-rpms-el7-el8.yml	107–110	131
modules/profile/files/_github/workflows/tag_deploy_rubygem__github-rpms.yml	177–180	201

    strategy:
      matrix:
        os:
          - centos7
          - centos8
    steps:
      - name: Trigger RPM release workflow (${{ matrix.os }})
        ...
                build_container_os: '${{ matrix.os }}'

Evidence

14 release_rpms.yml runs since 2026-07-28 — every one failed, exactly 2 per tag:

Date	Repo	Runs
2026-07-28	pupmod-simp-auditd	2, both failed
2026-07-28	pupmod-simp-simp	2, both failed
2026-07-29	pupmod-simp-pam	2, both failed
2026-07-29	pupmod-simp-sssd	2, both failed
2026-07-31	pupmod-simp-simplib	2, both failed
2026-08-10	pupmod-simp-svckill	2, both failed
2026-08-12	pupmod-voxpupuli-selinux	2, both failed

Sample job names, showing the override in action on repos that do have #80's fix:

• simp/pupmod-simp-svckill 5.1.0 (build os: centos7) — its release_rpms.yml has default: 'el8' and the ghcr image line
• simp/pupmod-simp-simplib 7.0.0 (build os: centos8) — same
• simp/pupmod-voxpupuli-selinux simp-6.0.0 (build os: centos8) — this repo has not received Pull RPM build containers from ghcr.io/simp/simp-<os>-build (fixes fleet-wide RPM build failures) #80 yet, so it is failing for both reasons

Proposed fix — drop the caller matrix, move it into release_rpms.yml

Decision (2026-08-12): build el8, el9, and el10 RPMs per tag, and express that in exactly one place.

Two goals that pull in opposite directions unless the matrix moves:

• the build-OS list must not be duplicated across 4 templates × 79 repos, and
• one tag must produce three RPMs.

A caller-side matrix can only satisfy the second by reintroducing the first. So:

1. Callers dispatch once and say nothing about the OS. Remove the strategy: block and drop build_container_os from the dispatch inputs in all four templates. The matrix is referenced 5 times in pupmod/.../tag_deploy.yml but only one is functional:

Line	Use	Kind
136-139	strategy.matrix.os: [centos7, centos8]	definition
141	step name Trigger RPM release workflow (${{ matrix.os }})	cosmetic
149	console.log(... for os '${{ matrix.os }}')	cosmetic
160	build_container_os: '${{ matrix.os }}'	functional
163	console.log(... from ${{ matrix.os }}: status ...)	cosmetic

2. release_rpms.yml gains the matrix. It currently has a single job, create-and-attach-rpms-to-github-release (line 93), with no strategy: — so add one driven by a single input, e.g.:

      build_container_oses:
        description: "Build container OSes (JSON list)"
        required: false
        default: '["el8","el9","el10"]'
...
jobs:
  create-and-attach-rpms-to-github-release:
    strategy:
      matrix:
        os: ${{ fromJSON(github.event.inputs.build_container_oses) }}

That keeps one source of truth, still lets a manual run narrow the set ('["el9"]'), and makes "which EL versions do we ship RPMs for" a one-line answer.

RPM filenames do not collide — dist tags .el8 / .el9 / .el10 differ — so all three attach to the same release cleanly.

Split the job in two while adding the matrix

Both of the fan-out hazards below come from the same structural issue: the single job
create-and-attach-rpms-to-github-release mixes per-release work (resolve/create the release,
optionally wipe its assets) with per-OS work (build, sign, upload). Harmless while it runs once;
racy the moment it runs three times. Rather than guard each symptom, separate the concerns:

jobs:
  resolve-release:            # runs ONCE
    # - GET release by tag, create on 404 when autocreate_release == 'yes'
    # - wipe assets when clean == 'yes'
    outputs:
      release_id: ...

  build-and-attach:           # matrix: el8, el9, el10
    needs: [resolve-release]
    strategy:
      matrix:
        os: ${{ fromJSON(github.event.inputs.build_container_oses) }}
    # - build + sign + upload for one OS, using needs.resolve-release.outputs.release_id

This is barely more work than adding the matrix to the existing job, and it removes both hazards
structurally instead of patching them:

• clean: 'yes' asset-wipe race. The "Wipe all previous assets from GitHub Release" step
  (line 264) deletes all assets on the release. With three parallel legs, one can wipe what
  another just uploaded. The alternative to splitting — gating the wipe with
  if: matrix.os == 'el8' — is fragile: it silently stops running the moment someone narrows the
  matrix to '["el9"]'.
• Autocreate race. The release-api step (lines 124-198) GETs the release by tag and creates
  it on 404. Three parallel legs against a missing release all attempt the create; two lose. The
  alternative to splitting is 422 already_exists retry logic in the script.

Severity note: neither hazard is reachable from the tag-triggered path — callers pass
clean: 'no', and create-github-release runs first via needs:, so the release always exists.
Both only bite manual release_rpms.yml dispatches. That makes this important but not urgent; the
reason to do it now is that the matrix change touches this job anyway, and clean: 'yes' deleting
release assets is a poor footgun to leave armed.

Also required: make the inputs non-required

build_container_os and build_container_tag are currently required: true (with defaults). Once
callers omit them, behaviour depends on GitHub applying defaults to omitted workflow_dispatch
inputs. It does, and required is effectively a UI-form concern — but if that were ever untrue,
every tag-triggered release would fail at the dispatch call, which is worse than today (no run
at all rather than a visibly failing one). Flip both to required: false; they have defaults, so
nothing else changes.

Consequence to state plainly

Output changes from two RPMs per tag (centos7, centos8) to three (el8, el9, el10). Intended — but worth noting so the change in artifact count is not read as a regression either way.

Also settle while here

tag_deploy_github-rpms-el7-el8.yml is EL7/EL8-specific by name. With EL7 gone fleet-wide, decide whether it still has a purpose or should be retired rather than updated.

Scope

79 repos have a tag_deploy.yml naming centos7/centos8 — all of them with exactly that pair. Because this is template-owned, it is a sync rollout, not per-repo edits.

Notes

• Reconcile the pupmod baseline templates with the deployed fleet and re-assert it (#88) #42 touches pupmod/_github/workflows/tag_deploy.yml but does not fix this — it only covers pdk → rake pupmod:build, action/Ruby version bumps, and module-archive artifact upload. No matrix change.
• Once fixed, the affected releases above should be re-buildable by re-dispatching release_rpms.yml — no re-tagging required, since the tags already exist.
• Immediate workaround for anyone releasing before this lands: dispatch release_rpms.yml manually with build_container_os=el8 rather than relying on the tag-triggered path.

Affected repos (79)

• pupmod-simp-acpid
• pupmod-simp-aide
• pupmod-simp-at
• pupmod-simp-auditd
• pupmod-simp-autofs
• pupmod-simp-clamav
• pupmod-simp-compliance_markup
• pupmod-simp-cron
• pupmod-simp-crypto_policy
• pupmod-simp-dconf
• pupmod-simp-deferred_resources
• pupmod-simp-dhcp
• pupmod-simp-ds389
• pupmod-simp-fips
• pupmod-simp-freeradius
• pupmod-simp-gdm
• pupmod-simp-gnome
• pupmod-simp-haveged
• pupmod-simp-ima
• pupmod-simp-iptables
• pupmod-simp-issue
• pupmod-simp-krb5
• pupmod-simp-libreswan
• pupmod-simp-libvirt
• pupmod-simp-logrotate
• pupmod-simp-mate
• pupmod-simp-mockup
• pupmod-simp-mozilla
• pupmod-simp-named
• pupmod-simp-nfs
• pupmod-simp-oath
• pupmod-simp-oddjob
• pupmod-simp-openscap
• pupmod-simp-pam
• pupmod-simp-pki
• pupmod-simp-polkit
• pupmod-simp-postfix
• pupmod-simp-pupmod
• pupmod-simp-resolv
• pupmod-simp-rsync
• pupmod-simp-rsyslog
• pupmod-simp-selinux
• pupmod-simp-simp_apache
• pupmod-simp-simp_authselect
• pupmod-simp-simp_banners
• pupmod-simp-simp_ds389
• pupmod-simp-simp_firewalld
• pupmod-simp-simp_gitlab
• pupmod-simp-simp_grub
• pupmod-simp-simpkv
• pupmod-simp-simplib
• pupmod-simp-simp_nfs
• pupmod-simp-simp_options
• pupmod-simp-simp_rsyslog
• pupmod-simp-simp_snmpd
• pupmod-simp-simp
• pupmod-simp-site
• pupmod-simp-ssh
• pupmod-simp-sssd
• pupmod-simp-stunnel
• pupmod-simp-sudo
• pupmod-simp-svckill
• pupmod-simp-swap
• pupmod-simp-tftpboot
• pupmod-simp-tlog
• pupmod-simp-tpm2
• pupmod-simp-tuned
• pupmod-simp-useradd
• pupmod-simp-vnc
• pupmod-simp-vsftpd
• pupmod-simp-x2go
• pupmod-voxpupuli-selinux
• puppet-gpasswd
• simp-doc
• simp-environment-skeleton
• simp-gpgkeys
• simp-rsync-skeleton
• simp-selinux-policy
• simp-utils

[silug]

Decision: drop the matrix entirely rather than retarget it, so the build OS lives in exactly one place (release_rpms.yml's input default) instead of being duplicated in every caller.

The matrix is only load-bearing in one spot

Five references in pupmod/_github/workflows/tag_deploy.yml, four of them cosmetic:

Line	Use	Kind
136–139	strategy.matrix.os: [centos7, centos8]	definition
141	step name Trigger RPM release workflow (${{ matrix.os }})	cosmetic
149	console.log(... for os '${{ matrix.os }}')	cosmetic
160	build_container_os: '${{ matrix.os }}'	functional
163	console.log(... from ${{ matrix.os }}: status ...)	cosmetic

So the change is small:

     env:
       TARGET_REPO: ${{ github.repository }}
-    strategy:
-      matrix:
-        os:
-          - centos7
-          - centos8
     steps:
-      - name: Trigger RPM release workflow (${{ matrix.os }})
+      - name: Trigger RPM release workflow
         uses: actions/github-script@v6
         ...
         script: |
-            console.log( `== Building tag: '${ process.env.TARGET_TAG }' for os '${{ matrix.os}}'` )
+            console.log( `== Building tag: '${ process.env.TARGET_TAG }'` )
             ...
               inputs: {
                 release_tag: process.env.TARGET_TAG,
                 clean: 'no',
                 clobber: 'yes',
-                build_container_os: '${{ matrix.os }}'
               }
             }).then((result) => {
-              console.log( `== Submitted workflow dispatch to build RPMs from ${{ matrix.os }}: status ${result.status}` )
+              console.log( `== Submitted workflow dispatch to build RPMs: status ${result.status}` )
             })

Same treatment for the other three caller templates listed above.

⚠️ Do this in the same change: make the inputs non-required

release_rpms.yml currently declares both as required: true:

      build_container_os:
        description: "Build container OS"
        required: true
        default: 'el8'
      build_container_tag:
        description: "Build container tag"
        required: true
        default: 'latest'

The whole point of dropping the matrix is to omit build_container_os from the POST .../dispatches payload and let the default apply. GitHub does apply input defaults for omitted workflow_dispatch inputs, and required is effectively a UI-form concern — but if that is ever not the case, every tag-triggered release breaks at the dispatch call, which is a worse failure than the current one (no run at all, rather than a failing run).

Cheap insurance: flip both to required: false in the same PR. They have defaults, so nothing else changes, and the behaviour no longer depends on how GitHub interprets required for API dispatches.

Recommend validating with one real tag (or a manual release_rpms.yml dispatch that omits the input) before syncing to all 79 repos.

Consequence to be aware of

This changes the output, not just the plumbing: tags currently produce two RPM builds (centos7 + centos8); afterwards they produce one (el8). That is the intended outcome, but worth stating so nobody treats the missing second artifact as a regression. If multi-EL RPMs are wanted later, the right place to express that is a matrix inside release_rpms.yml — one source of truth — not back in the callers.

Also settle while here

tag_deploy_github-rpms-el7-el8.yml is EL7/EL8-specific by name. With EL7 gone fleet-wide, decide whether it still has a purpose or should be retired rather than updated.

[silug]

Revised above (2026-08-12): the target is el8 + el9 + el10 RPMs per tag, not el8 alone.

That changes the shape of the fix from my earlier comment. Dropping the caller matrix and building three OSes are only compatible if the matrix moves rather than disappears — a caller-side matrix would reintroduce exactly the duplication we are removing. So:

• callers dispatch once and omit build_container_os entirely;
• release_rpms.yml grows the matrix, driven by one input (build_container_oses, default ["el8","el9","el10"]), so the shipped-EL-versions list has a single home and a manual run can still narrow it.

The required: true → required: false point from my earlier comment still stands. Two additional hazards that only appear once the job fans out three ways are now in the issue body: the clean: 'yes' asset-wipe race (parallel legs can wipe each other's uploads) and the autocreate-release race on manual dispatches against a tag with no release. Neither affects the tag-triggered path, but both are easy to trip manually.

Superseded: the "just use el8" recommendation in my previous comment.

[silug]

Folded the two fan-out hazards into a single split the job in two item in the body, rather than listing them as separate patches.

Both come from the same place: create-and-attach-rpms-to-github-release does per-release work (resolve/create, wipe) and per-OS work (build, sign, upload) in one job. Splitting into resolve-release (runs once, emits release_id) + build-and-attach (the el8/el9/el10 matrix, needs: the first) removes both without defensive code — and avoids an if: matrix.os == 'el8' guard on the wipe that would silently stop firing if the matrix is ever narrowed.

The wider question this raised — that release_rpms.yml exists as 77 independent copies, which is why #80 fixed one half of this bug and left the other — is now tracked separately.