Consider reusable workflows (workflow_call) instead of copying 13 workflow templates into ~79 repos

[silug]

Problem

The GitHub Actions workflows puppetsync distributes are copied into every repo, not called from a central place. release_rpms.yml alone exists as 77 independent copies. Every change to shared CI is therefore a fleet-wide sync, and a fix is only as complete as the sync that carried it.

That distribution model is not merely inconvenient — it has already produced a real, months-long outage. #80 fixed release_rpms.yml to pull builders from ghcr.io/simp/simp-<os>-build with build_container_os defaulting to el8. But the caller, tag_deploy.yml, still dispatched a hardcoded [centos7, centos8] matrix, so the new default was never reached and RPM builds kept failing on stale images (#84). One half of a two-file change landed; the other did not. With a single central definition, that class of partial-rollout bug cannot occur.

Concretely, the currently distributed workflow templates are:

modules/profile/files/_github/workflows/
  add_new_issue_to_triage_project.yml   release_rpms.yml
  tag_deploy_github-only.yml            tag_deploy_github-rpms.yml
  tag_deploy_github-rpms-el7-el8.yml    tag_deploy_rubygem.yml
  tag_deploy_rubygem__github-only.yml   tag_deploy_rubygem__github-rpms.yml
  validate_tokens_asset.yml
modules/profile/files/pupmod/_github/workflows/
  pr_tests.yml   pr_tests.simp-simp.yml   tag_deploy.yml   validate_tokens.yml

Thirteen templates × up to ~79 repos, each copy independently driftable. #41 documents the same failure mode from the other direction: the pr_tests.yml template and its deployed copies have diverged so far that the ruby-style job is disabled in the template but running (and ignoring failures) in the modules.

Proposal

Move the substantive logic into reusable workflows (on: workflow_call) hosted in one repo, and reduce each module's file to a thin caller:

# .github/workflows/release_rpms.yml in each module — the entire file
name: 'RELENG: Build + attach RPMs to GitHub Release'
on:
  workflow_dispatch:
    inputs: { release_tag: { required: true }, ... }
jobs:
  call:
    uses: simp/<central-repo>/.github/workflows/release_rpms.yml@v1
    with:
      release_tag: ${{ github.event.inputs.release_tag }}
    secrets: inherit

A fix then becomes a tag bump on the central workflow (or nothing at all, if callers track a moving @v1), rather than a 77-repo sync.

This is a genuine strategic question, not an obvious cleanup

Reasons to be cautious, stated up front:

• It reduces puppetsync's remit. Workflow templates are a large part of what puppetsync currently manages. If workflows centralize, puppetsync's job shrinks toward Gemfile/.rubocop.yml/.puppet-lint.rc/metadata — worth deciding deliberately alongside Build inventory dynamically from the GitHub API instead of hand-maintained repolists #55, Scheduled org-wide puppetsync runs (continuous baseline enforcement) #56 and Keep metadata.json dependencies up-to-date dynamically via the Forge API (single lookup pass) #68, which are all pushing puppetsync toward being a continuously-running service.
• The pattern is not established here. An org-wide search finds essentially no workflow_call usage (only simp-core's build_container.yml) and zero repos consuming a central simp/github-action-* reusable workflow. This would be a new convention, not an extension of an existing one.
• secrets: inherit and permissions need thought. These workflows use SIMP_DEV_GPG_SIGNING_KEY*, SIMP_CORE_REF_FOR_BUILDING_RPMS, SIMP_AUTO_GITHUB_TOKEN__REPO_SCOPE, and PUPPETFORGE_API_TOKEN. Reusable workflows can inherit secrets, but the trust boundary changes and should be reviewed rather than assumed.
• Versioning policy matters. A floating @v1 gives instant fleet-wide fixes and instant fleet-wide breakage; a pinned tag restores deliberate rollout but reintroduces a (much cheaper) bump-everywhere step. Renovate can manage the pinned form.
• Not everything should move. Per-repo matrices, nodeset lists, and module-specific overrides (e.g. pr_tests.simp-simp.yml, which exists precisely because simp/simp needs different parallelism) still belong locally. The split between "shared logic" and "local configuration" needs drawing before any migration.

Suggested approach

1. Pick one workflow as a pilot — release_rpms.yml is the natural candidate: highest copy count, most recent partial-rollout failure, and no per-repo variation.
2. Land tag_deploy workflows hardcode centos7/centos8 RPM builds; move the OS matrix into release_rpms.yml (el8/el9/el10) #84 first on the copied version, so RPM builds are working before the plumbing changes underneath them.
3. Convert it to workflow_call in a central repo, migrate a handful of modules, confirm signing and asset upload still work end to end.
4. Decide floating vs pinned versioning, then let puppetsync sync the thin callers.
5. Revisit the remaining twelve templates individually — several may be better retired than migrated (tag_deploy_github-rpms-el7-el8.yml is EL7/EL8-specific and probably dead; see tag_deploy workflows hardcode centos7/centos8 RPM builds; move the OS matrix into release_rpms.yml (el8/el9/el10) #84).

Related

• tag_deploy workflows hardcode centos7/centos8 RPM builds; move the OS matrix into release_rpms.yml (el8/el9/el10) #84 — the partial rollout that motivated this
• Reconcile the Ruby Style (rubocop) job in the pr_tests.yml baseline — template and deployed modules have diverged #41 — the same drift problem in pr_tests.yml (template vs deployed)
• Build inventory dynamically from the GitHub API instead of hand-maintained repolists #55, Scheduled org-wide puppetsync runs (continuous baseline enforcement) #56, Keep metadata.json dependencies up-to-date dynamically via the Forge API (single lookup pass) #68 — puppetsync's evolving remit