ExaDev GitHub Actions Runner

An Ansible collection, exadev.github_runner, for a self-hosted GitHub Actions runner fleet: GitHub's own Actions Runner Controller (ARC) on a k3s cluster that spans Apple Silicon (arm64) and Intel (amd64) hosts on different networks, joined over Tailscale or Headscale. It also holds the runner image and the fleet-health platform's images. A fleet's own inventory, host variables and Helm values live in that fleet's repository, which installs this collection; ExaDev's is the private ExaDev/github-runner-fleet.

Getting started

There are two ways to deploy. A fleet's hosts are managed from a control machine with Ansible, from the fleet's own inventory (see Ansible below). bootstrap.sh, described here, brings up a single machine from a checkout of this repository without a control node or inventory. It runs the same Ansible role against the machine itself, so both paths deploy identically.

Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in roles/github_runner_cluster/README.md), python3, jq, openssl, and gh. On macOS, the playbook installs helm, kubectl and Docker through Homebrew itself. Elsewhere, install helm and kubectl from the host's package manager first.

1. Create a GitHub App on the organisation with Self-hosted runners: read & write permission, and install it on the organisation. playbooks/github_app_setup.yml does this through GitHub's manifest flow (see roles/github_runner_arc/README.md).
2. Configure the environment. Copy .env.bootstrap.example to .env.bootstrap and fill in the following (the file's own comments cover the rest, such as joining an existing cluster or running as an agent):
   • K3S_TOKEN: any random string (openssl rand -hex 32). It authenticates node-join inside the cluster only; it has no relation to GitHub.
   • K3S_VPN_AUTH_JOIN_KEY, K3S_TLS_SAN_LIST: see Tailscale below.
   • RUNNER_ORG, RUNNER_APP_ID, RUNNER_APP_PRIVATE_KEY_PATH: the organisation, the App ID and a path to its private key PEM file (for example under ./secrets/, already gitignored).
   • RUNNER_APP_INSTALLATION_ID: optional. Leave it blank and the role resolves it automatically from the App ID and key.
   • RUNNER_IMAGE, and either RUNNER_VALUES_FILE (a Helm values template) or RUNNER_MAX_RUNNERS (the ARC role's own template with that many runners at most).
   • GHCR_PULL_USERNAME, GHCR_PULL_TOKEN: a registry credential, such as a GHCR personal access token with read:packages scope, to pull the runner image and the fleet-health platform's own images.
   • HEARTBEAT_GH_TOKEN, HEARTBEAT_GIST_ID: see Heartbeat below.
3. Build and push the runner images. A fleet spanning both arm64 and amd64 hosts needs a real multi-arch push, not a plain docker build: docker buildx build --platform linux/arm64,linux/amd64 --push -t ghcr.io/exadev/github-runner:latest . for the main runner image (see Build, test, and smoke-test for why this is currently the only working path, not just a manual fallback), plus the same for heartbeat/Dockerfile → ghcr.io/exadev/github-runner-heartbeat:latest and autoscaler/Dockerfile → ghcr.io/exadev/github-runner-autoscaler:latest - both now run as in-cluster pods that can land on either architecture, so both need pulling from either too.
4. Bootstrap everything:

   ./bootstrap.sh

   This installs Ansible into a repo-local virtualenv (.ansible-venv/, gitignored), turns .env.bootstrap into role variables, and runs playbooks/site.yml against this machine over a local connection. The role starts k3s, waits for it to be ready, installs the ARC controller and the fleet-health platform (heartbeat + autoscaler, as in-cluster Deployments), resolves the App installation ID if needed, creates the GitHub App and GHCR pull secrets, and installs the org's runner scale set. It is safe to rerun at any time, for example after rotating the App's private key. Extra arguments pass through to ansible-playbook, so ./bootstrap.sh --check is a dry run. The role keeps the node's Compose project in ~/.github-runner (GITHUB_RUNNER_CLUSTER_DIR changes it) and writes a .env there on every run, which is why bootstrap.sh reads its own input from .env.bootstrap instead. Multiple orgs and multiple scale-set profiles per org are configured through Ansible host_vars (see below), not bootstrap.sh.

Ansible

The repository is laid out as an Ansible collection, exadev.github_runner (see galaxy.yml): its roles live in roles/ and its playbooks in playbooks/: site.yml brings up the whole fleet, arc.yml installs only ARC into a cluster that already exists (from the control machine, with a kubeconfig), and github_app_setup.yml creates the runners' GitHub App through GitHub's manifest flow, and recover_in_cluster_mesh.yml recovers the in-cluster Headscale provider's bootstrap server. A fleet keeps its inventory, host_vars and Helm values in a repository of its own, installs the collection there, and runs the playbooks by name (exadev.github_runner.site); examples/ has inventories to start from, and ExaDev's own fleet is ExaDev/github-runner-fleet. ansible/ holds a generic ansible.cfg and the collections the roles require, for running the playbooks from a checkout of this repository as bootstrap.sh and CI do; playbooks/collections/ links back to the checkout, so a playbook run from it finds exadev.github_runner in place without installing it. Verified end to end against real infrastructure (three hosts, a real single-node-to-3-node etcd HA cutover, real jobs scheduling across all three). bootstrap.sh (see Getting started) is a thin wrapper that runs this same playbook locally.

# in the fleet's own repository, next to its inventory
ansible-galaxy collection install git+https://github.com/ExaDev/github-runner.git
ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 --check   # dry run
ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1           # real run

The playbook runs up to four roles, each with its own README: github_runner_cluster brings the host up as a k3s node, github_runner_arc installs ARC and the fleet-health platform, github_runner_k8s_client (included by the other two) gives a host the kubernetes Python client, and github_runner_secrets_onepassword reads the secrets from 1Password when github_runner_secrets_source is onepassword. The cluster and ARC roles read every secret (K3S_TOKEN, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see examples/ansible-vault/, examples/env/ and examples/onepassword/). With github_runner_secrets_source: onepassword, every secret is read live from one 1Password item at task time; roles/github_runner_secrets_onepassword/README.md lists the item's fields and the op:// references. bootstrap.sh sets the same variables directly from .env.bootstrap. The cluster role ships the node's docker-compose.yml and k3s image build context in roles/github_runner_cluster/files/, copies them on every run into each host's github_runner_cluster_dir (~/.github-runner by default), and templates the .env for their k3s/k3s-agent services there, so a host needs no checkout of this repository. The Compose project name is pinned to github-runner (github_runner_cluster_compose_project) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.

Which hosts are k3s servers is worked out from the inventory, not set per host. The github_runner_cluster inventory group lists the cluster's hosts in order: the first N become servers (control-plane/etcd members running k3s server, genuine voting members of the raft quorum), where N is the largest odd number no greater than both the number of hosts and github_runner_cluster_max_servers (5 by default), so 1 or 2 hosts give 1 server, 3 or 4 give 3, and 5 or more give 5. The first server bootstraps the cluster (--cluster-init, embedded etcd) and every other host joins it, the rest as agents (k3s agent, schedulable capacity with no control-plane role). Join URLs and every server's --tls-san list are derived from each server's node name and github_runner_cluster_tailnet_domain. A host can still set github_runner_cluster_node_role, github_runner_cluster_bootstrap, github_runner_cluster_server_url or github_runner_cluster_tls_sans to override its part, and the run fails with an explanation if the result has an even number of servers, none at all, or more than one bootstrap host; roles/github_runner_cluster/defaults/main.yml describes each. Installing the ARC controller, each org's runner scale set, and the fleet-health platform is not tied to a node role at all: each is driven purely by whether the installing host's own github_runner_arc_orgs/github_runner_arc_heartbeat_gist_id is set, so any server host can be the one Ansible installs against, and moving which host does the installing is a config change (which host's host_vars sets it), not a role change. See the Architecture section for why 3 servers is the minimum sane count for genuine HA (2 is worse than 1 for quorum math, not better). To add a server or agent, add the host to the github_runner_cluster group in the fleet's inventory and give it its own host_vars/<host>.yml; keep the bootstrap host first.

github_runner_arc_orgs[].scale_set_profiles is a list of {suffix, values_file?, node_selector?, runs_on_label?, autoscale?, sizing?} entries (see roles/github_runner_arc/README.md for every key) - one Helm release and one namespace per entry, pooled under the org's own shared runs-on label via scaleSetLabels (not runnerScaleSetName, which is a scale set's own unique GitHub-side registration identity, not just a label - two releases sharing it collide outright) unless a profile sets its own runs_on_label to deliberately opt out of that pool - e.g. a dedicated image-building profile ordinary CI jobs have no business landing on. node_selector is optional per profile, not mandatory: omitting it leaves that profile's pods genuinely unpinned, letting the Kubernetes scheduler place them on whichever node in the fleet actually has room - real failover if one machine goes down, not just capacity pooling. Values files are rendered on the control node from github_runner_arc_values_dir, a directory in the fleet's own repository, so no fleet host needs them. ExaDev's own fleet uses two profiles: an ordinary CI profile unpinned across the whole fleet (sized with Guaranteed QoS for the smallest machine's real capacity), and a second, separately-labelled, Docker-in-Docker profile pinned to one host for building the runner image - a deliberate isolation choice, not a capacity constraint.

Build, test, and smoke-test

• Build the runner image locally (the only working path right now): docker buildx build --platform linux/arm64,linux/amd64 --push -t ghcr.io/exadev/github-runner:latest . from a machine with buildx/QEMU cross-platform support (Docker Desktop has this built in). A plain docker build with no --platform only produces an image for the building machine's own architecture, which silently breaks scheduling on the fleet's other architecture. The fleet-health platform's own two images need the identical treatment, for the identical reason (they're genuinely unpinned pods now too - see Heartbeat/Autoscaler): docker buildx build --platform linux/arm64,linux/amd64 --push -f heartbeat/Dockerfile -t ghcr.io/exadev/github-runner-heartbeat:latest . and the same with autoscaler/Dockerfile → ghcr.io/exadev/github-runner-autoscaler:latest, pull-secret-renewer/Dockerfile → ghcr.io/exadev/github-runner-pull-secret-renewer:latest (the image of the ARC role's App-sourced pull Secret renewal), and node-recovery/Dockerfile → ghcr.io/exadev/github-runner-node-recovery:latest (see Node recovery).
• Build and push via CI: .github/workflows/release.yml's build-images job builds and pushes all five images (this one, heartbeat, autoscaler, pull-secret-renewer and node-recovery) multi-arch on GitHub-hosted ubuntu-latest whenever a release is published - see Releases below.
• Smoke-test the ARC scale set (confirms a job gets a real ephemeral pod): gh workflow run test-arc-runner.yml, then watch kubectl get pods -n arc-runners-<org> -w. A pod must appear only once the job is queued, run to completion, and be deleted within seconds.
• Smoke-test ubuntu-latest routing (confirms runner-fallback-action still reaches GitHub-hosted runners when needed): gh workflow run test-ubuntu-latest.yml.
• Generate load for the autoscaler (see Autoscaler below): gh workflow run test-autoscaler.yml.
• Verify cluster health:

  kubectl get nodes                              # must show every server/agent host as Ready
  kubectl get pods -n actions-runner-controller  # controller pod and the scale set's listener pod both run here
  kubectl get pods -n arc-runners-<org>           # zero runner pods when idle is expected, not a bug

• Confirm no disk accumulates across repeated runner jobs: the Docker host's disk usage (inside the VM, on a host that runs Docker in one) must stay flat across repeated test-arc-runner.yml runs.
• Lint: .github/workflows/ci.yml runs Shellcheck, actionlint, yamllint, hadolint, an Ansible syntax check, ansible-lint (at its production profile), and a collection build check on every push and pull request, plus commitlint (on every commit in a pull request, and on the pull request's own title) on pull requests specifically - all gated by one required-checks job so a repository ruleset only ever has to require that one check regardless of how many lint jobs exist. hadolint's warning-level pinning advice is visible but non-blocking (--failure-threshold error): this repo deliberately floats several images/packages on latest rather than pinning. yamllint's config lives in .yamllint, relaxing the three default rules that conflict with this project's own conventions (long single-line comments, no leading ---, GitHub Actions' on: key).

There is no application test suite in this repo; the checks above are the project's actual verification surface.

Releases

Every push to main runs .github/workflows/release.yml, driven by semantic-release (.releaserc.json) against Conventional Commits (commitlint.config.js, enforced by the commitlint/commitlint-pr-title CI jobs above). Almost every conventional-commit type triggers at least a patch release (see .releaserc.json's commit-analyzer releaseRules: feat is minor, a breaking change (! or a BREAKING CHANGE: footer) is major, everything else conventional is patch), so in practice nearly every push to main produces a release.

A release:

1. Stamps the new version into galaxy.yml's version: field (which stays 0.0.0 in source between releases) and into the published-image tag defaults in roles/github_runner_arc/defaults/main.yml (github_runner_arc_heartbeat_image, github_runner_arc_autoscaler_image, github_runner_arc_image_pull_secret_renewer_image, github_runner_arc_node_recovery_image) - scripts/release/stamp_version.py does this by matching each variable's own name, not a line number, so it stays correct even while those files are edited concurrently by unrelated work.
2. Builds the collection tarball (ansible-galaxy collection build --force) and attaches it to the GitHub Release.
3. Updates CHANGELOG.md and commits it, galaxy.yml, and the stamped defaults file back to main as chore(release): <version> [skip ci].
4. Builds and pushes all five container images (Dockerfile, heartbeat/Dockerfile, autoscaler/Dockerfile, pull-secret-renewer/Dockerfile, node-recovery/Dockerfile) multi-arch (linux/arm64,linux/amd64) to their ghcr.io/exadev/github-runner* repositories, each tagged with the exact version, the floating major version, and latest - all three tags on the same manifest, via docker/metadata-action's type=semver patterns. To pin a host to a specific release rather than floating on latest, set roles/github_runner_arc/defaults/main.yml's image variables (or a host's own override) to an exact version tag instead.
5. Optionally publishes the collection to Ansible Galaxy, only if a GALAXY_API_KEY secret is present - the [skip ci] release commit and the image build both happen unconditionally, in this same run, regardless of whether Galaxy publishing is configured.

The [skip ci] in the release commit message is deliberate: the image build already happens in this same run using the exact version semantic-release just decided, so there is nothing useful for a second, separately-triggered run of this workflow (or of ci.yml) to do against a commit that only changed a version stamp and a changelog - ci.yml's own lint jobs already ran against every commit before it reached main.

No release has ever run yet, so no tag exists on this repo. The next push to main that reaches the release job will make semantic-release fall back to its own default first-release version, 1.0.0, since there is nothing to bump from. If a smaller starting point is wanted (matching galaxy.yml's own 0.0.0 convention, for example), tag the current tip v0.0.0 before that push lands - see this collection's own migration plan for when that seed tag is created; this workflow doesn't create it itself.

What the release push needs: a deploy key with write access on ExaDev/github-runner, whose private key is the RELEASE_DEPLOY_KEY repository secret release.yml checks out with. If main gains a ruleset, add deploy keys as its bypass actor so the release commit can land. Publishing to Ansible Galaxy additionally needs a GALAXY_API_KEY repository secret (an API key from https://galaxy.ansible.com/me/preferences); without it, the publish step logs that it's skipping and the release still completes.

Architecture

An earlier version of this repo ran a fixed number of long-lived containers via Docker Compose, cycling the GitHub runner registration between jobs (EPHEMERAL=true) but never resetting the container's own filesystem. Two problems followed: build artifacts, node_modules, and Playwright browser caches accumulated indefinitely inside the same container and filled the host's Docker disk allocation, and a fixed replica count meant jobs queued whenever more were in flight than there were replicas, with no way to burst beyond it.

ARC's gha-runner-scale-set solves both by design: it runs one fresh pod per job and deletes it afterwards, and it autoscales on queue depth. ARC needs a real Kubernetes cluster to run in. k3s ships an official Docker image that runs the whole control plane and kubelet inside one privileged container, a "cluster in a box". This makes the cluster itself just two Compose services (k3s on every control-plane host - the same service definition whether it bootstraps or joins, see the Ansible section's node-role split - and k3s-agent on a pure-worker host), portable to any machine with Docker (plus helm and kubectl on the host), with no Colima-specific flags, no separately installed kind or k3d CLI. ARC itself (the controller, the runner scale sets, and the ephemeral runner pods), plus the fleet-health platform (heartbeat/autoscaler - see Heartbeat/Autoscaler below), are not more Compose services; they're all Kubernetes-native resources installed by the Ansible role (Helm and kubernetes.core.k8s) into this cluster (see roles/github_runner_arc/).

The control plane itself runs genuine embedded-etcd HA across three server-role hosts, not a single node with worker capacity bolted on: one host bootstraps with --cluster-init (the one supported migration path from k3s's default single-node SQLite datastore to embedded etcd), and two further hosts join as full k3s server members - a real vote each in etcd's own raft quorum, not just schedulable capacity. 3 is the practical minimum: etcd tolerates the loss of any minority of its members without losing quorum, and with 2 servers that minority is zero - a second server can only ever make quorum strictly worse than staying at one, never better, so "2-node HA" isn't a real intermediate step, it's a trap. Getting there over Tailscale rather than a shared LAN needed real, non-obvious work - see the Non-obvious behaviour section's etcd-specific entries for what actually broke and why.

Two or more physical hosts on different home networks can't reach each other's LAN IPs directly, so cross-node cluster traffic (a second node joining, cross-node pod-to-pod traffic, the API server's proxy to a remote kubelet, and now etcd's own peer traffic between server hosts) needs a real overlay between them - see Tailscale below for how that overlay is built and the DNS pitfall it introduces.

The custom runner toolchain (GCC 11/G++ for C++20, Bun, the gh CLI — see Dockerfile) carries over from the previous design, rebased onto ARC's own actions/actions-runner base image instead of catthehacker/ubuntu:act-latest. ARC's runner pods use their own registration flow, so the previous design's vendored myoung34/docker-github-actions-runner scripts (registration, GitHub App JWT signing, ephemeral re-registration) are not needed.

The images are built on GitHub-hosted runners rather than on the fleet, so a broken image can never take down the pool it would need to be rebuilt on. See Releases below for how a release builds them.

.github/workflows/ci.yml, mesh-integration.yml and litestream.yml run on GitHub-hosted ubuntu-latest, which is free for a public repository, so checking the fleet's configuration never depends on the fleet itself being up. test-arc-runner.yml and test-autoscaler.yml are the ones that target the fleet, since the fleet's own behaviour is what they test.

Tailscale

Each node's k3s process joins a Tailscale tailnet as its own device, using k3s's own built-in (experimental) --vpn-auth integration: --vpn-auth="name=tailscale,joinKey=<key>" on both k3s server and k3s agent. This runs Tailscale inside the k3s container/process itself, separate from the host machine's own Tailscale identity if it has one, and flannel uses Tailscale's own subnet-route advertisement for pod-to-pod traffic instead of its usual VXLAN encapsulation - no manual --node-ip, no Docker port-publishing, no host-level Tailscale dependency at all. k3s/Dockerfile layers the tailscale/tailscaled binaries from the official Tailscale image onto rancher/k3s's own minimal base (which has no package manager to install them at runtime), and k3s/node-entrypoint.sh (the Compose services' entrypoint, bind-mounted from the node's Compose project directory; both live in roles/github_runner_cluster/files/) starts tailscaled itself before execing k3s, since k3s's vpn-auth integration only runs tailscale up and assumes the daemon is already running.

The mesh's control server does not have to be Tailscale's. The cluster role's github_runner_cluster_mesh switches it to Headscale: a server someone else runs, one the role runs under Compose outside k3s, or one it runs inside the cluster itself; see roles/github_runner_cluster/README.md. The rest of this section describes the default, Tailscale.

Provisioning the join key: create a reusable, pre-authorized Tailscale auth key (Tailscale admin console → Settings → Keys, or the POST /api/v2/tailnet/-/keys API), tag it so an ACL grant can target it, and set K3S_VPN_AUTH_JOIN_KEY in every host's .env (or, via Ansible, set github_runner_cluster_tailscale_join_key, which the 1Password adapter reads from a tailscale-join-key field, as roles/github_runner_secrets_onepassword/README.md describes). The tailnet's ACL needs a grants entry permitting traffic between the nodes' own tag and the pod CIDR itself (e.g. 10.42.0.0/16), not just the tag - Tailscale evaluates a subnet-routed packet's actual embedded source IP, not just the sending device's identity, so a grant scoped to the tag alone silently blocks cross-node pod traffic (k3s-io/k3s#8372).

Tailscale's own DNS management is a real trap here. By default (--accept-dns=true, Tailscale's own default), tailscale up overwrites the container's /etc/resolv.conf with MagicDNS's resolver (a fixed, well-known 100.100.100.100) plus whatever search domains the tailnet's own DNS settings configure - which can include search domains meant for entirely unrelated, personal use of the same tailnet. Kubernetes' dnsPolicy: ClusterFirst then copies the node's search domains into every pod's own /etc/resolv.conf alongside the cluster's own three, and the default ndots:5 means a pod's resolver tries every search-suffixed name before the bare hostname for anything with fewer than 5 dots - which is essentially every real-world hostname. If any of those inherited search domains carries a wildcard DNS record, a pod's lookup of a real external hostname can resolve to that wildcard's target instead, while the application still sends the original hostname as its TLS SNI - silent certificate-verification failures for arbitrary outbound HTTPS traffic, indistinguishable at a glance from a hung network. Both k3s server/k3s agent invocations set --resolv-conf to a synthetic file containing only nameserver 100.100.100.100 and no search domains, so pods still resolve everything correctly through Tailscale's own DNS proxy without inheriting a search list they have no use for.

Heartbeat

ARC's autoscaling means there is normally no pre-existing "online runner" to check the way the previous design's fallback logic did (querying /orgs/{org}/actions/runners for an online match); with minRunners: 0, pods exist only while a job is running. To still support falling back to ubuntu-latest if the whole fleet goes down, heartbeat (an in-cluster Deployment - see roles/github_runner_arc/tasks/install_platform.yml) checks every HEARTBEAT_INTERVAL_SECONDS (default 180s) that a k3s node and the ARC controller are healthy, and if so, refreshes a single secret gist with a unix timestamp set HEARTBEAT_WINDOW_SECONDS (default 600s) into the future. exadev/runner-fallback-action reads that gist over the unauthenticated GitHub API and routes to exadev-runners while the timestamp is still fresh, or to ubuntu-latest otherwise. See that repo's docs/spec.md for the algorithm.

Runs as a genuinely unpinned Kubernetes Deployment (replicas: 1, no nodeSelector), the same pattern ARC's own controller pods already use, not a host-pinned Docker Compose service — Kubernetes' own scheduler places and reschedules it on any server host with no config to move if one goes down. kubectl needs no KUBECONFIG: running as a pod with a mounted ServiceAccount token, in-cluster config is auto-detected. RBAC is scoped to exactly what it reads: cluster-wide nodes get/list, deployments get/list in the actions-runner-controller namespace, and get on the autoscaler's own status ConfigMap by name (see Autoscaler below) — no wildcards.

Create the secret gist once:

echo "$(($(date +%s) + 600))" | gh gist create --filename arc-healthy-until --desc "ARC fleet health heartbeat" -
# put the hex id from the returned gist URL into the installing host's github_runner_arc_heartbeat_gist_id
# (or .env.bootstrap's HEARTBEAT_GIST_ID for bootstrap.sh),
# and into whatever reads it, such as runner-fallback-action's gist-id input

Confirm it refreshes: gh gist view "$HEARTBEAT_GIST_ID" must show a timestamp near now + 600, advancing roughly every 3 minutes.

Autoscaler

A scale set's static maxRunners has to be safe for the pool's real worst case, every pod at its limit at once, so it leaves real idle capacity unused whenever the jobs in flight need less than that. autoscaler (an in-cluster Deployment, same shape/placement as heartbeat above) raises and lowers maxRunners on the one profile's AutoscalingRunnerSet with autoscale: true, based on real, current memory usage and pressure, never on job identity or GitHub Actions queue depth: every consuming repo's CI keeps using the scale set's one runs-on label unchanged, and no job is ever classified as light or heavy. RBAC is scoped to exactly what it reads/writes: cluster-wide nodes.metrics.k8s.io get/list, get/patch on that AutoscalingRunnerSet by name and pods.metrics.k8s.io get/list in its own namespace, and get/patch on its own status ConfigMap by name, with no wildcards.

Each poll (AUTOSCALER_POLL_SECONDS, default 45s):

• Reads the currently-running count (status.currentRunners) and the per-pod hard memory limit (spec.template.spec.containers[0].resources.limits.memory) straight from the live AutoscalingRunnerSet, never a static config value, so both always match what is actually deployed.
• Sums real, current memory usage of the running runner pods (kubectl top pod in the scale set's namespace) — this, not a static per-pod request or limit, is the entire point of "usage-driven".
• Reads cluster-wide memory-availability pressure via kubectl top nodes, taking the worst-case (minimum) available-memory percentage across all nodes, not an average — a pool average can look healthy while the specific node a new runner pod would actually land on is not, consistent with the algorithm's own bias toward lowering eagerly (lowering never disrupts in-flight jobs). This is a genuine improvement over the pod's own earlier host-pinned design, which could only ever see one fixed machine's pressure via /proc/meminfo, not the whole pool's. No swap-pressure signal: metrics-server's API has no swap field at all, and the only alternative (the kubelet's own Summary API) needs a materially broader RBAC grant for a signal most kubelet versions don't even surface unless an off-by-default feature gate is on — dropping swap detection is a deliberate, documented trade-off once the pod is genuinely unpinned to any node, not an oversight; memory-availability pressure remains the dominant signal (the incident the values file's own comments were tuned from).
• Raises maxRunners by exactly +1 per cycle (never straight to a computed target) once headroom has covered a full pod's hard limit for AUTOSCALER_RAISE_CONFIRM_POLLS (default 2) consecutive polls, capped at AUTOSCALER_MAX_CEILING (github_runner_arc_autoscaler_max_ceiling, or the profile's sizing): the bin-packing-safe ceiling across the pool at the profile's pod size, not an arbitrary cap.
• Lowers immediately, with no delay or averaging, the moment headroom drops below a pod's hard limit or real pressure is detected — lowering never disrupts in-flight jobs, since ARC only gates new claims. Never patches below the currently-running count.
• Fails safe on any measurement error (kubectl unreachable, kubectl top nodes/kubectl top pod failing): patches down to AUTOSCALER_FLOOR (github_runner_arc_autoscaler_floor, which must equal the profile's own static maxRunners) immediately rather than skipping the cycle silently.

Ships with AUTOSCALER_DRY_RUN=true by default: it computes and logs the target and writes its own status, but never patches. Generate real concurrent traffic to watch it react: gh workflow run test-autoscaler.yml, then watch kubectl logs -n github-runner-platform deploy/autoscaler -f and kubectl get configmap autoscaler-status -n github-runner-platform -o jsonpath='{.data.status\.json}'. Only set AUTOSCALER_DRY_RUN=false after watching real dry-run output across genuine CI traffic. Status and its raise-confirm counter live in a shared Kubernetes ConfigMap (autoscaler-status, pre-created empty so neither pod's own ServiceAccount ever needs create RBAC on it, only get/patch) rather than a bind-mounted directory, since the two pods can land on different nodes now — scripts/heartbeat.sh reads that same ConfigMap back out and republishes it as a second file in the heartbeat gist, reusing its existing gist-write credential rather than giving the autoscaler its own.

Every Helm upgrade above reverts maxRunners to the values file's safe floor. The role's install_org.yml triggers one immediate autoscaler poll after every such upgrade (kubectl exec deploy/autoscaler -- ..., reaching the pod through the API server regardless of which node it's on), so the safe-floor window after a redeploy is seconds, not a full poll interval.

Node recovery

When a node stops (its host is switched off or asleep, Docker quits, or the k3s container is stopped), the node controller marks it Unknown once its kubelet has missed heartbeats for the node-monitor grace period, and after the pods' five-minute unreachable toleration marks them for deletion. Only the node's kubelet can confirm a deletion, so they then stay Terminating indefinitely. A Deployment's pods are replaced regardless, which is why the ARC controller comes back on its own, but ARC creates each scale set's listener as a single pod and waits for the old one to go, so a listener on the stopped node is never replaced and every job for that scale set queues until someone force-deletes it.

node-recovery (an in-cluster Deployment in the platform namespace, see roles/github_runner_arc/templates/node-recovery.yaml.j2 and scripts/node-recovery.sh) applies Kubernetes' non-graceful node shutdown: every NODE_RECOVERY_POLL_SECONDS (default 15s) it gives each node whose Ready condition has been Unknown for github_runner_arc_node_recovery_after_seconds (default 120s) the node.kubernetes.io/out-of-service=nodeshutdown:NoExecute taint. The control plane then evicts every pod on that node at once, unless it tolerates that taint, and the pod garbage collector force-deletes the ones already terminating there, so ARC starts the listener again on a healthy node. When the node reports Ready again the watcher removes the taint, and the node's kubelet stops the containers of the pods that were deleted while it was away. With the defaults a stopped node stops holding up job pickup within roughly four minutes: the grace period (50s on current Kubernetes), the threshold, one poll, the garbage collector's 20-second sweep, and the listener's own restart. github_runner_arc_node_recovery_enabled: false removes the watcher and any taint it applied.

The watcher only acts on a node whose kubelet has gone silent (Unknown), never on one reporting NotReady (False), whose kubelet is alive and finishes its pods' deletion itself. It never taints the node it runs on, removes only taints it applied (recognised by its github-runner.exadev/out-of-service-applied-at annotation), and sends each change as a JSON patch that first tests the node's resourceVersion, so a node that came back since it was read is left for the next poll. RBAC is get, list and patch on nodes, with no delete: the watcher never removes a Node object, and the taint does not touch k3s's etcd membership, which only deleting a server's Node object changes. Its own pod tolerates an unreachable or not-ready node for only 10 seconds, so if it was running on the node that stopped, its ReplicaSet starts a replacement elsewhere well before the threshold passes.

The threshold is also where a stopped node's runner pods are given up, earlier than the five-minute default eviction would. Their containers stopped with the node, so their jobs are lost either way unless the node returns within that time; lower the threshold for faster recovery, or raise it to wait longer for a node that is only asleep.

Its image, github_runner_arc_node_recovery_image, is pulled without a pull Secret, so the published package must be public.

Conventions

Adding a second org is additive, never a change to anything existing:

1. Add a values file for the new org to the fleet's github_runner_arc_values_dir (an existing org's is a starting point), or leave values_file out of the profile and use the ARC role's own values template.
2. Add another entry to the installing host's github_runner_arc_orgs in its host_vars/<host>.yml, with a private_key holding that org's App private key from any source Ansible reads, or a private_key_op_reference pointing at it in 1Password.

The controller install stays shared: it watches all namespaces by default, so one install serves every org's runner scale set release. No changes to the Dockerfile, the controller install, or any other org's release are needed to add an org.

The role must stay idempotent — every step uses kubernetes.core.k8s (server-side apply) or helm upgrade --install semantics, never a plain create, so reruns never fail on an already-existing resource. bootstrap.sh must stay a thin translation from .env.bootstrap to role variables: any deployment step added there instead of in the role reintroduces a second implementation that drifts.

Dockerfile must not set its own ENTRYPOINT/CMD, and must not touch the base image's runner user or working directory. The ARC Helm chart's pod template supplies command: ["/home/runner/run.sh"] itself; it only needs the toolchain layered on top.

The k3s node's docker-compose.yml and k3s/ build context live only in roles/github_runner_cluster/files/. Keep github_runner_cluster_compose_project stable on a running host: the node's named volumes are prefixed with it, so a new name starts a new, empty node.

Commits follow Conventional Commits (fix:, feat:, tune:, and so on).

Non-obvious behaviour

• A scale-set's listener pod can crash-loop forever with ephemeralrunnersets... not found after a Helm upgrade, even though the release itself succeeds. ARC names the EphemeralRunnerSet it creates for a release with a spec hash suffix, and gives it a new name whenever the pod template changes (a helm upgrade that touches resource limits, nodeSelector, or similar). The AutoscalingListener is supposed to track this rename, but can be left pointing at the old, now-deleted name - it then restarts every second or two, each time failing at startup with could not patch ephemeral runner set <old-name>... not found, and the scale set's own job queue never gets serviced even though every other release's listener stays healthy. Diagnose with kubectl get autoscalinglisteners -n actions-runner-controller -o custom-columns='NAME:.metadata.name,ERS:.spec.ephemeralRunnerSetName' against kubectl get ephemeralrunnersets -A - a mismatch confirms it. Fix: kubectl delete autoscalinglistener <name> -n actions-runner-controller; the controller recreates it immediately with the current EphemeralRunnerSet name.
• k3s cannot overwrite an existing kubeconfig through Colima's macOS share. The container-root-to-Mac-user file mapping leaves /output/kubeconfig.yaml owner-read-only, so k3s crash-loops with "permission denied" on every restart (for example after a Mac reboot) unless the file is removed first. k3s/node-entrypoint.sh deletes it before starting k3s each time — do not remove that step.
• The k3s node name must stay stable across container recreations. Without a pinned hostname, k3s derives the node name from the container's hostname (its ID), so every recreation (for example on an image pull) registers a new node and leaves the old one stale in etcd. The cluster role's docker-compose.yml pins every server host's own hostname to ${K3S_NODE_NAME:-k3s-server}, Ansible-templated per host (defaulting to that host's own inventory name; github_runner_cluster_node_name overrides it, for example to keep a pre-existing device identity rather than forcing a disruptive rename).
• kubectl get nodes can succeed with zero nodes. It returns an empty, still-successful list before the node object has registered. The role polls for at least one node before calling kubectl wait, which fails outright ("no matching resources found") against zero nodes.
• Zero runner pods in an arc-runners-<org> namespace (or its per-profile -<suffix> variant) when idle is expected, not a bug — minRunners: 0 means pods exist only while a job is running.
• An org's Settings > Actions > Runners page shows the runner scale set itself, not individual long-lived runner names the way the previous design did. ARC's ephemeral runners do not persist between jobs, so there is nothing to list while idle.
• An autoscaled profile's static maxRunners and per-pod resource limits have to come from the pool's measured capacity, not arbitrary defaults: a higher maxRunners without matching headroom leads to out-of-memory kills and pod evictions (ExaDev's fleet repository records the incident its own values were tuned from). scripts/autoscaler.sh depends on this value staying the safe floor it reverts to.
• kubectl top pod/kubectl top nodes require metrics-server. k3s bundles it by default (the cluster role's docker-compose.yml only disables traefik), but confirm it is actually running during the first real autoscaler dry run — scripts/autoscaler.sh fails safe to the static floor if either is unreachable. A very short-lived job (seconds, not minutes) can complete and its pod be garbage-collected before metrics-server's own scrape interval ever captures its usage - confirmed live against test-arc-runner.yml's ~15-20s jobs, where kubectl top pod genuinely never returns data. This correctly triggers the fail-safe rather than a wrong reading, and doesn't matter in practice: a job that short never accumulates meaningful memory pressure anyway. test-autoscaler.yml's own probe jobs hold for 120s specifically so metrics-server has time to observe them.
• Recreating a k3s-agent or joining k3s server container can leave it permanently NotReady with "Node password rejected" in its logs. /etc/rancher/node/password lives inside the container's own writable filesystem, not a persisted volume, so a fresh container generates a brand-new random password - but the etcd-bootstrap host still holds the previous container's password for that node name, stored as a Kubernetes Secret (kube-system/<node>.node-password.k3s), and rejects the mismatched rejoin. Fix: on the bootstrap host, kubectl delete secret <node>.node-password.k3s -n kube-system (and, for an agent, kubectl delete node <node>), then restart the container so it registers fresh. Never delete a server's Node object: k3s then removes that server's etcd member, and a removed member's etcd refuses to start until its local datastore (/var/lib/rancher/k3s/server/db) is moved aside, after which it rejoins as a new member. roles/github_runner_cluster/tasks/main.yml self-heals the password mismatch automatically, only for hosts whose node is missing or not Ready, and deletes Node objects only for agents.
• etcd's own peer traffic does not follow vpn-auth's --node-ip/--advertise-address the way every other kind of cluster traffic does - it stays on the container's own internal, cross-machine-unroutable Docker-bridge IP by default. k3s logs this outright ("Etcd IP (PrivateIP) remains the local IP. Running etcd traffic over VPN is not recommended due to performance issues") and deliberately avoids the VPN interface for etcd specifically, assuming a real local network is the better alternative - an assumption that doesn't hold here, since a container's own bridge IP is never reachable from another machine at all. Confirmed live: a second server's join failed outright with MemberAdd request timed out, because it was trying to reach the first server's own unroutable bridge IP. An --etcd-arg="initial-advertise-peer-urls=..." passthrough looks like the fix but isn't: it only overrides etcd's own raw flag, while k3s independently builds its own --initial-cluster string from config.PrivateIP, computed separately and earlier - the two then disagree and etcd's bootstrap validation rejects the mismatch outright ("--initial-cluster has ...=<bridge-ip> but missing from --initial-advertise-peer-urls=<tailscale-ip>"). The actual fix, traced through k3s's own source (pkg/cli/server/server.go, pkg/executor/embed/embed.go): config.PrivateIP is populated from --node-ip's own value if set, and this happens before --vpn-auth's own executor-layer code later rebuilds NodeIP from its own Tailscale detection - so passing --node-ip explicitly ourselves, to the same real Tailscale IP vpn-auth would itself have detected, is what actually makes PrivateIP (and therefore etcd's peer-advertise address) correct, with no second, disagreeing source of truth for the same value. docker-compose.yml's k3s service entrypoint brings Tailscale up itself and reads tailscale ip -4 before exec'ing k3s specifically to learn this IP first, since exec replaces the script's own process and nothing can react afterward to what --vpn-auth's own internal call would have produced.
• A global --etcd-arg override is not scoped to the "real" server's own etcd instance - it also applies to k3s's separate, internal "temporary etcd", spun up on every restart (never on first boot) to reconcile with an already-initialized on-disk datastore before the real instance starts. An earlier --etcd-arg="listen-peer-urls=https://0.0.0.0:2380" (kept as a "harmless" belt-and-braces bind override after fixing the peer-URL issue above) forced that temporary instance onto the same TLS 0.0.0.0:2380 bind it has no certificates for by design (it normally listens on a local, ephemeral, non-TLS port). Confirmed live: this only broke on a restart against an already-initialized datastore, never on a fresh --cluster-init, which is exactly why it wasn't caught until well after the first working deploy - fatal: "cannot listen on TLS for [::]:2380: KeyFile and CertFile are not presented". --node-ip alone (above) is sufficient; etcd's own default listen-peer-urls already derives from the same PrivateIP correctly, so there's nothing left for an explicit override to add.
• Calling tailscale up --authkey=... unconditionally on every container restart re-registers a brand new tailnet device each time, instead of reconnecting the persisted one - even with the join key's own state volume intact. A reusable Tailscale auth key is designed to authorize provisioning new devices, and a naive tailscale up call on every restart (needed here to learn this node's own Tailscale IP before execing k3s - see above) doesn't check whether the daemon is already authenticated first, so it keeps minting new devices under names like node1, node1-1, node1-2... k3s's own --vpn-auth integration avoids exactly this by checking the Tailscale backend state first and skipping its own tailscale up call entirely once already "Running" (confirmed against pkg/vpn/vpn.go's StartVPN()) - the manual pre-empt call needs the identical guard: check tailscale ip -4 succeeds before calling tailscale up at all, only calling it when not already authenticated.
• Tailscale assigns an IP and wires up its own MagicDNS resolver as two separate steps, not atomically - a node having a real IP is not proof DNS resolution against another tailnet member is working yet. A joining server needs to resolve the bootstrap host's own MagicDNS hostname (K3S_JOIN_SERVER_URL) before it can bootstrap against it at all. Confirmed live: k3s failed outright with "dial tcp: lookup k3s-server.example.ts.net: no such host" immediately after tailscale ip -4 had already returned a real address for the local node. Fix: wait for a real nslookup success against the join server's own hostname specifically, not just local Tailscale readiness, before invoking k3s (the k3s service's entrypoint only does this when K3S_JOIN_SERVER_URL is actually set - the bootstrap host has nothing else to resolve).
• A single remaining healthy etcd server can get stuck indefinitely if another member is registered but permanently unreachable (e.g. a partial join that never completed), even though that member is a non-voting learner and shouldn't block quorum. Symptom: "failed to publish local member to cluster through raft" with context deadline exceeded, repeating forever, alongside kubectl returning 503 the server is currently unable to handle the request. The fix is k3s's own documented recovery mechanism, run once against the affected node: stop the container, then k3s server --cluster-reset (with the SAME flags the real service uses, including --node-ip set to the node's own persisted Tailscale IP - getting this wrong re-registers the single remaining member under the wrong address, e.g. the invoking container's own ad-hoc bridge IP, and the real service then fails to start with "this server is a not a member of the etcd cluster") - this resets etcd cluster membership back to a single member using the existing local data, it does not wipe anything. It's a one-shot operation: k3s writes a guard file after a successful reset and refuses to run --cluster-reset again until it's manually deleted (rm /var/lib/rancher/k3s/server/db/reset-flag) if a second reset is genuinely needed. After a reset, restart the service normally (without the flag) and clean up the stale member's own now-orphaned Kubernetes Node object and node-password Secret before it rejoins fresh.

References

• ExaDev/runner-fallback-action — reads the heartbeat gist; its docs/spec.md documents the routing algorithm.
• actions/actions-runner-controller — upstream ARC project.
• ExaDev/github-runner-fleet (private): ExaDev's own fleet, whose inventory, host_vars and Helm values are a full working configuration, including the resource-tuning rationale referenced above. The controller's values come from the ARC role's own templates/controller-values.yaml.j2.
• actions/actions-runner-controller gha-runner-scale-set-controller docs — the AutoscalingRunnerSet CRD (actions.github.com/v1alpha1) scripts/autoscaler.sh reads and patches.