Runners in the exadev-runners scale set are dropping mid-job. GitHub then shows the runner as offline but still busy, and the job either hangs past its timeout or is failed with no log uploaded.
Seen today on ExaDev/Exie, both times during the Run checks step (pnpm run check, which takes under a minute locally):
Exie's CI has not had a green run on main since 2026-08-26.
The fleet host (100.81.26.32) shows as offline in tailscale status, last seen about three hours before these runs, so I couldn't check the pods. Worth checking on the host: kubectl get events -A for OOMKilled or evicted runner pods, the node's memory headroom while a pnpm run check job runs, and whether the host is sleeping or losing its network.
Runners in the
exadev-runnersscale set are dropping mid-job. GitHub then shows the runner as offline but still busy, and the job either hangs past its timeout or is failed with no log uploaded.Seen today on ExaDev/Exie, both times during the
Run checksstep (pnpm run check, which takes under a minute locally):exadev-runners-hxcqx-runner-9kcxgwent offline and ignoredgh run cancel. Only the force-cancel endpoint stopped it.exadev-runners-hxcqx-runner-dts5vwent offline while busy, then was deregistered, and the job was marked failed withRun checksnever completed.Exie's CI has not had a green run on main since 2026-08-26.
The fleet host (100.81.26.32) shows as offline in
tailscale status, last seen about three hours before these runs, so I couldn't check the pods. Worth checking on the host:kubectl get events -Afor OOMKilled or evicted runner pods, the node's memory headroom while apnpm run checkjob runs, and whether the host is sleeping or losing its network.