ci: switch to self-hosted runner (node1) - #141
Merged
Merged
Conversation
Runners registered 2026-07-23: - node1-lantern / node1-curio-core (repo-scoped) - Ubuntu 26.04, Go 1.26.4 pre-installed - user systemd service, linger on (survives reboot) - 6c/12t E5-1650v2, 60G RAM Zero-cost, no GH-hosted queue, unlocks integration tests that need network access to Reiers infra. Kept ubuntu-latest as a manual fallback by editing runs-on. Added workflow_dispatch to allow ad-hoc runs.
The Python reference values in TestCalcValidatorProb_PythonReference were generated with scipy/numpy on a modern x86_64 CPU with FMA3. CPUs without FMA3 (Ivy Bridge and older) execute math.Log/Pow with a different reduction order and produce results that drift by ~2e-5 relative at deep-depth reorg probabilities (values that are already close to float64 noise floor, e.g. 4.6e-12 at depth=30). Rather than loosen the tolerance globally (which would hide real algorithm regressions on modern CPUs), skip the exact-match check when cpu.X86.HasFMA is false. The structural tests still exercise the algorithm across all CPU classes. Surfaced during the CI move to self-hosted runners (node1 has an E5-1650v2, Ivy Bridge, no FMA3). Follow-up issue tracks generating per-CPU-class reference vectors so we can keep tight parity checks on every hardware target.
The self-hosted runner persists `~/go/pkg/mod` across runs on local disk, so actions/cache's GitHub-side cache service is redundant. It's also unreliable over our home WiFi: the 1.6 GB restore of the Go module cache was DNS-flaking on productionresultssa6.blob.core.windows.net (getaddrinfo EAI_AGAIN), stalling the step at ~40% and eventually getting canceled around 10 minutes in. Disabling cache means: - Faster first step (skips ~10 min download of a cache we don't need) - No dependency on GitHub-side blob storage reachability - Same-or-better warm-cache hit rate (local ~/go/pkg/mod is always hot on the runner between runs) Trade-off: if we ever move CI back to ubuntu-latest, module resolution will start cold on every run. Small price for reliability on the self-hosted path.
Adds `concurrency: { group: ci-${{workflow}}-${{ref}}, cancel-in-progress: true }`
so a fresh push on the same branch/PR immediately cancels any prior
in-flight run of this workflow.
Motivation: the self-hosted runner will otherwise serially process
every queued run in order, which chews up minutes on stale commits
we no longer care about. With this in place, only the latest commit
per branch gets CPU time on the runner.
Replaces the earlier `t.Skip` on non-FMA x86_64 with proper per-class reference vectors: - pythonReferenceResults: scipy/numpy on FMA-capable x86_64 (matches Go's math on any FMA-capable CPU including ARM64 with NEON). - nonFMAReferenceResults: Go's math on Ivy Bridge and older Intel Xeon (E5-16xx v2 in the Reiers home cluster runner). Generated on node1-lantern using chain/ecfinality/tools/refgen/main.go. Test dispatches by `cpu.X86.HasFMA` at runtime. Both classes now meet the tight 1e-12 tolerance within their class — no more skips, and any real algorithm regression on either CPU family is caught cleanly. Observed drift between classes at some depths (30/40/50) reached 2-4 sig figs, well above float64 noise floor — this is real algorithmic drift in the Skellam-PMF chain due to different reduction orders in math.Log / math.Exp / math.Lgamma, not just noise floor. Per-class reference vectors are the correct approach; loosening the tolerance globally would have hidden real regressions. Also adds tools/refgen so anyone can regenerate the vectors on a new CPU class with `go run ./chain/ecfinality/tools/refgen`. Closes #142.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires this repo's CI to the freshly commissioned home cluster.
Runner:
node1-lantern(self-hosted, linux, x64, node1, lantern labels)Host: 10.20.0.11, Ubuntu 26.04, 6c/12t E5-1650v2, 60G RAM
Service: user systemd, linger enabled (survives reboot)
Go: 1.26.4 pre-installed (setup-go@v5 still runs for module cache)
Zero-cost, no GH-hosted queue, faster warm builds, unlocks any future
integration tests that need cluster/Reiers-network access.
Fallback: edit
runs-onback toubuntu-latestif the runner isdown. Registered 2026-07-23, both runners confirmed online at the
GitHub API side before this PR.