fix(#150): --fallback-rpc daemon flag (Glif is not always reachable) - #151
Merged
Conversation
Live incident 2026-07-23: Glif public mainnet began 403'ing our Hetzner census's egress IP (-32156 "Access denied"). Sync loop kept ticking (head=6217239 in its own view), but the persisted store head froze at 6217164 forever: every chain-fetch got 403, and the gossip inline- backfill's 3-epoch cap could not cover the growing gap. Silent stall. Add a `--fallback-rpc <URL>` daemon flag that overrides the hardcoded Glif URL for the four RUNTIME call sites (combined-source race entry, Sync source, gossip-enabled Sync source, second combined.Source entry). The boot-anchor's Glif entry (fetchVerifiedTrustedHead) intentionally stays hardcoded: it participates in the #54 two-independent-operators agreement rule and silently swapping it to a different provider would weaken the anchor trust model. Documented in the flag help. Empty override (default) preserves current behavior on every call site. Whitespace-only override is treated as empty; a valid override is whitespace-trimmed. Unit tests cover both. Live-verified: Hetzner census restarted with --fallback-rpc https://api.chain.love/rpc/v1 clears the -32156 error and catches up to network head.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #150.
Live incident this evening: Glif public mainnet started 403'ing our Hetzner census's egress IP. Sync loop kept ticking in its own view (head=6217239) but the persisted head was frozen at 6217164 for ~90 epochs. The gossip inline-backfill's 3-epoch cap couldn't cover the gap, and the daemon had no CLI to point the chain-fetcher anywhere else.
Fix: add
--fallback-rpc <URL>tolantern daemon. Overrides the hardcoded Glif URL on the four runtime call sites (combined-source race entry, Sync source, gossip-enabled Sync source, second combined.Source entry). Deliberately does NOT override the boot-anchor Glif source — that one participates in the #54 two-operator anchor agreement and silently swapping it would weaken the trust model. Documented in the flag help.Empty default preserves existing behavior. Whitespace-only override is treated as empty; valid overrides are whitespace-trimmed. Unit tests cover both.
Verified working alternates from the affected Hetzner IP (via curl):
https://api.chain.love/rpc/v1,https://filfox.info/rpc/v1,https://rpc.ankr.com/filecoin.Deploying on Hetzner census immediately after merge to clear the incident.