Cilium ≥1.19 socket-termination regression (cilium/cilium#48564) leaves nodes permanently deaf to discovery — harden the discv5 reader so a one-shot read error can't stop it #3053

Summary

Root cause: a Cilium regression. On Kubernetes clusters running Cilium ≥ 1.19 as the kube-proxy replacement (socket-LB with socket termination), replacing a peer's pod makes Cilium destroy the discovery UDP sockets of every node that was talking to it. Destroying unconnected UDP sockets is a regression introduced in 1.19, which v1.16–v1.18 didn't do: cilium/cilium#48564. It is fixed on main by cilium/cilium#48725. That fix's backport is labelled for 1.20 only, and it isn't in a release yet.

Why that is catastrophic for SSV, and the part this issue asks to change. The destroyed socket keeps working: it stays open and bound to its explicit port. Only its next read returns ECONNABORTED, once. But go-ethereum's discv5 reader (UDPv5.readLoop) treats that as fatal and exits, silently:

• Nothing supervises it. The dispatch and table loops keep running and keep sending PINGs and FINDNODEs, but no discovery packet is ever read again until the process restarts.
• The only trace is a Debug-level "UDP read error" line, emitted only when DISCOVERY_TRACE is on.

After one such event on every long-running node (bootnode included), discovery is dead cluster-wide. Existing connections survive, but a node that restarts can't find anyone, and nobody can find it. Peers only redial a peer that discovery re-finds after it disconnected, and nothing is re-found any more.

Priority

1. Primary fix, Cilium (infra/operators). Run a Cilium release containing bpf: only terminate sockets still connected to the dying backend cilium/cilium#48725, or get a 1.19 backport, or turn the socket-termination option off (see Mitigations). This removes the trigger.

2. Hardening, SSV (this issue). Keep the discovery reader alive across one-shot read errors, and make discovery health observable. This protects operators whose infrastructure we don't control. It also covers rarer non-Cilium triggers of the same one-shot error:

   • SOCK_DESTROY via ss -K or Android's netd;
   • ECONNREFUSED via IP_RECVERR;
   • any other errno that IsTemporaryError doesn't treat as temporary.

   It does not replace 1.

3. Optional, go-ethereum. A robustness request for its read loop (see Upstream).

Both v2.4.3 and stage are affected: go-ethereum's reader is unchanged from v1.16.4 through v1.17.6 and master. #2980 fixes a different failure (dispatch blocked on a full Unhandled channel), and doesn't cover this one.

Impact (observed)

Seen on a 4-operator Sepolia cluster running v2.4.3 (100 validators):

• Operator restart kills discovery for everyone else. Restarting one operator killed the discovery reader on the three other operators and on the bootnode, within the same second.
  • The cluster kept attesting on its existing TCP connections, but with no fault tolerance left.
  • An operator restarted afterwards stayed at 0 peers, and is still isolated at the time of writing.
• Full restart = total outage. Earlier, all four operators restarted while every long-running discovery instance (bootnode included) was already deaf in this way. None of them found a single peer, and the cluster performed no duties for ~3h45m until the bootnode was restarted.
• It can go unnoticed for months. The cluster's exporter had been deaf for over two months without anyone noticing. Nothing in logs or metrics points at it, and the discovery iteration counter looks "busy" or "quiet" for unrelated reasons (a full table is replayed at ~10/s; an empty table waits a minute per iteration).

Evidence

1. The reader goroutine is gone; everything else in discovery is alive. Goroutine profiles (Pyroscope) of an affected process, compared with a healthy one, show:

The reader vanished from the profiles of three processes in the same minute. That's the minute a peer pod was deleted.

2. Cilium destroyed exactly those sockets, one second after the peer went away. These were logged by the agents on the hosts of exactly the processes whose readers died (4 sockets: 3 operators and the bootnode):

level=info msg="Forcefully terminated sockets" module=agent.controlplane.loadbalancer-reconciler.socket-termination
  filter="{DestIp:<deleted peer pod IP> DestPort:<peer discovery port> Family:2 Protocol:17 States:65535 ...}" success=1

The same happened at the next pod replacement, but only on hosts whose processes were still sending to the replaced peer. That's why a node whose table was already empty survived.

3. From outside, affected processes don't answer discv5. Neither protocol ID (ssvdv5 or discv5) gets a reply. Unaffected processes behind the same NodePort path do answer.

4. Reproduced with go-ethereum alone. This is a standalone program using only go-ethereum's public API: two ListenV5 listeners on localhost, one of them given one real kernel error. The error is triggered by setting IP_RECVERR and sending a datagram to a closed port, which makes the next read return ECONNREFUSED.

Results are identical on go-ethereum v1.16.4 and v1.17.5:

before: B->A PONG | A->B PONG | discv5 read loops running: 2 (A and B)
trigger (recverr) on A's socket
after #1: B->A NO ANSWER (RPC timeout) | A->B NO ANSWER (RPC timeout) | discv5 read loops running: 1
after #3: B->A NO ANSWER (RPC timeout) | A->B NO ANSWER (RPC timeout) | discv5 read loops running: 1
A's debug log: level=DEBUG msg="UDP read error" err="read udp4 127.0.0.1:40401: recvfrom: connection refused"
A's socket is still open and receiving: direct read got a packet from 127.0.0.1:40402 (err=<nil>)

Wrapping A's socket with a reader that retries non-close errors makes the same trigger harmless:

   [fix] read error "read udp4 127.0.0.1:40401: recvfrom: connection refused" -> retrying
after #1: B->A PONG | A->B PONG | discv5 read loops running: 2

An injected one-shot ECONNABORTED (what udp_abort() leaves on an explicitly bound socket) behaves exactly the same, with and without the wrapper. The reproducer also includes a real SOCK_DESTROY (netlink NETLINK_SOCK_DIAG) trigger. It needs CAP_NET_ADMIN and CONFIG_INET_DIAG_DESTROY, which our test kernel lacks, so that variant hasn't been run yet.

Affected code

go-ethereum (p2p/discover/v5_udp.go; same in v1.16.4, v1.17.6 and master, and in the discv4 readLoop):

nbytes, from, err := t.conn.ReadFromUDPAddrPort(buf)
if netutil.IsTemporaryError(err) {
	t.log.Debug("Temporary UDP read error", "err", err)
	continue
} else if err != nil {
	// Shut down the loop for permanent errors.
	if !errors.Is(err, io.EOF) {
		t.log.Debug("UDP read error", "err", err)
	}
	return
}

On Linux, IsTemporaryError only accepts EINTR, EMFILE, ENFILE and timeouts; beyond those it has only a Windows-specific "packet too big" case. So ECONNABORTED, ECONNREFUSED, EHOSTUNREACH and the like all end the loop.

SSV (network/discovery/dv5_service.go, initDiscV5Listener): the post-fork listener gets the raw socket, and so inherits the behaviour.

udpConn, err := newUDPListener(bindIP, opts.Port, n)
...
sharedConn := NewSharedUDPConn(dvs.ctx, dvs.logger, udpConn, unhandled)
...
dv5PostForkListener, err := listenV5(udpConn, localNode, *dv5PostForkCfg)   // <- reads the raw socket
...
dv5PreForkListener, err := listenV5(sharedConn, localNode, *dv5PreForkCfg)  // reads Unhandled; unaffected

Suggested implementation (hardening in SSV)

1. Keep the reader alive: a resilient socket wrapper for the post-fork listener

Add network/discovery/resilient_conn.go:

// resilientUDPConn keeps go-ethereum's discv5 read loop alive across read errors that leave the socket usable
// (e.g. ECONNABORTED after SOCK_DESTROY/udp_abort, ECONNREFUSED via IP_RECVERR, ENOBUFS). Only a closed socket
// ends the read loop.
type resilientUDPConn struct {
	*net.UDPConn // writes, LocalAddr and Close pass through unchanged
	ctx    context.Context
	logger *zap.Logger
	// backoff / rate-limit state, touched only by the single reader goroutine
}

func (c *resilientUDPConn) ReadFromUDPAddrPort(b []byte) (int, netip.AddrPort, error) {
	for {
		n, addr, err := c.UDPConn.ReadFromUDPAddrPort(b)
		if err == nil {
			// record liveness (see 2), reset backoff
			return n, addr, nil
		}
		if errors.Is(err, net.ErrClosed) || c.ctx.Err() != nil {
			return n, addr, err // real shutdown: let go-ethereum's read loop exit
		}
		// count by errno, WARN (rate-limited), back off (10ms doubling to 1s), retry
	}
}

Wire it in initDiscV5Listener: listenV5(newResilientUDPConn(dvs.ctx, dvs.logger, udpConn), localNode, *dv5PostForkCfg).

Leave these unchanged:

• dvs.conn stays the raw *net.UDPConn, so Close() still closes the real socket and unblocks the retry loop.
• SharedUDPConn keeps writing through the same raw socket and reading from Unhandled.

Details to get right:

• Why the socket can be reused after the error. udp_abort() → __udp_disconnect() only keeps the socket's local port when it was bound to an explicit port (SOCK_BINDPORT_LOCK); with port 0 the kernel would silently re-autobind (see cilium#48564). newUDPListener always binds opts.Port, so that's fine. Add a guard or comment so a future Port: 0 doesn't quietly reintroduce the problem.
• No hot loop on a persistent error. Use bounded exponential backoff reset by a successful read. If errors continue with no successful read for more than ~1 minute, log once at ERROR. Keep retrying rather than returning: returning is exactly the silent, permanent failure this fixes.
• Log independently of DISCOVERY_TRACE. Log the first error at WARN with the errno, then rate-limit. That way the next incident explains itself.

2. Make discovery health observable

The current ssv_p2p_discovery_iterations_total is not a health signal: a full table is replayed at ~10/s, and an empty table waits a minute per iteration. Add:

Suggested alerts:

• no packets received for 10 minutes;
• table empty for 15 minutes;
• any increase in read errors.

3. Optional: a watchdog as defense in depth

If no packets have been received for N minutes and the table is empty, re-bootstrap discovery: close both listeners and the socket, then re-run initDiscV5Listener on the same port. Mind the listener lifecycle, and SharedUDPConn.drain's ownership of Unhandled. This could land after 1 and 2, or stay a documented follow-up.

4. Tests

• Unit test (fails today, passes with 1).
  • Start two listenV5 instances on localhost, and wrap the post-fork conn of one of them so a single read returns &net.OpError{Op: "read", Err: os.NewSyscallError("recvfrom", syscall.ECONNABORTED)}.
  • Assert it still answers PINGs and its own PINGs still complete.
  • Repeat with ECONNREFUSED.
• Linux, unprivileged, real kernel error. Set IP_RECVERR on the socket and send one datagram to a closed port. Assert discovery keeps working and the error counter increments.
• Linux, privileged (skip on EPERM/EOPNOTSUPP). SOCK_DESTROY the socket via NETLINK_SOCK_DIAG. This is the exact Cilium, ss -K and netd path.
• Shutdown. After DiscV5Service.Close(), the wrapper returns net.ErrClosed promptly (no spin, no leaked goroutine), and both listeners stop.

5. Rollout

The root-cause fix depends on each operator's Cilium version, which we don't control. Every pod replacement in a Cilium 1.19 deployment without the fix will keep triggering this. So consider shipping the hardening in a patch release on the v2.4.x line as well as stage.

Mitigations

• Fix the trigger in Cilium (primary). Upgrade to a Cilium release containing bpf: only terminate sockets still connected to the dying backend cilium/cilium#48725, or ask for a 1.19 backport; the fix is currently labelled for 1.20 only. Until then:

  • disable bpf-lb-sock-terminate-pod-connections,
  • or limit socket-LB to the host namespace.

  Both are cluster-wide changes, so they're for the infra team to decide.

• Recover a deaf process by restarting it. The bootnode matters most, since it's the only rendezvous. In our setup, restarting it doesn't trigger terminations: traffic to it goes through a cloud load balancer hostname, which socket-LB doesn't rewrite.

• Use TRUSTED_PEERS between the known operators of a committee, so a restarted node can rejoin without discovery.

• Enable DISCOVERY_TRACE=true to see go-ethereum's "UDP read error" lines until the WARN log from Suggested implementation 1 exists.

Upstream

• Cilium (the root cause): bpf-lb-sock-terminate-pod-connections destroys unconnected UDP sockets since v1.19 cilium/cilium#48564 (unconnected UDP sockets destroyed since v1.19), fixed on main by bpf: only terminate sockets still connected to the dying backend cilium/cilium#48725.
• go-ethereum (optional): a robustness request. Its read loop ends permanently on errors that leave the socket usable, although its own comment says permanent errors end the loop by design, so it may be declined. The SSV change above doesn't depend on it. It'll be linked here if filed.