[flyte2] Parent action hangs forever when the WatchForUpdates subscriber channel drops a child's terminal update #8104

Summary

ActionsClient.notifySubscribers drops an update whenever a parent's subscriber channel is full. If the dropped update is a child's terminal phase, the parent's SDK informer never learns that the child finished, and the parent waits forever. Nothing re-sends the update and the SDK never re-lists, so the run can't recover without an abort. The chart pins the channel size at 100, below the Go default of 1000 set in #7657, so moderate fan-out is enough to hit this.

Versions

• Chart flyte-binary v2.0.50, image ghcr.io/flyteorg/flyte-binary-v2:v2.0.50, chart-default actions.watchBufferSize: 100
• SDK flyte 2.10.2, with _U_USE_ACTIONS=1
• EKS, single backend replica

What we saw

The run had a root task that fanned out 42 parent tasks, 11 at a time. Each parent fanned out 350–560 children, 300 at a time, so about 3,300 task pods were running at peak.

• 30 parents succeeded. The other 12 are still RUNNING 12+ hours later. Every child of every parent is SUCCEEDED according to the API (Action.listall), and each stuck parent submitted its full set of children.
• The parents that hung were exactly those still running once the burst wound down. All 30 that succeeded had finished before the first one hung.
• Each stuck parent pod logs only Informer cache sync timed out at startup. That timeout isn't the cause: three parents that logged it still succeeded.
• Only the 12 stuck parents' TaskAction CRs remain. The children's CRs have been deleted.
• There's no server-side trace. The drop is logged at WARN, which the chart-default logging.level: 1 hides, and there's no metric yet (Add OTEL counter for dropped subscriber updates #7641).

Mechanism

1. Subscribe creates a channel of depth watchBufferSize for each WatchForUpdates stream (client.go#L456-L468).
2. notifySubscribers does a non-blocking send and drops the update when the channel is full (client.go#L721-L733). Each child produces several updates (queued, running, terminal, deleted), and 300 children per parent completing in a burst easily pushes more than 100 into one channel faster than the stream drains them.
3. The stream keeps going after a drop (actions_service.go#L139-L173), so the client gets no error and doesn't reconnect.
4. The SDK waits on each child's completion event with no timeout for task actions (_core.py#L347). The informer only re-lists when the stream errors (_informer.py#L221), and watch_conn_timeout_sec is stored but never used.
5. A reconnect wouldn't help anyway. The snapshot comes from ListChildActions, which lists live TaskAction CRs (client.go#L397), and terminal children's CRs are already gone.

On main, the Go default is 1000, but the chart still sets watchBufferSize: 100 for both actions and runs, and the drop path is unchanged.

Suggested fixes

1. Make the chart match fix(actions): bump TaskAction watch workers and buffer size #7657: flyte-core-components.actions.watchBufferSize: 1000, or remove the override and let the Go default apply. We've set 1000 in our values as a workaround.
2. Make a drop recoverable instead of silent. One option is to end that subscriber's stream with an error on overflow, so the SDK reconnects and re-syncs. That only works if the reconnect snapshot includes children whose CRs have been cleaned up, for example by reading terminal state from the runs store rather than live CRs. Another option is to block that subscriber's send with a bounded timeout before giving up.
3. On the SDK side, consider a periodic re-sync or a completion timeout as a backstop for pending children, so a lost update degrades into a delay rather than a permanent hang.

The counter in #7643 would make this visible but wouldn't stop the hang.

Repro sketch

Fan out several parents concurrently, each running flyte.map (or an equivalent asyncio.gather) over ~500 short child tasks with concurrency ≥ 200, on a single backend replica with the chart defaults. Some parents stay RUNNING after all their children succeed.

AI disclosure: written with Claude Code as a code assistant, reviewed by me.