fix(evm-rpc): retry transient provider and proxy errors instead of failing

[mo4islona]

Summary

Several transient failures from RPC providers and load-balancing proxies were classified as fatal. evm-dump exited on them and the data service restarted ingestion, although the same call succeeds when repeated. The error shapes, with handling before and after this change and the ones left fatal on purpose, are listed in #583.

Changes

• rpc-client: HTTP 520–524 (Cloudflare origin errors) are retried like 502/504.
• evm-rpc, EvmRpcClient.isConnectionError:
  • rate limits: code -32429, and the messages throughput limit and exceeded … capacity;
  • transient errors: upstream and aggregator failures, messages that say the failure is temporary, and "block not there yet" errors from a backend lagging the head. "Temporary", "please retry" and "at this time" count only when neither the JSON-RPC code (-32700, -32600, -32601, -32602) nor the message blames the request, so advice like "please retry with a smaller block range" stays fatal;
  • a proxy error is judged by the causes nested under data, not by its outer code and message, which the proxy normalizes. An unsupported-method, ignored-method, unauthorized, client-side or execution cause keeps the error fatal;
  • an HTTP 500 whose JSON-RPC body is such an error; for a batch body, only when every error in it is transient, whatever their order. A bare 500 is still governed by retryInternalServerErrors.
• evm-rpc, result validation: null where a value is due, or a non-hex string in result, throws RetryError instead of DataValidationError. Malformed objects still fail. See the trade-off below.

Metrics

Errors that used to stop evm-dump or restart ingestion are now retried, so an endpoint that keeps failing no longer shows up as restarts. To keep that visible:

• sqd_chain_rpc_retried_errors_total{url, kind}, in evm-dump and the EVM data service.
  • kind is one of rate_limit, transient, no_result, internal, timeout, http, connection. Outside EVM it can also be retry or other.
  • url is redacted like in the other RPC metrics.
  • Only errors the client actually retries are counted. The error that fails a request after its attempts are spent is not, and neither is a failed batch that reduceBatchOnRetry splits in halves instead of retrying.
  • In the data service the RPC clients run in worker threads. Their counts are read on each scrape, with a 1 s timeout per worker. A worker that does not answer keeps its last counts, and the final counts of closed backfill workers are kept, so the counter does not go down.
• sqd_hotblocks_ingestion_restarts_total{reason}: reason is the name of the error that stopped ingestion, or fork, or ended.
• API: RpcClient.getRetryKind() and getMetrics().retriedErrors in rpc-client. EvmRpcClient refines the kinds.

Trade-off

null or error text in result means the node does not have the data. Usually that is a backend lagging the head, which a retry fixes, but it can also be data the node will never have, for example pruned transaction history. The response does not tell these apart.

These retries draw on the client's shared retry budget, which evm-dump sets unbounded. So on permanently missing data the dump now retries with backoff, logging each attempt as a connection failure and counting it as no_result, instead of exiting. In the data service the budget is 5 attempts, after which the error surfaces as before. A per-request cap on no-data retries is left as a TODO in getResultValidator.

Related

• Supersedes fix(rpc-client): retry rate-limit responses returned as an HTTP 200 string result #582: its /rate limit/ check on string results misses other error text delivered the same way, for example an upstream connect-error page.
• Overlaps fix(rpc-client): retry transient 'upstream not synced' RPC errors #563, fix(evm-rpc): retry eRPC upstreams-exhausted errors instead of crashing the dumper #545 and fix(evm-rpc): retry transient eRPC ErrEndpointMissingData instead of crashing the dump #555, each of which covers one case of these classes. fix(evm-rpc): retry eRPC upstreams-exhausted errors instead of crashing the dumper #545 retries every ErrUpstreamsExhausted / ErrFailsafeRetryExceeded, and the same wrapper also arrives for a method the endpoint will never serve, so that error would be retried forever. Here the nested cause decides.
• Closes evm-rpc: in-body rate-limit error with code -32429 is not retried and crashes the dump #583.

Tests

• evm/evm-rpc/test/rpc.transient-errors.test.ts:
  • every observed error shape, transient and permanent;
  • proxy errors with nested causes;
  • HTTP 500 bodies;
  • through a local server: error text or null in place of a block throws RetryError, a malformed block throws DataValidationError, and the call recovers on retry;
  • retry kinds, and the counter after a recovered call.
• util/rpc-client/src/client.connection-error.test.ts: HTTP status classification, retry kinds, and the counts against a local server: every retry is counted, while the error that fails a request, or any error when retries are off, is not.
• util/util-internal-data-service/src/rpc-metrics.test.ts: summing across workers, keeping the counts of a closed worker, a hung or dead worker, a worker that stops answering keeping its last counts, a worker removed during a scrape counted once, the Prometheus output, and the last values surviving a failed read.
• util/util-internal-data-service/src/data-service.test.ts: a failed ingestion session increments the restart counter by error name.
• util-internal-dump-cli and evm-data-service have no test setup. Their wiring is checked by compilation and by the tests of the pieces they call.
• Results:
  • rpc-client: 34 passed;
  • evm-rpc: 325 passed, 29 skipped;
  • util-internal-data-service: 14 passed;
  • tsc: clean for rpc-client, util-internal-data-service, evm-rpc (with tsconfig.test.json), evm-data-service and util-internal-dump-cli.

🤖 Generated with Claude Code