fix(rpc): retry a bare HTTP 500 and pre-check failures of traced mined blocks

[mo4islona]

Summary

Two failures that go away when the same call is repeated were classified as permanent, so evm-dump exited and restarted instead of retrying. Closes #585.

Changes

• rpc-client, RpcClient.isConnectionError: an HTTP 500 is retried unless its body is a JSON-RPC response.
  • A body counts as one when it, or an item of a batch array, has an envelope (jsonrpc or id) and an outcome (result or error). Bitcoin Core's JSON-RPC 1.0 errors, which have id but no jsonrpc, therefore stay fatal, and so does a JSON-RPC body sent as text or bytes.
  • A text, HTML or empty body, or JSON that is not JSON-RPC, such as a bare {"error": {"message": "Internal Server Error"}}, comes from a proxy in front of the node failing and is retried.
  • Every client built on RpcClient gets this, EvmRpcClient included.
• rpc-client, HTTP transport: when a server rejects a whole batch with one JSON-RPC error envelope, the error carries rpcMethod if every call in the batch has the same method, as an error in answer to a single call does. A batch of several methods gets none.
• rpc-client, HTTP transport: a failed response labeled application/json whose body does not parse (empty, or a truncated page) is kept as text. Before, parsing it threw a SyntaxError before an HttpError was built, so the status never reached isConnectionError and the call failed whatever the status.
• evm-rpc, EvmRpcClient.isTraceStateError: insufficient funds for gas, nonce too low / nonce too high and intrinsic gas too low are retried for calls that replay transactions already in a block: debug_traceBlockByHash, debug_traceBlockByNumber, debug_traceTransaction, trace_block, trace_replayBlockTransactions, trace_replayTransaction, trace_transaction.
  • Whatever the code, and whether the text is in the outer message or in a proxy's nested cause. An execution-exception cause no longer keeps them fatal.
  • eth_call, eth_estimateGas, debug_traceCall and trace_call run a new transaction and still fail at once.
  • The method comes from the rpcMethod that RpcClient attaches to an error received in answer to a call, single or batched, or that the transport attaches to a whole-batch error. A whole-batch error of a batch of several methods stays fatal.
• retryInternalServerErrors now decides only a 500 that carries a JSON-RPC error; its doc says so.

Metrics

• The trace errors above are counted in sqd_chain_rpc_retried_errors_total under a new kind, wrong_state.
• In EvmRpcClient, a bare 500 is now counted as http, as in RpcClient, instead of internal / transient.

Trade-off

The dumper retries without limit, so an error misjudged as transient stalls the dump instead of restarting it. wrong_state is kept apart from transient so that a node that never recovers is visible in the metrics.

Not covered

A pre-check failure inside the JSON-RPC body of an HTTP 500 is judged without the method, since the transport error carries no request, and stays fatal unless retryInternalServerErrors is set.

Tests

• util/rpc-client/src/client.connection-error.test.ts:
  • 500 bodies that are retried: text, HTML, empty, bytes, a JSON string or null, JSON that is not JSON-RPC, an error object or string with no envelope, an envelope with neither result nor error, an empty array or an array of proxy errors;
  • 500 bodies kept fatal: JSON-RPC 1.0 and 2.0 errors, a 2.0 error with a null id, a result, a batch, JSON-RPC sent as text or bytes;
  • only a 500 is judged by its body: a 400 with text stays fatal, a 502 with a JSON-RPC error is retried;
  • through a local server: a 500 with plain text, HTML, an empty or malformed JSON-labeled body, a proxy error object, or no content type recovers on retry and is counted as http; a 500 with a JSON-RPC error, labeled as JSON or as text, fails with HttpError and is not counted;
  • through a local server: a 502 whose JSON-labeled body does not parse is retried; a 400 like it fails with HttpError carrying the status and the raw body; malformed JSON in a 200 still fails with RpcProtocolError;
  • through a local server: a whole-batch error envelope carries the method of a single-method batch, and none for a batch of several.
• evm/evm-rpc/test/rpc.transient-errors.test.ts:
  • the pre-check messages for each mined-transaction trace method, under the node's code, as a proxy passes them on, in a cause nested several levels deep, and case-insensitively;
  • kept fatal: the same error for eth_call, eth_estimateGas, eth_sendRawTransaction, debug_traceCall, trace_call and trace_callMany, or with no method; other execution errors of a trace (execution reverted, insufficient funds for transfer, out of gas, a base-fee check);
  • a 500 with a proxy error object is retried and counted as http; envelope-less bodies in the existing retry-kind cases replaced with JSON-RPC envelopes;
  • through a local server, on a pre-check failure: a trace call, a batch of traces failing call by call, and a batch failing as a whole recover and are counted as wrong_state; retries go on while the node keeps failing and stop once the budget is spent; a whole-batch error of a mixed batch, eth_call, and a whole-batch error of eth_call fail at once;
  • through a local server, on HTTP 500: the bare text 500 from the issue and a proxy error object recover as http; a transient JSON-RPC error recovers as transient; a permanent JSON-RPC error fails with HttpError.
• Every fix was checked by reverting it: its tests fail without it.
• Results:
  • rpc-client: 69 passed;
  • evm-rpc: 370 passed, 29 skipped;
  • tsc: clean for rpc-client and evm-rpc (with tsconfig.test.json).

🤖 Generated with Claude Code