rpc : optimize eth_getLogs by lupin012 · Pull Request #20561 · erigontech/erigon

lupin012 · 2026-04-14T19:45:19Z

The changes are:

rawdbv3: add MaxWithCursor/MinWithCursor accepting a caller-provided cursor; Min/Max delegate to them. MapTxNum2BlockNumIter opens one cursor lazily and reuses it for all Min+Max calls across block changes, avoiding 2 cursor open/close per block in getLogsV3.
types/log: extract BuildTopicMap + FilterWithTopicMap so the topic map is built once per request instead of once per log batch; Filter() delegates to them.
eth_receipts: getLogsV3 calls BuildTopicMap once before the loop and uses TryGetCachedReceipt to skip TxnByIdxInBlock (snapshot open + RLP decode) on cache hits.
receipts_generator: receiptCache keyed by txNum (uint64) instead of txnHash, so TryGetCachedReceipt can look up receipts without resolving the hash first.

Performance misure:

filter	type	1 blk	10 blk	100 blk
usdc_only	addr	-13%	-17%	-26%
usdc_transfer	addr+topic	-26%	-30%	-32%
transfer_all	topic	-18%	-22%	-28%
multi	addr+topic	-47%	-37%	-36%
avg		-30%	-28%	-30%

Main SW

./build/bin/rpc_perf -p perf/pattern/mainnet/stress_test_eth_getLogs_15M.tar -t 10000:15,20000:15,30000:15 -r 5 -y eth_getLogs
Performance Test started
Test repetitions: 5 on sequence: 10000:15,20000:15,30000:15 for pattern: perf/pattern/mainnet/stress_test_eth_getLogs_15M.tar
Test on port: http://localhost:8545
[1.1] rpcdaemon: executes test qps: 10000 time: 15 -> success= 98.07% lat=[p50= 272µs p90= 17.54ms p95=105.63ms p99=453.71ms max= 2.79s] error=Post "http://localhost:8545": dial tcp 127.0.0.1:8545: connect: connection refused (x1)
[1.2] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=166.60ms]
[1.3] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=145.91ms]
[1.4] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=142.72ms]
[1.5] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=155.32ms]

[2.1] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=166.88ms]
[2.2] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=167.75ms]
[2.3] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=156.63ms]
[2.4] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=153.00ms]
[2.5] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=124.04ms]

[3.1] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=176.64ms]
[3.2] rpcdaemon: executes test qps: 30000 time: 15 -> success= 99.65% lat=[max=199.70ms] error=503 Service Unavailable (x1)
[3.3] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=219.24ms]
[3.4] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=204.67ms]
[3.5] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=176.81ms]

Current Branch

./build/bin/rpc_perf -p perf/pattern/mainnet/stress_test_eth_getLogs_15M.tar -t 10000:15,20000:15,30000:15 -r 5 -y eth_getLogs -P
Performance Test started
Test repetitions: 5 on sequence: 10000:15,20000:15,30000:15 for pattern: perf/pattern/mainnet/stress_test_eth_getLogs_15M.tar
Test on port: http://localhost:8545
[1.1] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=576.59ms]
[1.2] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=122.24ms]
[1.3] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max= 86.22ms]
[1.4] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max=120.32ms]
[1.5] rpcdaemon: executes test qps: 10000 time: 15 -> success=100.00% lat=[max= 87.61ms]

[2.1] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=100.07ms]
[2.2] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=139.87ms]
[2.3] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=116.74ms]
[2.4] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=117.39ms]
[2.5] rpcdaemon: executes test qps: 20000 time: 15 -> success=100.00% lat=[max=119.72ms]

[3.1] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=134.90ms]
[3.2] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=124.48ms]
[3.3] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=145.09ms]
[3.4] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=125.88ms]
[3.5] rpcdaemon: executes test qps: 30000 time: 15 -> success=100.00% lat=[max=141.82ms]

…he fast path - rawdbv3: add MaxWithCursor/MinWithCursor accepting a caller-provided cursor; Min/Max delegate to them. MapTxNum2BlockNumIter opens one cursor lazily and reuses it for all Min+Max calls across block changes, avoiding 2 cursor open/close per block in getLogsV3. - types/log: extract BuildTopicMap + FilterWithTopicMap so the topic map is built once per request instead of once per log batch; Filter() delegates to them. - eth_receipts: applyFiltersV3 opens one shared cursor for Min+MaxWithCursor. getLogsV3 calls BuildTopicMap once before the loop and uses TryGetCachedReceipt to skip TxnByIdxInBlock (snapshot open + RLP decode) on cache hits. - receipts_generator: receiptCache keyed by txNum (uint64) instead of txnHash, so TryGetCachedReceipt can look up receipts without resolving the hash first. Add TryGetCachedReceipt: checks receiptCache (per-tx, hot path) then receiptsCache (block-level, populated only by eth_getBlockReceipts). - cmd/scripts: add bench_eth_getlogs.py, bench_getlogs.sh, bench_stress_getlogs.sh for before/after latency and throughput benchmarking. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

This PR optimizes the eth_getLogs RPC hot path by reducing repeated work (cursor opens, topic-map construction, and receipt retrieval) and by improving receipt caching lookups.

Changes:

rawdbv3: add MinWithCursor/MaxWithCursor and reuse a lazily-opened kv.MaxTxNum cursor in TxNums2BlockNums iteration to cut cursor churn.
types/log: factor out BuildTopicMap + FilterWithTopicMap so topic lookups are built once per request instead of per batch.
receipts: add a receipt-cache fast path keyed by txNum to avoid TxnByIdxInBlock on cache hits; update getLogsV3 to use it.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
rpc/jsonrpc/receipts/receipts_generator.go	Switch per-tx receipt cache key to `txNum` and add `TryGetCachedReceipt` fast path.
rpc/jsonrpc/eth_receipts.go	Reuse `kv.MaxTxNum` cursor for Min/Max lookups; prebuild topic map once; use cached receipt fast path in `getLogsV3`.
rpc/jsonrpc/erigon_receipts.go	Prebuild topic map once and reuse it for log filtering.
execution/types/log.go	Introduce reusable topic-map builder and topic-map-based filter helper; keep `Filter` as wrapper.
db/kv/rawdbv3/txnum.go	Add `MinWithCursor`/`MaxWithCursor` and reuse one cursor inside `MapTxNum2BlockNumIter`.
db/kv/rawdbv3/txnum_test.go	Add benchmark covering cursor open/close overhead for iterator block changes.
cmd/scripts/bench_stress_getlogs.sh	Add script to run/compare rpc_perf stress tests for `eth_getLogs`.
cmd/scripts/bench_getlogs.sh	Add wrapper script for the direct HTTP benchmarking tool.
cmd/scripts/bench_eth_getlogs.py	Add direct HTTP benchmark tool measuring cold/warm `eth_getLogs` latencies across scenarios.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…move stress script - bench_eth_getlogs.py: fix from=end-n+1 so n_blocks matches the inclusive range - BuildTopicMap: skip allocation for wildcard positions (len(v)==0 → nil entry) - log_test.go: add TestFilterWithTopicMapEquivalence — equivalence of Filter vs FilterWithTopicMap(BuildTopicMap) across wildcard, exact, alternatives, maxLogs cases - receipts: add TestTryGetCachedReceipt covering blockHash hit, stale reorg miss, postState mismatch, receiptsCache txIndex fallback, txIndex<0 and out-of-bounds - remove bench_stress_getlogs.sh (rpc_perf launched directly) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- add t.Parallel() to all subtests (each creates its own generator, no shared state) - switch to testify require assertions to match handler_test.go style Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-19T01:17:43Z

+		if r, ok := api.receiptsGenerator.TryGetCachedReceipt(header.Hash(), txNum, txIndex); ok {
+			for _, filteredLog := range r.Logs.FilterWithTopicMap(addrMap, topicMap, 0) {
+				if maxResults != 0 && len(logs) >= maxResults {
+					return nil, fmt.Errorf("%s: %d", errExceedLogResults, maxResults)


In the TryGetCachedReceipt fast-path, exceeding maxResults returns a plain error (via fmt.Errorf), while the other branches return *rpc.InvalidParamsError. This makes eth_getLogs error codes/messages depend on cache hits, which is observable and inconsistent for clients. Return the same error type here as in the other branches (InvalidParamsError with Message formatted the same way).

Suggested change

return nil, fmt.Errorf("%s: %d", errExceedLogResults, maxResults)

return nil, &rpc.InvalidParamsError{

Message: fmt.Sprintf("%s: %d", errExceedLogResults, maxResults),

}

Copilot · 2026-04-19T01:17:44Z

+	tx, err := db.BeginRo(ctx)
+	require.NoError(b, err)
+	defer tx.Rollback()
+	b.Cleanup(tx.Rollback)
+
+	// Benchmark 1: baseline
+	// Min() and Max() each open their own kv.MaxTxNum cursor on every block change.
+	b.Run("CurrentCursorPerCall", func(b *testing.B) {
+		b.ReportAllocs()
+		for i := 0; i < b.N; i++ {
+			it := TxNums2BlockNums(ctx, tx, TxNums, stream.Array(txNumsPerBlock), order.Asc)
+			for it.HasNext() {
+				if _, _, _, _, _, err := it.Next(); err != nil {


This benchmark opens a read-only transaction (tx := db.BeginRo) outside of the sub-benchmarks and then uses it inside b.Run closures. kv.Tx is documented as only usable from the goroutine that created it; sub-benchmarks run their functions in separate goroutines, so reusing tx across them violates that contract and can lead to undefined behavior/races. Open the RoTx inside each b.Run (and reuse it within that sub-benchmark’s goroutine), or use db.View per sub-benchmark.

AskAlexSharov requested a review from Copilot April 15, 2026 02:52

Copilot started reviewing on behalf of AskAlexSharov April 15, 2026 02:53 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

lupin012 and others added 3 commits April 15, 2026 07:26

rpc/jsonrpc/receipts: polish TryGetCachedReceipt test

3884825

- add t.Parallel() to all subtests (each creates its own generator, no shared state) - switch to testify require assertions to match handler_test.go style Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'main' into lupin012/opt_eth_getLogs

547e56d

lupin012 added performance RPC labels Apr 15, 2026

Merge branch 'main' into lupin012/opt_eth_getLogs

82e5317

lupin012 marked this pull request as ready for review April 18, 2026 21:21

lupin012 requested review from AskAlexSharov, Giulio2002, canepat, mh0lt, sudeepdino008 and yperbasis as code owners April 18, 2026 21:21

Merge branch 'main' into lupin012/opt_eth_getLogs

7a6bc1d

AskAlexSharov requested review from JkLondon and Copilot April 19, 2026 01:13

Copilot started reviewing on behalf of AskAlexSharov April 19, 2026 01:13 View session

Copilot AI reviewed Apr 19, 2026

View reviewed changes

JkLondon approved these changes Apr 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc : optimize eth_getLogs#20561

rpc : optimize eth_getLogs#20561
lupin012 wants to merge 6 commits intomainfrom
lupin012/opt_eth_getLogs

lupin012 commented Apr 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 19, 2026

Uh oh!

Copilot AI Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-					return nil, fmt.Errorf("%s: %d", errExceedLogResults, maxResults)
+					return nil, &rpc.InvalidParamsError{
+						Message: fmt.Sprintf("%s: %d", errExceedLogResults, maxResults),
+					}

Conversation

lupin012 commented Apr 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants