perf(rpc): eliminate the full account scan from indexer-blobs via single-block replay - #498
Draft
ezeike wants to merge 6 commits into
Draft
perf(rpc): eliminate the full account scan from indexer-blobs via single-block replay#498ezeike wants to merge 6 commits into
ezeike wants to merge 6 commits into
Conversation
changedBlobKeys diffed accounts/pools/validators by building a map[string][]byte per side and comparing via map lookups - two full hash-map builds per indexer-blobs request at ~1.35M accounts. Add mergeChangedBlobKeys, a two-pointer merge walk over the already Pebble-sorted entry slices, and use it for accounts and validators, whose extracted keys (account/validator address bytes) match their Pebble storage-key order exactly. Pools stay on the map-based diff: poolEntryKey extracts Pool.Id's raw varint wire bytes, which diverges from KeyForPool's big-endian storage encoding at varint length boundaries (e.g. id 16383 vs 16384), so entry order there does not reliably match key order. Adds TestMergeChangedBlobKeys_MatchesMapBasedDiff, which asserts the merge-walk output equals the original map-based diff output on the same scenario, plus an empty-previous edge case.
cold_read_time was running ~3s higher than what current+previous IndexerBlob's own instrumented steps accounted for (even after weighting accounts_iterate/ validators_iterate/block_non_signers_get by the previous-reuse miss rate) -- DeltaIndexerBlobs (parsing every account/pool/validator entry into a map on both sides to diff them) and the final lib.Marshal call were both inside the coldStart timing window but had no ObserveIndexerBlobStep of their own, making them an invisible chunk of the total. Adds delta_compute and delta_marshal steps to canopy_indexer_blob_step_time so that gap is attributable instead of inferred.
…gle-block replay Replace the full account prefix scan of both heights in IndexerBlobsCached with a per-block account delta computed by replaying the single block that produced the requested height: - fsm: AccountChangeCollector classifies every account touched during one ApplyBlock call into added/changed/removed against the height-1 baseline; ApplyBlock gains collector/skipRoot params (replay-only, never commit) - fsm: IndexerBlob gains skipAccounts; AssembleAccountDeltaSides rebuilds the wire-ready sides with the reward/slash force-include and ascending address order the full scan provided for free - controller: GetAccountDelta replays the block against a TimeMachine snapshot, restoring the root DEX cache from the committed certificate as the live commit path does - rpc: IndexerBlobsCached sources account deltas from the fast path; height 2 keeps the full-scan fallback to stay byte-identical - tests: differential test pinning fast-path output byte-for-byte to the old full-scan-and-diff, plus force-include, prefix-leak, and replay regression coverage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ezeike
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The indexer-blobs endpoint previously computed its account delta by prefix-scanning the entire account set (~1.35M accounts) at both the current and previous heights and diffing the results — the dominant cost of every cache-miss request. This PR sources the account delta by replaying the single block that produced the requested height instead, plus a set of perf/observability improvements to the same endpoint.
Account-delta fast path
AccountChangeCollectorclassifies every account touched during oneApplyBlockcall into added/changed/removed against the height-1 baseline (captured on first touch, before the block's own writes shadow it).ApplyBlockgainscollector/skipRootparams — used only by throwaway replays, never by a real commit.IndexerBlobgainsskipAccounts;AssembleAccountDeltaSidesrebuilds the wire-ready sides, reproducing the reward/slash force-include and ascending-address ordering the full scan provided for free.GetAccountDeltareplays the block against aTimeMachinesnapshot, restoring the root DEX cache from the committed certificate exactly as the live commit path does (required for nested-chain correctness).IndexerBlobsCachedsources account deltas from the fast path. Height 2 keeps the full-scan fallback so its response stays byte-identical (there is no previous height to diff against).Perf / observability
DeltaIndexerBlobscanopy_indexer_blob_cold_read_timefor cache-miss reads that outlive the RPC write deadlineTesting
go build ./...andfsm/controller/cmd/rpcsuites pass with-count=1🤖 Generated with Claude Code