Rearchitect live ingestion for high-TPL ledgers: pipeline, batched 5-way COPY commits, once-per-op decoding, staggered compression - #682
Open
aditya1702 wants to merge 28 commits into
Conversation
The wallet_db_query_duration_seconds histogram topped out at 0.38s (ExponentialBuckets(0.0001, 2.5, 10)), so every multi-second bulk COPY landed in +Inf and histogram_quantile panels clipped at ~0.38s, under-reporting exactly the queries being optimized. Widen to ExponentialBuckets(0.0001, 3, 12) — 0.1ms through ~17.7s.
…-ledger lookup The oldest-ledger lookup (SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1) has two consumers — backfill gap detection's left bound (GetOldestLedger) and the hourly reconcile_oldest_cursor TimescaleDB job — and neither had an ordered path: the transactions migration dropped TimescaleDB's default partition-column index as consumer-less, an audit that counted WHERE-clause consumers and missed that this query consumes the index's ordering. Without it the planner pulls the first row from every chunk instead of running an ordered ChunkAppend that stops at the oldest (12-187s on an 87GB DB; the hourly job alone accounted for 45% of all DB disk-read time and flushed shared_buffers every run). Keep the default index (fresh databases get it from create_hypertable; already-migrated environments need a one-time manual `CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC)`) and drop the to_id tie-break from both query sites: close times increase strictly, so rows sharing a ledger_created_at carry the same ledger_number and the tie-break could only force an incremental sort on top of the index's pathkeys. Covered by a new test that runs the reconciliation job via run_job against out-of-order chunks.
…flag ingestService carried four fields (appTracker, getLedgersLimit, knownContractIDs, contractMetadataService) and Indexer two (ingestionMetrics, networkPassphrase) that were assigned at construction and never read — consumers get these through their own constructors or through internal/ingest.Configs. Deleting them also drops AppTracker, GetLedgersLimit, and ContractMetadataService from IngestServiceConfig (the checkpoint service keeps its own metadata-service wiring). Also removes the --latest-ledger-cursor-name flag, which only fed a deprecation warning, and the phantom load_current_state phase label from the ProtocolStateProcessingDuration help text (never emitted).
Every remaining parameter of persistLedgerData had exactly one production value: cursorName was always the latest-ledger cursor (the blind-Update branch was reachable only from tests), ledgerMeta was never nil, and the numTxs/numOps returns only fed two counters the buffer already knows via GetNumberOfTransactions/GetNumberOfOperations. The signature is now (ctx, ledgerSeq, ledgerMeta, plan, contractData, buffer) error, the cursor write is unconditionally guarded, and the contractDataMemo nil-receiver branch (test-only) is gone. IndexerBufferInterface shrinks to its real seam — the indexer's fold-in plus the five getters insertIntoDB reads; everything else is reached on the concrete *IndexerBuffer. Tests now exercise the guarded cursor path persistLedgerData actually runs in production.
…Backoff persistLedgerDataWithRetry (né ingestProcessedDataWithRetry) hand-rolled the same attempt loop, exponential backoff, cap, and context handling that utils.RetryWithBackoff already provides — the body is now a single RetryWithBackoff call with isPermanentPersistError as the classifier. Metric semantics preserved: transient retries count RetriesTotal, permanent errors count ErrorsTotal and return immediately, exhaustion counts both plus RetryExhaustionsTotal, and context cancellation counts neither. RetryWithBackoff's exhaustion return now wraps the exported sentinel utils.ErrRetriesExhausted alongside the final error, which is how the wrapper tells exhaustion apart from a permanent-error exit.
…ist stages The live loop ran fetch (~0.3s of waiting), process_ledger (~0.8s of Go CPU), and the DB persist (~1.1s) strictly in sequence, so the per-ledger time was their sum — above a 2s ledger cadence at high transaction volume even though each stage individually fits. The loop is now a three-stage errgroup pipeline over depth-1 channels: while ledger N persists, N+1 processes and N+2 is fetched, making the ledger time the slowest stage instead of the sum. Persist stays strictly sequential in ledger order (the guarded cursor and per-protocol CAS chain advance N-1 → N) and is the only writing stage, so the advisory-lock liveness probe moves there. Classification planning also stays on the persist stage: its known-hash lookup is a non-transactional pool read that must observe the previous ledger's committed protocol_wasms rows. Two IndexerBuffers rotate through a freeBuffers channel, cleared only at take so a clear can never tear a persist still reading the maps the getters alias. Any stage error cancels the pipeline and the process exits, resuming from the cursor on restart — the Duration metric now sums the ledger's stage times, since wall time under a pipeline would count queue wait.
…with coordinated late commits The live persist ran its three COPY families — transactions(+accounts), operations(+accounts), state_changes — sequentially inside one transaction on one connection, so the insert phase cost their sum and a single Postgres backend did all the index maintenance. They now stream concurrently on three sibling connections, each in its own transaction, while the coordinating transaction stages everything else (assets, contracts, classification, protocol state, token changes, cursor). The table sets are disjoint with no FKs among them, and every goroutine only reads the quiescent buffer. Commits are the visibility point and are held until all four transactions have done their work: siblings commit first (sub-ms each), the coordinating transaction — whose cursor is the authority on which ledgers exist — strictly last. A failure before the first commit rolls everything back and stays retryable exactly as before; a failure after it wraps the new ErrPartialPersist sentinel, which isPermanentPersistError classifies as fatal, because COPY has no ON CONFLICT and re-running the ledger would collide on primary keys. The only crash state this ordering can produce is orphaned bulk rows for the single ledger past the committed cursor, so startup runs IngestStoreModel.DeleteRowsAboveLedger before resuming: one transaction of TOID-bounded deletes (rows of ledgers > cursor are exactly rows with TOID >= toid.New(cursor+1,0,0)) across the five bulk tables, kept chunk-local by each table's chunk skipping. Backfill keeps the single-transaction insertIntoDB path unchanged.
aditya1702
force-pushed
the
replay-loadtest-ingest-pipeline
branch
from
August 10, 2026 21:23
0d92cb5 to
0a18a10
Compare
…the schedule interval Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fire at the same instant and compress their just-closed chunks concurrently — an I/O storm that starves the persist stage (measured on the loadtest rig: persist p50 0.68s -> 3.0s for 18 of every 60 minutes). Converge each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot offset, so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts, preserving next_start and run history.
…ommits When the process stage finishes ledgers faster than persist drains them, the persist stage now folds up to --live-persist-max-batch-size (default 5) consecutive ledgers into one commit set: each sibling COPY streams the whole batch on its connection, the coordinating transaction stages the batch's ledgers in order, and the commit barrier fires once — amortizing COPY setup, index-page churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is unchanged. A ledger with classification inputs always opens its own batch: its plan's pool reads see exactly what the previous batch committed, which preserves the deployed-contract-sees-prior-wasm invariant. The cursor stays the authority — one guarded update lands on the batch's last ledger, and a pre-commit failure rolls back and retries the whole batch. wallet_ingestion_persist_batch_size observes coalescing; per-ledger duration histograms record each ledger's amortized share so panels stay comparable across batch sizes.
…on WAL flush SET LOCAL synchronous_commit = off on each sibling session removes up to three serialized fsync waits per persist. Durability is unchanged: the coordinating transaction commits synchronously and strictly last, and its flush covers all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings; rows a crash could lose are exactly the unacknowledged ones startup reconciliation deletes.
Each pipe's reader now buffers two decoded frames beyond the one in flight, so a writer streams its next frames through the FIFO while the consumer merges and processes earlier ledgers. With lockstep delivery every GetLedger waited on the slowest writer's in-flight frame — a constant ~0.6s/ledger at full volume that the one-frame handoff could not hide. Backpressure still bounds the writers, now with three frames of slack; an epoch's terminating error stays ordered behind its frames, so restart handling is unchanged.
…ary key TOID encoding makes a state change's parent transaction to_id derivable from its operation_id (to_id = operation_id &^ 0xFFF), so BatchGetByOperationID and BatchGetByOperationIDs now bind (ledger_created_at, to_id, operation_id) — a primary-key prefix seek — instead of filtering operation_id across every entry sharing the timestamp (measured 7 vs 307 buffers on a single-chunk fixture). That leaves idx_state_changes_operation_id with no consumer; it is removed. idx_state_changes_account_category narrows to idx_state_changes_account_id (account_id + the PK sort key): the category and reason columns only ever seeded an ordered seek for the one filter shape that pins both, while every other shape and BatchGetAccountStateChangesByToIDs seek on account_id alone. Filtered variants scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need the one-time manual DDL (drop both old indexes, create the narrowed one); TimescaleDB rejects CREATE INDEX CONCURRENTLY on hypertables.
…lings The live persist path's sibling COPY streams grow from three to five: transactions_accounts and operations_accounts — the two largest index-maintenance payloads, each carrying an account_id-leading unique PK — stream concurrently with their parent tables instead of serially behind them. TransactionModel and OperationModel each split BatchCopy into a parent-table COPY and BatchCopyAccounts for the link table, with the link COPY's duration and batch size recorded under its own metric labels rather than folded into the parent's. Each sibling now writes exactly one table, so the disjointness at the commit barrier is per-table; the backfill path calls the five inserts in the old order and is behaviorally unchanged. Persist holds six connections at its commit barrier; the pool default rises to 12 so the persist path never queues on Acquire behind the advisory-lock session or pool-side classification reads.
… random order The native_balances and trustline_balances batch upserts arrive in Go map-iteration order and unconditionally rewrite every conflicting row. On the loadtest rig the pattern measured 60+ heap buffer touches per row (vs ~6 healthy) against a table bloated to 97% free space by the churn. Two changes: rows sort by the primary-key columns before the UNNEST arrays are built, so the btree descent and heap touches run in key order; and the DO UPDATE carries an IS DISTINCT FROM guard, so an identical row produces no new tuple version, no dead tuple, no WAL, and no index churn.
…cores runtime.NumCPU() reports the node's cores in a container with a CPU limit, so a 6-core pod on a 36-core node ran 72 indexer workers over 6 schedulable threads — pure scheduler churn on CPU-bound work. GOMAXPROCS honors the limit.
The buffer stored transactions keyed by hash while tracking their participants keyed by ToID. The streaming-loadtest backend's merged bootstrap ledgers can carry the same envelope at several tx-set positions — distinct ToIDs, one hash — and the two maps then disagreed: duplicate transactions were silently dropped from the transactions COPY while their participant links survived, whose ledger_created_at lookup missed and COPYed the zero timestamp into a year-0001 chunk (observed on the rig: 6 orphaned transactions_accounts rows, 5 missing transaction rows across bootstrap ledgers). Real networks cannot repeat a hash within or across ledgers, so both maps now share the ToID key domain by construction, mirroring the operations pair. BatchCopyAccounts on both link tables now errors on a ToID/opID with no parent row instead of silently writing a zero timestamp.
…per processor Every state-change and effects processor independently called Transaction.GetOperationChanges for the same operation, and the SDK accessor re-decodes the op's meta, re-allocates, and re-sorts the changes on every call — 19% of process-stage CPU in GetChangesFromLedgerEntryChanges, 14% in sortChanges, and a large share of GC pressure on a production profile. All of an operation's processors share one TransactionOperationWrapper, so Changes() now memoizes the extraction on the wrapper; all 14 call sites route through it, audited read-only (the memoized slice is shared — callers must not mutate it or write through Pre/Post). A wrapper copied to describe a different operation resets its memo so the cache always matches Index. BenchmarkProcessRealLedger on the five real-ledger fixtures: 40-50% less wall time, ~50% fewer bytes and ~57% fewer allocations per ledger.
aditya1702
changed the title
Batching is opt-in per deployment: at mainnet/testnet cadence the close time comfortably exceeds the persist time, so every ledger commits on its own; high-TPL/short-block deployments raise the flag to amortize backlogs.
…dger's close time DeleteRowsAboveLedger scanned every chunk TOID chunk-skipping stats could not exclude — stats only cover compressed chunks, and three of the five tables have no TOID-leading index — which took over 10 minutes on a loadtest DB carrying hours of full ledgers, long enough that the unread meta pipes wedged all three generators into liveness-probe kills. Close times are monotone, so every orphan above the cursor carries ledger_created_at at or after the cursor ledger's own close time; binding that as a partition-column predicate statically excludes all older chunks regardless of compression, and turns state_changes' scan into a primary-key range seek. A cursor ledger with no transactions row leaves the bound unresolvable and the deletes fall back to the unbounded scan.
…allel apply-load's meta write is synchronous: core does not start generating its next ledger until the consumer drains the current frame, and the reader drained only as fast as it decoded — putting XDR decode inside every producer's ledger cycle (measured: cycle = generation + decode, which capped the merged stream well under target cadence). Each pipe's reader now slurps a record's raw bytes at transfer speed and hands them to a per-pipe decode goroutine, so the producer starts its next ledger while the previous frame decodes. Frame and error ordering are preserved: a drain or decode error is always the last element delivered and ends the epoch exactly as before.
The metadata fetcher's fetched/failure caches are in-memory only, so a token whose contract_tokens row already carries metadata was still re-fetched over RPC — once per process lifetime after every restart on a real network, and forever on a 5-minute backoff against a deployment whose RPC can never resolve metadata (the loadtest rig's dead endpoint with externally seeded rows), where each retry burned the ~850ms simulate backoff ladder inside prepare_classification. After Apply persists a batch's rows, contracts whose row has metadata are marked fetched, making the database the durable cache. Tokens whose metadata is genuinely missing stay unmarked and keep the existing backoff retry.
The classification-safety cut isolated any ledger carrying protocol wasm/contract observations, but a plan's pool reads are only unsound for inputs no committed batch has classified yet — and re-observations of known contracts are the overwhelmingly common case (synthetic loadtest traffic re-observes the same token contracts every ledger, which cut every batch to size one and disabled batching entirely; goroutine dumps showed the process stage starved of buffers held by the always-cut pending queue). The persist goroutine now keeps seen-sets of wasm hashes and contract IDs folded in after each successful commit; only a ledger introducing unseen inputs opens its own batch. A rolled-back batch marks nothing, and a restart starts empty — conservative until re-warmed.
The seen-set comment block split the struct's alignment group, leaving the fields below it on the wider alignment. golangci-lint's gofmt check fails on it; `make check` cannot catch this because its `tidy` step rewrites the file before the `fmt` gate reads it.
…eachable A buffer stays checked out from the moment the process stage takes it until the batch carrying it commits, so the rotation needs 2*cap buffers to sustain a full batch: cap for the batch in flight, cap-1 queued in processed, one being filled. It held cap+1, so process starved on freeBuffers after filling (cap+1)-k and the batch settled at the fixed point k = (cap+1)-k. The batch size was therefore pinned below the configured cap at every load — loadtest measured exactly 3.00 against a cap of 5 across 17 sample windows spanning 3.9k to 9.8k tx/s, and the new test measures 2.04 against a cap of 3 when the pool is sized the old way.
…e bar The phase histogram stepped 0.5 -> 1 -> 2 seconds, so a stage sitting at 0.9s and a stage sitting at 1.9s both reported the same p99 bucket. The pipeline's contract is that the slowest stage's p99 stays under the ledger close time, so the metric could not grade the thing it exists to grade. Boundaries now land exactly on 0.6s (Phase-3 block rate) and 1s (the loadtest rig's 1.67x-sized merged ledgers), with 0.5 and 0.75 around them.
Pairing metas with envelopes needs every envelope's hash, and building it was the pipeline's largest serial stretch: on the loadtest rig it ran ~0.37s single-threaded ahead of the fan-out, 41% of the process stage, of which 85% was the hashing itself. Reading transactions out of a LedgerCloseMeta now hashes envelopes across the indexer's worker pool instead of on the caller's goroutine. Three costs went with it. The network id is a hash of the passphrase alone, so it is computed once per ledger rather than once per transaction. Each chunk marshals through one xdr.EncodingBuffer, which reuses its scratch space across transactions instead of allocating a buffer per hash — this path alone was 8.5% of everything the ingest process allocated. And the envelope's transaction is tagged in place rather than copied by value. On the real pubnet fixtures, where the fan-out barely engages at a few hundred transactions per ledger: 2.8-3.8x faster, 4.3-8.6x fewer allocations. Also restores the independence of the two reader-based oracles in the test suite. They materialized their transactions through the same function they were the oracle for, so they now go through getLedgerTransactionsViaReader, which drives the SDK reader and shares no code with the fan-out. That helper is the merge gate for this change too, via TestGetLedgerTransactions_EquivalenceOnRealLedgers.
isLiquidityPool and isClaimableBalance ran a full strkey decode — base32, CRC16 validation, and a byte copy of the input — on both endpoints of every transfer event. Those endpoints are ordinary account and contract addresses almost every time, so nearly all of that work only ever concluded "no". A strkey version byte is a multiple of 8, so its top five bits are exactly what the first base32 character encodes: only an 'L' can decode to a liquidity pool and only a 'B' to a claimable balance, whatever the payload. Checking that character first skips the decode for everything else, and a string that passes it still goes through the decode, so validation is unchanged. Together this was 13.6% of the token-transfer processor on the loadtest rig.
aditya1702
force-pushed
the
replay-loadtest-ingest-pipeline
branch
from
August 11, 2026 22:14
344e81b to
0c05682
Compare
…wrapper memos The persist stage extracted each ledger's ContractData changes through tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group — work the process stage had already done and memoized on the shared TransactionOperationWrapper. On the loadtest rig that duplicate path was 8.7s of a 239.7s profile (3.6% of ingest CPU, plus its allocation/GC halo), all of it on the serial persist goroutine. processTransaction now collects each successful transaction's ContractData changes from the wrappers' memoized Changes() slices, mirroring GetChanges' composition per meta version (tx-level before, operations ascending, tx-level after); transaction-level segments are only materialized when a type-tag scan of the raw group mentions ContractData at all. Results fold into the ledger buffer in transaction order and reach the persist stage on the buffer it already carries, replacing the contractDataMemo and the transactions plumbing through processedLedger; ProcessLedger no longer returns the materialized transactions. Merge gate: TestProcessLedgerTransactions_ContractDataChangesMatchReaderExtraction compares the buffer's collection against the reader-based tx.GetChanges() reference over every real-ledger fixture; synthetic tests pin the success gate, the meta-version guard, wrapper-memo serving, and the no-wrapper fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rearchitects live ingestion for sustained high-TPL ledgers: a fetch ‖ process ‖ persist pipeline with bounded batched commits, five parallel COPY streams, once-per-operation change decoding, staggered TimescaleDB compression, and a consumer-justified index diet on
state_changes. Also fixes a silent transaction-loss bug in the indexer buffer and hardens the balance upserts against heap bloat.Why
The network roadmap shrinks block close time 2s → 1s → 600ms while per-ledger transaction volume stays roughly constant, so live ingestion must process one full ledger in under the block time. The engineering contract is p99 of the slowest pipeline stage ≤ block time. On the loadtest rig (Phase-1 shape: 6000 SAC + 1000 classic + 4000 custom-token + 1500 soroswap TPL), serial ingestion took 7.7–8.4 s/ledger. The changes here address, in measured order of impact: serialized insert streams, the hourly compression I/O storm (persist p50 0.68 s → 3.0 s for 18 of every 60 minutes), redundant per-processor change decoding (19% of process CPU in
GetChangesFromLedgerEntryChanges, 14% in the SDK'ssortChanges), and index maintenance on the highest-volume table.How
Pipeline (fetch ‖ process ‖ persist)
ingestLiveLedgersruns three stages connected by channels: while ledger N persists, N+1 processes and N+2 is fetched. Ledger time becomes the slowest stage, not the sum. Persist stays strictly sequential in ledger order (guarded cursor and per-protocol CAS chains advance N-1 → N); any stage error cancels the pipeline and the process resumes from the cursor after restart.wallet_ingestion_duration_secondssums the stage times so panels stay comparable to the old serial values; throughput lives inrate(wallet_ingestion_ledgers_total)and per-stage numbers inwallet_ingestion_phase_duration_seconds{phase}, whose buckets bracket the contract: boundaries sit on 0.6s (the 600ms block target) and 1s (the rig's 1.67×-sized merged ledgers), with 0.5 and 0.75 around them. The previous 0.5 → 1 → 2 steps could not separate a stage at 1.05s from one at 1.95s, so the metric could not grade the bar it exists to measure.Five sibling COPY streams + coordinated late commits
Each persist opens five sibling transactions —
transactions,transactions_accounts,operations,operations_accounts,state_changes, each owning exactly one table — plus a coordinating transaction for everything else (assets, contracts, classification, protocol state, token changes, cursor). All the slow work happens uncommitted; commits fire only after every stream succeeds, siblings first, coordinator strictly last. The cursor is the authority: the only crash state is orphaned bulk rows above the committed cursor, whichDeleteRowsAboveLedgerremoves at startup. Failure before the first commit is retryable; after it,ErrPartialPersistis fatal.Sibling sessions run
SET LOCAL synchronous_commit = off: the coordinator's synchronous, strictly-last commit flushes all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings — up to five fsync waits per persist removed with no durability change.Bounded batched commits (
--live-persist-max-batch-size, default 1)When process finishes ledgers faster than persist drains them, persist coalesces up to N consecutive ledgers into one commit set — each sibling streams the whole batch, the coordinator stages ledgers in order, the cursor lands on the batch's last ledger — amortizing COPY setup, index churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is exactly the unbatched persist. Batching is opt-in per deployment: the default 1 keeps commit-per-ledger on mainnet/testnet, whose 5s close time comfortably exceeds the persist time; high-TPL/short-block deployments (the loadtest rig, future network phases) raise it. A ledger carrying classification inputs no committed batch has yet classified opens its own batch, preserving the plan's requirement that its pool reads see the previous commit; re-observations of already-classified contracts ride mid-batch.
wallet_ingestion_persist_batch_sizeobserves coalescing; per-ledger histograms record amortized shares.The cap also sizes the process↔persist buffer rotation. A buffer stays checked out from the moment process takes it until the batch carrying it commits, so sustaining a full batch needs 2N+1 buffers: N held by the batch in flight, N-1 queued behind it, one being filled. Sized any tighter, process starves on free buffers before the queue can refill and the batch settles at a fixed point strictly below N — the configured cap becomes unreachable at every load, which the histogram reports as a suspiciously constant value. A test drives the pipeline with the process stage ahead and fails if the mean batch falls short of the cap. Each buffer retains a merged ledger's maps, so raising N raises ingest's memory floor with it.
Process stage: decode once, hash in parallel
Every state-change/effects processor called
Transaction.GetOperationChangesindependently, and the SDK re-decodes, re-allocates, and re-sorts on each call. All of an operation's processors share oneTransactionOperationWrapper, soChanges()now memoizes the extraction; all 14 call sites route through it (audited read-only — the memoized slice is shared).BenchmarkProcessRealLedgeron five real pubnet ledgers: 40–50% less wall time, ~50% fewer bytes, ~57% fewer allocations. The ledger-indexer pool also sizes offGOMAXPROCSinstead of node cores (a 6-core pod on a 36-core node ran 72 workers over 6 schedulable threads).Reading a ledger's transactions was what remained of the stage's serial time. Pairing metas with envelopes needs every envelope's hash — the transaction set carries envelopes in agreement order while metas are sorted by hash — and the SDK reader computes all of them on the caller's goroutine before returning the first transaction, so no amount of overlapping with the fan-out that follows can recover it. On the rig that was ~0.37 s single-threaded per ledger, 41% of the process stage. Hashing now runs across the indexer's worker pool, one contiguous chunk per schedulable thread, and three costs went with it: the network id is a hash of the passphrase alone, so it is computed once per ledger instead of once per transaction; each chunk marshals through one reused
xdr.EncodingBufferrather than allocating a buffer per hash — that path alone was 8.5% of everything the ingest process allocated, against a GC taking 25.7% of its CPU; and the envelope's transaction is tagged in place instead of copied by value. On the real pubnet fixtures, where a few hundred transactions per ledger barely engage the fan-out,BenchmarkGetLedgerTransactionsis 2.8–3.8× faster with 4.3–8.6× fewer allocations. A differential test against the SDK reader over every committed fixture is the merge gate, and it also restores the independence of the two other reader-based oracles in the package, which had been materializing their transactions through the very function they were the oracle for.The wrapper memo also feeds ContractData extraction now. The persist stage had been deriving each ledger's ContractData changes through
tx.GetChanges(), which rebuilds and ledger-key-sorts every operation's change group from the meta — the same work the wrappers had already done and memoized one stage earlier. Measured on the rig, that duplicate path was 3.6% of ingest CPU plus its allocation halo, all of it on the serial persist goroutine.processTransactionnow collects each successful transaction's ContractData changes from the memoized slices, mirroringGetChanges' per-meta-version composition (transaction-level before, operations ascending, transaction-level after), and folds them into the ledger buffer persist already receives — replacing the lazy extraction memo and the transactions plumbing through the process→persist handoff. Transaction-level segments are materialized only when a type-tag scan of the raw group mentions ContractData at all. The reader-basedtx.GetChanges()reference stays the merge gate over every committed fixture.isLiquidityPoolandisClaimableBalancestop decoding too. They ran a full strkey decode — base32, CRC16, and a copy of the input — on both endpoints of every transfer event, which are ordinary account and contract addresses almost every time. A strkey version byte is a multiple of 8, so its top five bits are exactly what the first base32 character encodes, and only anLor aBcan decode to those two version bytes whatever the payload. Testing that character first skips the decode for everything else while anything that passes it still goes through the full decode, so validation is unchanged. 13.6% of the token-transfer processor on the rig.Compression staggering
Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fired at the same instant and compressed ~an hour of chunks concurrently — measured 4.5× persist degradation for 18.5 min of every hour on the rig.
configureHypertableSettingsnow converges each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot (initial_start = date_bin(interval, now(), epoch) + interval + i/5·interval), so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts.Index diet on
state_changesTOID encoding derives a state change's parent transaction (
to_id = operation_id &^ 0xFFF), soBatchGetByOperationID(s)now bind(ledger_created_at, to_id, operation_id)— a PK-prefix seek, measured 7 vs 307 buffers — andidx_state_changes_operation_idis dropped.idx_state_changes_account_categorynarrows toidx_state_changes_account_id(account_id + the PK sort key): category/reason filters scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need a one-time manual DDL (drop both old indexes, create the narrowed one; TimescaleDB rejectsCREATE INDEX CONCURRENTLYon hypertables).Correctness fixes
ledger_created_at(materializing a year-0001 chunk). Both maps now share the ToID key domain, mirroring the operations pair;BatchCopyAccountserrors on a link row with no parent instead of writing a zero timestamp. Real networks cannot repeat a hash, so this was latent there and triggered only on the rig.reconcile_oldest_cursorjob had been 45% of all loadtest-DB disk reads; the defaulttransactions_ledger_created_at_idxis no longer dropped and the deadto_idtie-break is removed.native_balances,trustline_balances): rows sort by PK before UNNEST (map-iteration order scattered thousands of random btree/heap probes; measured 60+ heap buffer touches/row vs ~6 healthy) and the DO UPDATE carries anIS DISTINCT FROMguard so identical rows produce no dead tuple, WAL, or index churn.DeleteRowsAboveLedgerscanned every chunk TOID chunk-skipping stats could not exclude (stats only cover compressed chunks; three of five tables have no TOID-leading index) — over 10 minutes on a DB carrying hours of full ledgers, long enough that unread meta pipes wedged the loadtest generators into liveness-probe kills. Close-time monotonicity makes every orphan'sledger_created_at≥ the cursor ledger's own close time, so the partition-column bound statically excludes all older chunks and turnsstate_changesinto a PK range seek.contract_tokensrow already carries metadata is never re-fetched over RPC (previously once per process lifetime per token after every restart, and forever on a 5-minute backoff against an RPC that can never resolve it); genuinely missing metadata keeps the existing backoff retry.utils.RetryWithBackoff.Loadtest backend
Per-pipe readers split into a raw drain and a parallel decoder: the drain slurps each framed XDR record at pipe-transfer speed (apply-load's meta write is synchronous, so core cannot start generating its next ledger until the frame drains — decoding inline put the XDR decoder inside every producer's ledger cycle), and a per-pipe goroutine decodes from memory, buffering two frames ahead. Frame and error ordering are preserved; producer restarts roll epochs exactly as before.
Measured results (loadtest rig, Phase-1 shape)
The two rig windows are phase-matched — same 10 minutes past the hour, so the same age of the
one-hour active chunk and the same overlapping compression policy — because that turned out to
matter more than anything measured here. Persist swings 0.789 s to 3.369 s within a single
hour purely as the active chunk's indexes grow and then reset on the roll, with batching rising
to the cap and the rig flipping from supply-bound to sink-bound at the top of the ramp. Any
measurement on this rig that does not control for chunk phase is dominated by a 4.3× effect. It
is also a fresh argument for a shorter chunk interval: 15-minute chunks would cap how far up that
ramp the rig can climb.
The pod defaults changed: DB pool default rises to 12 connections (persist holds 6 at the commit barrier).
Deploy notes
state_changesindex changes (see above; done on loadtest, pending on dev mainnet/testnet/prod together with the earliertransactions_ledger_created_at_idxrestore, which prod still needs).--live-persist-max-batch-size(envLIVE_PERSIST_MAX_BATCH_SIZE), default 1 (commit-per-ledger, unchanged behavior). Raising it costs memory: the rotation holds 2N+1 buffers, each retaining a merged ledger's maps. The loadtest rig sets 3, which is what its ingest container fits alongsideGOMEMLIMIT.histogram_quantile(0.99, sum by (le, phase) (rate(wallet_ingestion_phase_duration_seconds_bucket[15m])))and awallet_ingestion_persist_batch_sizep50 panel.