experiment: combine batch proof verification (#1872), tx revalidation cache (#744) and live intra-block ledger states (#2050)

[chrispalaskas]

Overview

Combined test branch integrating three open performance PRs so they can be load-tested
together (perfnet A/B runs), plus two throughput-limit changes needed to actually reach the
regime the three PRs are meant to improve:

• experiment: batch verification #1872 — batch proof verification (ozgb-batch-proof-verification, head c9d9e55, incl. the 2026-08-17 ledger update)
• feat: replace soft tx cache with revalidation-based validation cache #744 — revalidation-based validation cache (skowron/tx-revalidation, head 3dd295e, incl. Oscar's tblock + stale-check fixes: 202ba1d4, d1d6cc9f, 624b3f55)
• feat(node): keep intra-block ledger states live instead of persisting them #2050 — keep intra-block ledger states live instead of persisting them (ozgb-ledger-intermediate-states-less-persists, head 1da3723e) — replaces fix: stop persisting intermediate ledger states #1443, which was on this branch until f9c315c8
• block length 1 MiB → 5 MiB and the flat per-transaction weight add-on zeroed (this branch only, not from an upstream PR)

Not intended to merge into main as-is — it exists to measure the changes in combination.

How the branch was assembled

Base is the #1872 head (freshest ledger pin). The others were integrated on top.

Merge of #744 (commit 2c5df2b) — semantic merge, not mechanical

Both PRs rewrote the same caching layer in ledger/src/versions/common/mod.rs: #744 replaces the STRICT/SOFT caches with a single revalidation-based TX_VALIDATION_CACHE; #1872's batch verification warmed STRICT/SOFT and recorded inline-crypto timings. Resolution decisions:

• feat: replace soft tx cache with revalidation-based validation cache #744's single revalidation cache is the base architecture. The STRICT/SOFT caches (and experiment: batch verification #1872's soft-cache warming) are gone; the batch-verification ingress (batch_verify_transactions → warm_verified_tx) now warms the revalidation cache with the batch's reference state and effective well_formed tblock. A later lookup at the same state strict-hits; a later state revalidates the cheap delta — which is exactly feat: replace soft tx cache with revalidation-based validation cache #744's stale-check semantics (a non-strict hit always re-runs the guaranteed dry-run).
• PROOF_VERIFICATION_CACHE re-keyed from Twox128(runtime_version ++ tx_bytes) to (runtime_version, ledger tx.hash()) — the same identity as the validation-cache key — so the cache-miss path (verify_transaction) can consult it without threading serialized bytes through feat: replace soft tx cache with revalidation-based validation cache #744's call chain. It is read there to defer already-batch-verified ZK crypto (Some(true) → defer, Some(false) → reject, None → verify inline + log error).
• Both PRs' metrics preserved: get_verified_transaction now returns (VerifiedTransaction, TxValidationCacheOutcome, Option<Duration>) — feat: replace soft tx cache with revalidation-based validation cache #744's hit/revalidation/miss counters and experiment: batch verification #1872's mode="inline" proof-verify baseline are both recorded on the apply and pre-dispatch paths.
• Pool provides-tag kept as the Twox128 key. feat: replace soft tx cache with revalidation-based validation cache #744 changed validate_transaction's returned hash (the and_provides tag) to the ledger tx.hash(); experiment: batch verification #1872's node-side batch_chain_api builds the Twox128 tag natively and must stay byte-for-byte identical to the runtime's. validate_transaction therefore still returns the restored tx_validation_cache_key (Twox128, zero-extended), while the validation cache itself uses feat: replace soft tx cache with revalidation-based validation cache #744's key.
• SOFT_TX_VALIDATION_CACHE invalidation on pre-dispatch is gone along with the cache itself, matching feat: replace soft tx cache with revalidation-based validation cache #744's "keep validation cache entries across rejections" fix.

Replacing #1443 with #2050 (commits f9c315c8, 6bd32a63)

#2050 supersedes #1443 with a materially smaller change, so the swap was done as a clean
revert-then-apply rather than by patching #1443 into #2050 in place:

• f9c315c8 reverts the fix: stop persisting intermediate ledger states #1443 application (c6590ef) wholesale — 24 files, back to pure experiment: batch verification #1872+feat: replace soft tx cache with revalidation-based validation cache #744. pallets/midnight/Cargo.toml is deliberately not reverted: feat(node): keep intra-block ledger states live instead of persisting them #2050 carries the identical test-utils feature block and [[test]] stanza for the same persist_refcount integration test, and ca3e1ab's optional-dependency promotion for feat: replace soft tx cache with revalidation-based validation cache #744's mock additions is still required.
• 6bd32a63 applies feat(node): keep intra-block ledger states live instead of persisting them #2050 as a squashed three-way patch of 8d3e0cf4..1da3723e (same criss-cross-history reason as the original fix: stop persisting intermediate ledger states #1443 application — git merge is not usable here).

The swap is a large simplification. #1443 changed runtime storage and the state-key type
(LedgerStateKey), shipped migration v3, and required a metadata rebuild. #2050 needs none
of that: intra-block intermediates are held live in a refcounted keep-alive cache inside the
ledger crate instead of being persisted. migrations/v3.rs, ledger-state-key-type.md, the
primitives/midnight and runtime/src/lib.rs changes and the metadata delta are all gone.

Adaptations to the combined branch:

• Cargo.lock regenerated rather than taken from feat(node): keep intra-block ledger states live instead of persisting them #2050: this branch pins midnight-ledger by rev (31529634, experiment: batch verification #1872's 2026-08-17 update), not by the ledger-9.1.0.0-rc.4 tag main uses. The only real change is moka 0.11.3 → 0.12.15, which feat(node): keep intra-block ledger states live instead of persisting them #2050 needs for Cache::and_compute_with.
• ledger/src/versions/common/mod.rs — one adjacency conflict: experiment: batch verification #1872's proof-verification cache helpers (insert_proof_result / get_proof_result / proof_verification_cache_size) sit exactly where feat(node): keep intra-block ledger states live instead of persisting them #2050 inserts its keep-alive block. Both kept; they touch different caches and do not interact.
• primitives/ledger/src/lib.rs — same shape: experiment: batch verification #1872's proof_verify_duration / proof_verify_txs and feat(node): keep intra-block ledger states live instead of persisting them #2050's ledger_state_cache_size gauge both kept.
• feat(node): keep intra-block ledger states live instead of persisting them #2050 does not touch host_api/ledger_9.rs, common/types.rs, pallets/midnight-system, the migrations or the metadata at all, so the revert simply restores those to experiment: batch verification #1872+feat: replace soft tx cache with revalidation-based validation cache #744 state.

Block length and the flat per-transaction weight (commit 0871f458)

Two independent limits closed a block well before the node ran out of execution capacity,
which would have masked what the three PRs above are measuring:

• frame_system::BlockLength 1 MiB → 5 MiB. It was max_with_normal_ratio(1 MiB, 75%), i.e. ~786 KiB usable for normal dispatches. Midnight transactions are large relative to a typical Substrate extrinsic, so blocks were hitting the length limit, not the weight limit.
• ConfigurableTransactionSizeWeight now defaults to Weight::zero(). It shared DefaultWeight with ConfigurableOnInitializeWeight and ConfigurableOnRuntimeUpgradeWeight, so its default was EXTRA_WEIGHT_TX_SIZE — 20 ms ref-time, ~1% of a 2 s block — added flat to every transaction on top of its real gas-metered cost, and it was the dominant term. It now gets its own #[pallet::type_value].

DefaultWeight and the get_tx_weight fallback for a transaction whose cost cannot be
metered are deliberately unchanged and still use EXTRA_WEIGHT_TX_SIZE. This changes exactly
the flat per-transaction add-on and nothing else; the value stays settable at runtime via the
root-only set_tx_size_weight extrinsic, so a chain that already zeroed it by extrinsic sees
no change from the new default.

spec_version bumped 002_001_000 → 002_001_001.

🗹 TODO before merging

• /bot rebuild-metadata — BlockLength is a frame_system constant and the weight default is a storage-entry default, so both are metadata-visible and the checked-in snapshots are stale. (Note this is newly needed for the throughput commit; feat(node): keep intra-block ledger states live instead of persisting them #2050 itself needs no metadata change, unlike fix: stop persisting intermediate ledger states #1443.)
• Decide whether this branch ever targets main or stays a perf-test vehicle
• Full node image build (earthly +node-image) — local WASM build not verified (environment lacked clang)
• Perfnet A/B results

📌 Submission Checklist

• All commits are signed off (git commit -s) for the DCO
• Changes are backward-compatible (or flagged if breaking)
• Pull request description explains why the change is needed
• Self-reviewed the diff
• I have included a change file, or skipped for this reason: change files from the source PRs are carried in by the merge; changes/runtime/changed/block-length-5mib-zero-tx-size-weight.md covers the two throughput changes
• If the changes introduce a new feature, I have bumped the node minor version
• Update documentation (if relevant)
• Updated AGENTS.md if build commands, architecture, or workflows changed
• No new todos introduced

🧪 Testing Evidence

All native, with SKIP_WASM_BUILD=1 (WASM runtime build not run locally — clang unavailable in the assembly environment):

• cargo check --workspace --all-targets — clean
• cargo clippy --workspace --all-targets — clean
• cargo fmt --all -- --check — clean (22f2ea32 fixes two pre-existing diffs in experiment: batch verification #1872's code)
• cargo test -p midnight-node-ledger — 81 passed
• cargo test -p pallet-midnight — 29 passed / 2 ignored (incl. feat: replace soft tx cache with revalidation-based validation cache #744's revalidation cache-metrics tests, and sets_extra_transaction_size_weight updated for the zero default)
• cargo test -p pallet-midnight --features test-utils --test persist_refcount — 1 passed (feat(node): keep intra-block ledger states live instead of persisting them #2050's keep_alive_invariants, the real functional proof that the keep-alive caches behave on the combined branch)
• cargo test -p midnight-node batch — 7 passed (incl. the provides-tag mirror assertion in batch_chain_api)

The ledger/pallet test counts are lower than the previous #1443-based revision (82 / 31) because #2050 drops #1443's LedgerStateKey and migration-v3 tests along with the code they covered.

• Additional tests are provided (if possible)

🔱 Fork Strategy

• Node Runtime Update
• Node Client Update
• Other:
• N/A

#1872, #744 and #2050 are all client-side (#2050 explicitly has no storage-layout, host-ABI
or runtime change — that is the main way it differs from #1443). The block-length and
per-transaction-weight changes are a runtime update and need a setCode. As with prior
perfnet work: host-function changes cannot be A/B'd on a single chain.

Links

• experiment: batch verification #1872
• feat: replace soft tx cache with revalidation-based validation cache #744
• feat(node): keep intra-block ledger states live instead of persisting them #2050 (replaces fix: stop persisting intermediate ledger states #1443)

🤖 Generated with Claude Code