Leios (Musashi w27): Dijkstra serialization & BlockChainDataListener integration (ADR 0010/0011 + implementation)

[beanbot-gh]

Summary

Leios (Musashi) serialization and sync-surface integration — the layer above the transport-only mini-protocols from #167. This PR now carries the design (ADR 0010, ADR 0011) and the full implementation, targeting the live prototype-2026w27 Musashi network. Closes #170.

Goals delivered:

1. BlockSync / BlockRangeSync consumers (yaci-store) keep receiving ranking blocks through BlockChainDataListener.onBlock(era, block, txs) with zero listener API change — verified against live w27 Musashi blocks and byte-identical for mainnet eras.
2. New default listener methods onEndorserBlock(EndorserBlockEvent) / onLeiosVotes(LeiosVotesEvent) surface Leios data on the same listener and connection, opt-in.
3. The same application code works on mainnet (Leios inactive) and Musashi (Leios active), selected by network magic + handshake via LeiosConfig (AUTO | ENABLED | DISABLED).
4. yaci-store's mandatory change reduces to mapping the new Era.Dijkstra constant.

What's in this PR

Design records

• ADR 0010 — serialization + listener integration design (two review rounds)
• ADR 0011 — the w27 Dijkstra block restructure: why the prototype respin broke parsing, re-pinned wire shapes (blueprint 93276ab / prototype-2026w27), remediation (two review rounds)
• docs/leios/ — living docs: source-tracking guide (repos to watch, release cadence, re-pin checklist), pin matrix, long-term support plan

Core (feat(core) commit)

• Era.Dijkstra(8); structural pre/post-Babbage header routing; w27 12-item header slots (leios_certified, leios_announcement/nil) plus the pre-w27 trailing-extension form
• DijkstraBlockBodySerializer + DijkstraTransactionExtractor: w27 nested [header, block_body] shape, whole 3-element transactions with byte-exact body/witness/aux slices (tx-hash correctness; issue-Deserialization and serialization of redeemer is causing invalid cbor in some scenarios #37 datum/redeemer fix-up fed from per-tx slices), tag-258 invalid-tx sets, real LeiosCertificate with raw slices
• model/leios/*: EndorserBlock, EndorserBlockTx (dedicated ns8 era mapping), arity-tolerant LeiosVote (3- and 4-element), LeiosCertificate, LeiosAnnouncement — raw CBOR retained on every model

Helper (feat(helper) commit)

• LeiosSyncCoordinator: offer-gated notify→fetch→assemble per EB point, failure-latch resets, duplicate/null emit guards, refs-only timeout, payload release after emit
• LeiosConfig gating: AUTO attaches agents only for Musashi magic 164 on tip-following clients; BlockRangeSync/BlockFetcher never attach under AUTO; activation is magic-parameterized (≥ V15, non-app-layer)
• Env-gated MusashiBlockSyncE2EIT live probe; MusashiW27FixtureCapture + ./gradlew :helper:captureW27Fixtures manual fixture-capture tool for per-release re-pins

Verification

• Full :core:test + :helper:test green, including Conway/Babbage byte-identity regressions
• Live w27 fixtures captured from the public relay (core/src/test/resources/leios/w27/, blocks 27142–27146): the 12-item header and nested body shapes confirmed on chain; fixture-driven tests included
• Multi-round review: two independent multi-agent review passes + cross-review; all confirmed findings fixed (fail-loud required_signers, BREAK-tolerant Dijkstra dispatch, redeemer fix-up independence, Dijkstra-gated raw-sliced legacy certs, restored chainSyncAgent.reset(point))

Known gaps (deliberate, documented)

• Dijkstra tx-body key 14 is guards, not Conway's required_signers. Keyhash-variant guards parse but surface under requiredSigners; credential-variant guards fail loudly via onParsingError. New Dijkstra body keys (23 sub_transactions, 25, 26) are ignored. A guards/sub-transactions model is a follow-up, gated on live tx-bearing fixtures.
• Fixture matrix is partial: captured blocks are empty (idle chain window) — no announced/certified blocks, EB bodies, or votes yet. The capture tool is in place; a capture during an active window plus a long MusashiBlockSyncE2EIT run is the final validation gate.
• EB events flow near tip only (no EB range fetch in the prototype); historical ranges yield blocks with header announcements but no EB bodies.
• Musashi is a weekly-respin prototype — support is pinned per release tag (prototype-2026w27); see docs/leios/leios-musashi-source-tracking-guide.md for the re-pin procedure.

Follow-ups (tracked, not in this PR)

• DijkstraTransactionExtractor single-pass offset optimization (hot-path decode/copy reduction)
• Shared Leios attachment helper for BlockFetcher/N2NChainSyncFetcher (removes ~50 duplicated lines)
• Unify LeiosCborReader with WitnessUtil/TransactionBodyExtractor raw-slice machinery
• Guards / sub-transactions tx-body model for Dijkstra

Stacked on feat/leios_protocol_impl (#167) — the diff shows only this layer.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[edridudi]

Reviewed the full diff against feat/leios_protocol_impl (56 files, +6357/-59), reading the pre-PR versions of every modified file to check the mainnet path. I verified the mainnet claim directly and it holds: under LeiosConfig.Mode.AUTO, shouldAttach only returns true for Musashi magic 164, so no Leios agent is constructed, versionTableFor resolves to the same v4AndAbove(protocolMagic) call as before, and the two new BlockChainDataListener methods are default, so existing consumers stay source- and binary-compatible. The coordinator's own state handling is sound — every access to pendingByEbHash / announcementCborByEbHash is under synchronized (this), both maps are LRU-bounded, markEmitted runs inside the lock so double-emit is impossible, and the scheduler is a daemon thread closed from both fetchers' shutdown(). The isPostBabbageHeader restructure is also correct: index 8 is block_body_hash (a byte string) pre-Babbage and operational_cert (an array) post-Babbage, so the new structural check is strictly more robust than the old "last item is an unsigned int" heuristic.

Eight findings, all from reading the code rather than from the tests. Two I want to single out:

The CBOR cursor can livelock. LeiosCborReader.readDataItem() returns a null item and consumes zero bytes at EOF, because CborDecoder.decodeNext() returns null rather than throwing. Every while (!reader.nextIsBreak()) loop in this PR then spins forever, and two of them append to an ArrayList on each pass, so it is a hang plus OutOfMemoryError that the catch (Exception e) guards in the coordinator cannot contain. I reproduced this against cbor-0.9 with the single byte 0x9f. It is not reachable from a peer today only because LeiosCborUtil.toRawCbor() re-encodes an already fully decoded DataItem — but both affected serializers are public Serializer<T> implementations, so any caller handing them untrusted bytes hits it. One three-line guard in readDataItem closes it for every call site at once.

EndorserBlockTx.txHash hashes the whole transaction, not the body. A Cardano txid is blake2b-256 over the transaction body alone; this line hashes the entire [body, witness_set, auxiliary_data]. If EB tx-refs are txids, verifyTxHashMembership in MusashiBlockSyncE2EIT mismatches on every transaction, and any consumer joining transactions back onto txRefs by hash gets nothing. This is the one claim I could not settle from the repo, because the captured w27 fixtures come from an idle window and carry no EB bodies or tx lists — nothing exercises that line. It needs a fixture from an active window before merge, since txHash is precisely the field a downstream indexer keys on.

The rest are about failure modes rather than happy paths, and they cluster on one theme: this PR is strict where a stall is expensive and quiet where silence is expensive. The required_signers and invalid_transactions throws both abort the whole block parse and stall a tip-following consumer permanently on a value the Dijkstra CDDL permits, while the swallowed witness fix-up and the debug-level extraItems log let real data loss pass as a clean parse. Given the weekly respin cadence, I would invert both: degrade loudly through onParsingError, and warn on anything the wire sent that this version does not understand.

Also flagged: computedHash is taken over re-encoded rather than wire bytes (a latent hash correctness bug that only bites once the prototype emits a non-minimal encoding — the fix belongs in #167, but this PR is the first thing to hash those bytes); onEndorserBlock can be dispatched from the scheduler thread concurrently with onBlock on the netty thread, breaking the serialized-callback contract every existing consumer relies on; a zero-tx EB is emitted with txsComplete=false when its closure is in fact complete; and once a tx fetch is requested there is no response deadline at all, so a peer that goes quiet drops the EB with nothing above a debug line.

The ADRs are unusually good — 0011 in particular does the thing most respin-tracking docs don't, which is write down why the previous shape broke and what to re-pin. No objection to the overall design; the listener-compatible surface is the right call.

[edridudi]

readDataItem() can return a null item and consume zero bytes, which lets callers spin forever.

CborDecoder.decodeNext() returns null (not an exception) when the stream is already at EOF:

// co.nstant.in.cbor.CborDecoder#decodeNext
symbol = inputStream.read();
if (symbol == -1) {
    return null;
}

So at EOF this method computes consumed == 0, leaves position unchanged, and hands back new DecodedItem(null, new byte[0]). Every indefinite-length loop in this PR is written as while (!reader.nextIsBreak()) { reader.readDataItem(); ... }, and nextIsBreak() is hasRemaining() && peek() == BREAK — which is false at EOF. The loop therefore never terminates.

Reproduced against cbor-0.9 with the single byte 0x9f (indefinite array header, no items, no break):

len=-1
hasRemaining=false nextIsBreak=false
iter 1: dataItem=null rawLen=0 pos=1
iter 2: dataItem=null rawLen=0 pos=1
iter 3: dataItem=null rawLen=0 pos=1
SPIN CONFIRMED: 2000001 zero-progress iterations

The affected loops are EndorserBlockTxListSerializer#deserialize:31, EndorserBlockSerializer#readMapEntries:134 / #readOmapEntries:150, and DijkstraTransactionExtractor#consumeArrayExtras. Two of them also add(...) to an ArrayList on each pass, so the spin allocates until the heap is exhausted — a hang plus OutOfMemoryError, which the catch (Exception e) guards in LeiosSyncCoordinator#onBlock / #onBlockTxs will not contain.

On today's call graph this is not reachable from a peer: LeiosCborUtil.toRawCbor() re-encodes an already fully decoded DataItem, so the bytes handed to these serializers are always complete. But EndorserBlockSerializer and EndorserBlockTxListSerializer both implement the public Serializer<T> interface, so any caller passing untrusted or truncated bytes (yaci-store, the fixture-capture tool, a user decoding from hex) hits it. It also means one future change to a raw-slice fetch path silently converts a parse error into a livelock.

Suggest making the cursor's contract explicit here rather than fixing each loop:

int consumed = before - input.available();
if (dataItem == null || consumed == 0) {
    throw new IllegalArgumentException("unexpected end of CBOR data");
}
position += consumed;

[edridudi]

txHash looks like it hashes the whole transaction, not the transaction body — so it can never match an EB tx-ref hash.

txBytes is the content of the tag-24 byte string, i.e. the whole 3-element transaction [body, witness_set, auxiliary_data] (that is what the accompanying era index is for). A Cardano transaction id is blake2b-256 over the body alone — that is exactly what the existing TxUtil.calculateTxHash is fed in TransactionBodyExtractorTest, and what TransactionBodySerializer.deserializeDI(body, bodyBytes) computes from the body slice.

If EndorserBlockTxRef.txHash keys are txids (which is what EndorserBlockSerializer validates them as — 32-byte hashes), then verifyTxHashMembership in MusashiBlockSyncE2EIT:400 will count a mismatch for every fetched transaction and emit a warning per tx, and any consumer joining EndorserBlockEvent.transactions back onto endorserBlock.txRefs by hash gets an empty join.

This is the one claim in the PR I could not settle from the repo, because the captured w27 fixtures are all from an idle window and carry no EB bodies or tx lists — so no test exercises this line. Two ways to resolve it:

• If EB tx-refs are txids: slice the body out of txBytes (the tag-24 content is preserved verbatim through toRawCbor, so a body slice here is byte-exact) and hash that.
• If the Leios prototype really does reference transactions by whole-tx hash: worth a comment on this line saying so, because it contradicts every other txHash in the codebase.

Either way this needs a fixture from an active window before merge — it is the one field a downstream indexer will key on.

[edridudi]

computedHash is taken over re-encoded bytes, not the bytes that came off the wire.

The bytes reaching here originate from LeiosCborUtil.toRawCbor():

public static LeiosRawCbor toRawCbor(DataItem dataItem) {
    return LeiosRawCbor.of(CborSerializationUtil.serialize(dataItem, false));
}

That is a re-serialization of an already-decoded DataItem, so the encoding is whatever cbor-java's encoder emits, not what the peer sent. An EB hash only means something if it is computed over the peer's exact bytes — that is the invariant the rest of this PR works hard to preserve (byte-exact body/witness/aux slices, raw-sliced legacy certs). Here it is not preserved, and computedHash is the field a consumer would use to check an EB against leios_announcement.ebHash from the header.

In practice a cardano-node peer emits minimal-length integers and definite-length maps, so the round-trip probably reproduces the original bytes today and the hash happens to be right. That makes this a latent correctness bug rather than a live one: it breaks silently the first time the prototype emits a non-minimal or indefinite encoding, and the failure mode is a wrong-but-plausible 32-byte hash.

The same reasoning applies to .cbor(...) on the line above, and to LeiosCertificate.cbor / LeiosVote.cbor. Note the inner tag-24 tx bytes in EndorserBlockTxListSerializer are not affected — byte-string contents survive re-encoding verbatim; only outer framing is at risk.

The fix belongs in #167 rather than here (have MsgLeiosBlockSerializer carry the raw slice instead of toRawCbor), but this PR is the first consumer that hashes those bytes, so it is worth deciding now.

[edridudi]

Failing loud here means "halt the sync", not "skip the transaction".

I agree with the reasoning in the PR description — hex-encoding CBOR framing and shipping it downstream as a signer is worse than throwing. But consider where this throw lands. TransactionBodySerializer.deserializeDI is called from DijkstraBlockBodySerializer:46 inside the per-transaction loop, outside any try/catch. The exception propagates out of BlockSerializer.deserialize and out of the block-fetch handler. So the first Dijkstra block containing a credential-variant guards entry does not skip that transaction — it fails to parse the whole block, and every retry of that block fails identically. A tip-following consumer stops at that block permanently.

That is a chain-halting failure triggered by an on-chain value that the Dijkstra CDDL explicitly permits (key 14 is guards, and the credential variant is legal). The PR notes this under "Known gaps" as fail-loud-by-design, which is right for a value we cannot represent — but the blast radius is larger than "surfaces under requiredSigners".

Two options that keep the loudness without the halt:

• Route it through onParsingError (as the PR text says it does) and leave requiredSigners unset for that transaction, so the block still parses and the consumer sees an explicit error rather than a stall.
• Or model guards properly before Musashi carries tx-bearing blocks. Given the follow-up list already has "Guards / sub-transactions tx-body model for Dijkstra", this may just be a question of which lands first.

Worth confirming: is there a path where onParsingError actually observes this today? I traced the Dijkstra body path and did not find one.

[edridudi]

Range-checking invalid_transactions against the ranking block's tx count assumes Leios never indexes EB transactions.

Throwing here aborts the whole block parse, and the same "one bad block stalls the consumer forever" argument from TransactionBodySerializer applies. The Conway path in BlockSerializer does no such range check, so this is new strictness on the exact era whose body layout is still moving week to week.

The specific worry: in Leios the set of transactions a ranking block validates is not necessarily the set it carries — that is the whole point of endorser blocks. If a future respin lets invalid_transactions index into the combined RB + EB transaction sequence, transactionCount (which is only bodySlice.transactions().size()) becomes the wrong denominator and every certified block throws.

Given that this PR cannot yet verify the assumption (no certified blocks in the captured fixtures), I would rather this log a warning and drop the out-of-range index than throw. A bogus index in invalidTransactions is recoverable downstream; a permanently unparseable block is not.

[edridudi]

Catching here silently ships datums and redeemers without their raw CBOR — the legacy path propagates instead.

The Conway/Babbage path calls handleWitnessDatumRedeemer, which is @SneakyThrows — a failure there surfaces. Here the same fix-up is wrapped in catch (Exception e) { log.error(...) } and parsing continues, so blockBuilder.transactionWitness(witnessesSet) is populated with Datum objects whose cbor is null and whose hash was never recomputed from the raw slice (issue #37's fix-up). A downstream store persists nulls and the block still looks successfully parsed.

Two eras, two behaviors for the same failure, and the quieter one is on the era we understand least. Either propagate (consistent with legacy) or, if the intent is that Dijkstra should be resilient here, mark the block so the consumer can tell the difference — an onParsingError callback, or at minimum not leaving half-fixed Datum instances in the result.

Related, on line 35–38: extraItems is only reported when log.isDebugEnabled(). Silently dropping unrecognized block_body items at debug level is how a wire-shape change becomes a mysterious data gap three weeks later. Given the weekly respin cadence, a one-shot log.warn seems more appropriate for a value that means "the node sent us something this version does not understand".

[edridudi]

onEndorserBlock can be delivered on the scheduler thread concurrently with onBlock on the netty thread.

Every other BlockChainDataListener callback — onBlock, onBlockTxs, onRollback, and the Leios callbacks reached via onBlockAnnouncement / onBlockTxs — is invoked from the single netty event-loop thread of the shared N2N connection, so today's consumers are entitled to assume callbacks are serialized. emitRefsOnlyOnTimeout breaks that: it runs on Executors.newSingleThreadScheduledExecutor (line 81) and reaches this dispatch at line 271 → 296.

Concretely: an EB is fetched, the peer never offers its transactions, txsOfferWaitMillis elapses, and the scheduler thread calls onEndorserBlock while the netty thread is inside onBlock for the next ranking block. A consumer batching into a non-thread-safe accumulator or a JDBC session — yaci-store's block processor, for one — now has two threads in it.

The coordinator's own state is fine (all map access is under synchronized (this), and markEmitted is inside the lock, so no double-emit). It is purely the handoff to user code that escapes.

Cheapest correct fix is to have the timeout task not dispatch, but instead hand the emit back onto the connection's event loop. If that plumbing isn't available, then BlockChainDataListener.onEndorserBlock's javadoc must state that it may be called concurrently with the other callbacks — right now it says nothing, and the default no-op implementation invites people to assume the ambient contract holds.

[edridudi]

An endorser block that references zero transactions is emitted with txsComplete=false, but its closure is complete.

When txCount == 0 this branch is taken and builds the event with txsComplete=false. An EB with no tx-refs has nothing left to fetch — the empty closure is the whole closure. A consumer that treats txsComplete=false as "transactions are missing, do not trust this event" will record a false gap, or re-request an EB that can never become more complete.

The flag conflates three genuinely different states, and the zero-tx case lands in the wrong one:

if (!leiosConfig.isFetchTxs() || txCount == 0 || txsToFetch <= 0) {
    event = buildEvent(pending, false);

• !isFetchTxs() — user opted out, transactions unknown. false is right.
• txsToFetch <= 0 (i.e. maxTxsPerEndorserBlock <= 0) — capped out, transactions withheld. false is right.
• txCount == 0 — nothing to fetch, closure resolved. Should be true.

Suggest buildEvent(pending, txCount == 0) here, or splitting the zero-tx case into its own branch so the intent is legible.

[edridudi]

The refs-only timeout is the only timeout — once a fetch is requested, nothing guards the response.

scheduleRefsOnlyTimeout is reached only from the !pending.txsOffered && pending.timeout == null branch. As soon as requestBlockTxs succeeds (txFetchRequested = true), the pending entry has no deadline at all. LeiosFetchAgent only arms a write timeout (scheduleWriteTimeout), not a read timeout, so if the peer accepts the request and then simply never sends MsgLeiosBlockTxs, onFetchError never fires and this EB is never emitted — it just sits in pendingByEbHash until LRU eviction pushes it out, silently.

The block path has the same hole: after requestBlock, if the reply never arrives, pending.endorserBlock stays null and no timeout is ever scheduled for it either.

This needs a misbehaving or wedged peer to bite, and the Leios stream is observational, so nothing downstream corrupts — the EB event is just dropped with no log line above debug. Still, "peer went quiet" is the normal failure mode on a weekly-respin prototype network, and right now it is indistinguishable from "no EBs were produced". A response deadline that fires emitIfEndorserBlockFetched(point, false) would make the difference observable.