Skip to content

fix(chunk-ingest): make optimistic chunk confirmation sticky per data_root#815

Open
vilenarios wants to merge 4 commits into
developfrom
fix/chunk-ingest-sticky-confirmation
Open

fix(chunk-ingest): make optimistic chunk confirmation sticky per data_root#815
vilenarios wants to merge 4 commits into
developfrom
fix/chunk-ingest-sticky-confirmation

Conversation

@vilenarios

Copy link
Copy Markdown
Contributor

Problem

Bundles our own AR.IO Bundler uploads are "findable-late" and 404 on /raw, and the indexer's unbundler fails for a sustained window — chronically for large bundles. On gw1 (turbo-gateway.com, where the bundler posts at scale) the failing 2 GB bundle zwpirOGSpS_AEs6DxxV68lXgk8_4ohTrvEVH_dCtrTc had only ~12% of its chunks on disk, gappy, with relative_offset 0 missing. Retrieval range-streams from offset 0 → immediate miss → peers (which don't yet have the fresh bundle) → fail. The 315 "failures" were ~9 such large bundles retried every 20 min.

This is a retention/confirmation defect, not a retrieval-addressing one. (A separate investigation ruled out the retrieval path: a hermetic repro showed the gateway already serves ingested chunks locally by (data_root, relative_offset) when they are present.)

Root cause

confirmChunkPlacements is a one-shot UPDATE fired by the TX_INDEXED event (system.ts): it only sets confirmed_at on placements that exist at that instant. A multi-GB bundle streams its thousands of chunks in over a window far longer than that single event, so most arrive after it, never get confirmed, and are TTL-evicted by the ingest GC. Metrics on gw1: chunk_ingest_evicted_total{reason="ttl"}=4000 vs chunk_ingest_confirmed_total=543.

Fix

Make confirmation sticky per data_root (statement + one small table; no retrieval-path change, no hot-path tx lookup):

  • saveChunkPlacement now inherits confirmed_at when the data_root is already confirmed, so every chunk ingested after the confirm event self-confirms at ingest.
  • New confirmed_data_roots marker table, populated by confirmChunkPlacementsgated on EXISTS in chunk_placements so it stays bounded by ingested bundles rather than the whole tx index. The marker is (a) the primary inheritance source and (b) excluded by the GC TTL sweep, so a confirmed bundle is never partially evicted regardless of per-row state.

Net effect: once the one-shot TX_INDEXED confirms the first batch of a bundle, the entire bundle (present + all later-arriving chunks) is confirmed and retained → servable and unbundlable without waiting for network re-propagation.

Correctness / scope

  • Bounded table: the EXISTS gate means only data_roots we actually ingested chunks for are marked.
  • No new leak class: confirmation still means "the tx was indexed"; confirmed chunks were already retained as the metadata index. This fix extends that to the whole bundle, which is the intent.
  • Retrieval unchanged.
  • Residual (documented): a chunk that arrives and expires before the very first confirm event still evicts — only reachable if the gateway's block-import lags beyond the ingest TTL (a sync pathology), not the observed failure.

Tests

src/database/chunk-placements.test.ts:

  • inherits confirmed_at from a confirmed sibling at ingest;
  • regression: confirm a batch, then ingest more → auto-confirmed and not TTL-selected;
  • marker shields a data_root from eviction even when a row is pending;
  • marker EXISTS-gate: an indexed tx we never ingested chunks for is not marked and grants no phantom inheritance;
  • explicit confirmed_at still wins.

All green (chunk-placements + ingest + GC + sql-loader + standalone-sqlite = 117/117); lint clean; migration up/down verified on a scratch DB; test/chunks-schema.sql regenerated.

Deploy notes

One additive migration (confirmed_data_roots), no data rewrite; the SQL statement change takes effect on rebuild. On-by-default.

🤖 Generated with Claude Code

…_root

Optimistically-ingested chunks were TTL-evicted before a large bundle could
be served or unbundled, leaving a gappy, unservable set (e.g. relative_offset
0 missing -> range streaming from 0 misses -> falls to peers, which don't yet
have the fresh bundle -> 404 / "Unbundling failed" until it re-propagates).

Root cause: confirmation is a one-shot UPDATE fired by TX_INDEXED that only
touches placements present at that instant. A multi-GB bundle streams its
thousands of chunks in over a window far longer than that single event, so
most arrive AFTER it, never get confirmed, and hit the ingest TTL. On gw1 the
failing 2 GB bundle had only ~12% of chunks on disk with
chunk_ingest_evicted_total{ttl}=4000 vs chunk_ingest_confirmed_total=543.

Fix (retention/confirmation, not retrieval):
- saveChunkPlacement now inherits confirmed_at when the data_root is already
  confirmed, so every chunk ingested after the confirm event self-confirms.
- New confirmed_data_roots marker table, populated by confirmChunkPlacements
  (gated on EXISTS in chunk_placements so it stays bounded by ingested
  bundles). It (a) is the primary inheritance source and (b) is excluded by
  the GC TTL sweep, so a confirmed bundle is never partially evicted.

Statement + one small table only; no change to the retrieval path. Adds
regression tests (confirm a batch, then ingest more -> auto-confirmed and not
TTL-selected) and marker tests. Updates docs/chunk-ingest-cache.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BFuubmcaMUDBGvRVRyp9BH
@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.95062% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.82%. Comparing base (feefd25) to head (279b792).

Files with missing lines Patch % Lines
src/database/standalone-sqlite.ts 76.19% 10 Missing ⚠️
src/workers/chunk-ingest-gc.ts 83.33% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #815      +/-   ##
===========================================
- Coverage    78.82%   78.82%   -0.01%     
===========================================
  Files          132      132              
  Lines        50172    50252      +80     
  Branches      3787     3790       +3     
===========================================
+ Hits         39548    39611      +63     
- Misses       10575    10592      +17     
  Partials        49       49              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: f7d44e00-bf7d-4f3e-b63b-5de084104734

📥 Commits

Reviewing files that changed from the base of the PR and between 52882a9 and 279b792.

📒 Files selected for processing (5)
  • migrations/2026.07.04T12.00.00.chunks.add-confirmed-data-roots.sql
  • migrations/2026.07.04T12.00.01.chunks.add-confirmed-sibling-index.sql
  • migrations/down/2026.07.04T12.00.01.chunks.add-confirmed-sibling-index.sql
  • src/workers/chunk-ingest-gc.test.ts
  • test/chunks-schema.sql
✅ Files skipped from review due to trivial changes (3)
  • migrations/down/2026.07.04T12.00.01.chunks.add-confirmed-sibling-index.sql
  • migrations/2026.07.04T12.00.01.chunks.add-confirmed-sibling-index.sql
  • migrations/2026.07.04T12.00.00.chunks.add-confirmed-data-roots.sql
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/workers/chunk-ingest-gc.test.ts

📝 Walkthrough

Walkthrough

Adds confirmed-data-root persistence and retention plumbing so chunk placements can inherit confirmation state, TTL GC skips confirmed roots, and confirmed markers are pruned on a separate retention window.

Changes

Sticky Confirmation Feature

Layer / File(s) Summary
Table schema and migrations
migrations/2026.07.04T12.00.00.chunks.add-confirmed-data-roots.sql, migrations/2026.07.04T12.00.01.chunks.add-confirmed-sibling-index.sql, migrations/down/..., test/chunks-schema.sql
Adds confirmed_data_roots(data_root, confirmed_at) and the sibling-confirmation index in migrations and the test schema.
Confirmation inheritance and GC exclusion
src/database/sql/chunks/cache.sql
saveChunkPlacement now inherits confirmed_at from explicit input, confirmed markers, or sibling rows; confirmed markers are inserted first-wins; expired-unconfirmed selection skips confirmed roots.
Worker confirmation and prune plumbing
src/database/standalone-sqlite.ts
confirmChunkPlacements writes confirmed-root markers and the worker path exposes pruneConfirmedDataRoots and countConfirmedDataRoots.
Retention config and sweep
src/config.ts, src/workers/chunk-ingest-gc.ts, src/types.d.ts, docker-compose.yaml, docs/envs.md, docs/chunk-ingest-cache.md, docs/glossary.md, src/metrics.ts
Adds retention configuration and GC sweep pruning for confirmed markers, plus docs and metrics for the new env var and sticky-confirmation flow.
Tests
src/database/chunk-placements.test.ts, src/workers/chunk-ingest-gc.test.ts
Adds coverage for inheritance, marker shielding, marker pruning, and GC worker prune calls.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

  • ar-io/ar-io-node#783: Introduces the chunk-placement confirmation and GC path that this PR extends with confirmed-root markers and pruning.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states the main fix: sticky optimistic chunk confirmation per data_root.
Description check ✅ Passed The description matches the changes and explains the retention/confirmation fix, tests, and migration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/chunk-ingest-sticky-confirmation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
src/database/standalone-sqlite.ts (2)

1036-1052: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add a TSDoc block for the sticky-confirmation contract.

Other non-trivial methods in this file (getTransactionAttributes, getBlockByWeaveOffset) use TSDoc blocks; this method's new dual-effect behavior (confirms rows + records the marker) would benefit from one too.

+  /**
+   * Confirms pending chunk placements for `dataRoot` (the one-shot update
+   * triggered by TX_INDEXED) and records a sticky `confirmed_data_roots`
+   * marker so chunks ingested afterward inherit confirmation at insert time
+   * and are retained past the TTL sweep.
+   */
   confirmChunkPlacements(dataRoot: string, confirmedAt: number): number[] {

As per coding guidelines, "Add or improve TSDoc comments on code you touch."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/database/standalone-sqlite.ts` around lines 1036 - 1052, Add a TSDoc
block for confirmChunkPlacements describing the sticky-confirmation contract: it
both confirms matching chunk_placements rows and records the data_root marker
via markDataRootConfirmed so future ingests stay confirmed and retained. Keep
the comment near the method signature in standalone-sqlite.ts, matching the
style used by getTransactionAttributes and getBlockByWeaveOffset.

Source: Coding guidelines


1036-1052: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Wrap confirmChunkPlacements + markDataRootConfirmed in a transaction.

The two writes aren't atomic; a crash between them leaves confirmed rows without the sticky marker, weakening TTL protection for any straggler chunk ingested right after restart. This file already wraps related multi-statement writes (e.g. insertTxFn, insertDataItemFn) in db.transaction(...).

🔒 Proposed fix
   confirmChunkPlacements(dataRoot: string, confirmedAt: number): number[] {
     const dataRootBuf = fromB64Url(dataRoot);
-    const rows = this.stmts.chunks.confirmChunkPlacements.all({
-      data_root: dataRootBuf,
-      confirmed_at: confirmedAt,
-    });
-    this.stmts.chunks.markDataRootConfirmed.run({
-      data_root: dataRootBuf,
-      confirmed_at: confirmedAt,
-    });
+    const rows = this.dbs.chunks.transaction(() => {
+      const confirmed = this.stmts.chunks.confirmChunkPlacements.all({
+        data_root: dataRootBuf,
+        confirmed_at: confirmedAt,
+      });
+      this.stmts.chunks.markDataRootConfirmed.run({
+        data_root: dataRootBuf,
+        confirmed_at: confirmedAt,
+      });
+      return confirmed;
+    })();
     return rows.map((row: any) => row.cached_at as number);
   }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/database/standalone-sqlite.ts` around lines 1036 - 1052, The
confirmChunkPlacements method currently performs two related writes,
confirmChunkPlacements.all and markDataRootConfirmed.run, without atomicity.
Wrap both operations in a db.transaction(...) in standalone-sqlite.ts so the
confirmed rows and sticky data_root marker are committed together; use the
existing transaction pattern from insertTxFn and insertDataItemFn, and keep the
same dataRootBuf/confirmedAt flow inside the transaction callback.
src/database/chunk-placements.test.ts (1)

168-307: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Solid regression coverage for the sticky-confirmation behavior, including the exact large-bundle TTL-eviction scenario this PR fixes.

One small gap: no test exercises markDataRootConfirmed's first-confirmation-wins (ON CONFLICT DO NOTHING) semantics — e.g. calling confirmChunkPlacements twice for the same dataRoot with different confirmedAt values and asserting the marker retains the first timestamp. Optional given existing coverage is already strong.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/database/chunk-placements.test.ts` around lines 168 - 307, Add a focused
test for first-confirmation-wins behavior on the data_root marker: in the
`confirmed_data_roots marker` suite, call `confirmChunkPlacements` twice for the
same `dataRoot` with different timestamps and assert the stored confirmation
remains the first value. Use the existing `confirmChunkPlacements` and
`getChunkPlacement` helpers to verify `markDataRootConfirmed`’s `ON CONFLICT DO
NOTHING` semantics, alongside the current sticky-confirmation coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/database/sql/chunks/cache.sql`:
- Around line 19-33: The sibling-confirmation fallback in the chunk placement
lookup is still scanning all rows for a data_root, causing saveChunkPlacement to
degrade to O(n) during pre-confirmation inserts. Update the SQL around the
COALESCE in cache.sql to support the sibling lookup with a partial index on
chunk_placements keyed by data_root and filtered on confirmed_at IS NOT NULL, so
the SELECT sibling.confirmed_at query can use it efficiently.

---

Nitpick comments:
In `@src/database/chunk-placements.test.ts`:
- Around line 168-307: Add a focused test for first-confirmation-wins behavior
on the data_root marker: in the `confirmed_data_roots marker` suite, call
`confirmChunkPlacements` twice for the same `dataRoot` with different timestamps
and assert the stored confirmation remains the first value. Use the existing
`confirmChunkPlacements` and `getChunkPlacement` helpers to verify
`markDataRootConfirmed`’s `ON CONFLICT DO NOTHING` semantics, alongside the
current sticky-confirmation coverage.

In `@src/database/standalone-sqlite.ts`:
- Around line 1036-1052: Add a TSDoc block for confirmChunkPlacements describing
the sticky-confirmation contract: it both confirms matching chunk_placements
rows and records the data_root marker via markDataRootConfirmed so future
ingests stay confirmed and retained. Keep the comment near the method signature
in standalone-sqlite.ts, matching the style used by getTransactionAttributes and
getBlockByWeaveOffset.
- Around line 1036-1052: The confirmChunkPlacements method currently performs
two related writes, confirmChunkPlacements.all and markDataRootConfirmed.run,
without atomicity. Wrap both operations in a db.transaction(...) in
standalone-sqlite.ts so the confirmed rows and sticky data_root marker are
committed together; use the existing transaction pattern from insertTxFn and
insertDataItemFn, and keep the same dataRootBuf/confirmedAt flow inside the
transaction callback.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 97f00106-e393-46ad-8478-6c4552f0be42

📥 Commits

Reviewing files that changed from the base of the PR and between feefd25 and f58c1c5.

📒 Files selected for processing (7)
  • docs/chunk-ingest-cache.md
  • migrations/2026.07.04T12.00.00.chunks.add-confirmed-data-roots.sql
  • migrations/down/2026.07.04T12.00.00.chunks.add-confirmed-data-roots.sql
  • src/database/chunk-placements.test.ts
  • src/database/sql/chunks/cache.sql
  • src/database/standalone-sqlite.ts
  • test/chunks-schema.sql

Comment thread src/database/sql/chunks/cache.sql
…+ prune

End-to-end testing (256 MB bundle, allowlisted bundler) exposed a hole in the
sticky-confirmation marker: it was gated on EXISTS in chunk_placements, on the
assumption that a bundle's chunks are ingested before its tx confirms. In
practice the opposite ordering dominates. The bundler's chunk-broadcast
TX-confirmation gate holds seeding until the tx confirms network-wide, and the
gateway imports that same block first — so the TX_INDEXED confirm event fires
~2s BEFORE the first chunk is seeded. With no chunk present, the EXISTS gate
skipped the marker, every later-seeded chunk stayed pending, and all 1025 chunks
of the test bundle TTL-evicted (offset 0 lost) — reproducing the original
gappy-bundle failure the fix was meant to prevent.

Fix: write the confirmed_data_roots marker unconditionally at confirm time so
later-seeded chunks inherit it regardless of ordering. Bound the table with an
age-based prune in the GC sweep (CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS,
default 1h) — a marker only needs to outlive the confirm->seed gap (seconds),
after which ingested chunks carry their own confirmed_at.

- cache.sql: drop EXISTS gate from markDataRootConfirmed; add pruneConfirmedDataRoots
- standalone-sqlite: pruneConfirmedDataRoots (worker + queue wrapper + handler)
- chunk-ingest-gc: prune stale markers each sweep
- config: CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS
- tests: confirm-before-chunks inheritance (the exact E2E scenario) + prune
- docs: chunk-ingest-cache.md, envs.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BFuubmcaMUDBGvRVRyp9BH
@vilenarios

Copy link
Copy Markdown
Contributor Author

End-to-end validation on a real bundler → gateway pipeline

Deployed this branch to a live gateway (chunk ingest on, allowlisted bundler) and pushed 256 MB / 1025-chunk bundles through it. Used a deliberately short ingest TTL (180 s, GC every 15 s) so unconfirmed chunks actually evict within the test window — otherwise the default 2–24 h TTL makes any short test pass trivially.

Run 1 caught a real bug (now fixed in 815b701)

The confirming TX_INDEXED event fired ~2 s before the bundler seeded the first chunk. This is the dominant ordering, not an edge case: the bundler's chunk-broadcast TX-confirmation gate holds seeding until the tx confirms network-wide, and the gateway imports that same block first. The marker's original EXISTS in chunk_placements gate found zero chunks → no marker set → all 1025 chunks stayed pending and TTL-evicted (offset 0 lost) — reproducing the exact gappy-bundle failure this PR targets.

Fix: record the confirmed_data_roots marker unconditionally at confirm time (so later-seeded chunks inherit it regardless of ordering), bounded by an age-based prune in the GC sweep (CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS, default 1 h).

Run 2 (with the fix) — clean pass

metric result
chunks on disk 1025 / 1025 (complete)
relative_offset 0 present yes
confirmed / pending 1025 / 0
late-arrivals confirmed (cached after confirm) 916
chunk_ingest_evicted_total{ttl} 0
duration retained 36 min (~12× the 180 s TTL)
marker present, table bounded (~517 rows, pruned)
unbundle succeeded from local cache
data item GraphQL-visible, bundledIn set

The whole bundle was retained and unbundled with zero network re-fetch, sustained far past the TTL that would have evicted it on the old code.

Caveat (honest): Run 2's timing happened to land the first chunks ~4 s before confirm, so it exercised sibling inheritance + the marker with a few chunks present; the confirm-before-any-chunk path (Run 1's timing) is covered by the added unit test and by Run 1's demonstrated failure, rather than a second live run at that exact race.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/workers/chunk-ingest-gc.ts (1)

151-157: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider surfacing the pruned-marker count for observability.

The return value of pruneConfirmedDataRoots is discarded. Given this table exists specifically to bridge a race window that was only caught via live E2E testing (per the PR discussion), exposing a counter/gauge (similar to chunkIngestEvictedCounter) would help detect if CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS is misconfigured too low in production.

♻️ Proposed addition
-      await this.chunkPlacementIndex.pruneConfirmedDataRoots(
-        now - this.confirmedRootRetentionSeconds,
-      );
+      const prunedMarkers = await this.chunkPlacementIndex.pruneConfirmedDataRoots(
+        now - this.confirmedRootRetentionSeconds,
+      );
+      if (prunedMarkers > 0) {
+        metrics.chunkIngestPrunedConfirmedRootsCounter?.inc(prunedMarkers);
+      }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/workers/chunk-ingest-gc.ts` around lines 151 - 157, The
confirmed-data-root pruning result is currently ignored in chunk-ingest GC, so
add observability for the number of pruned markers. In chunk-ingest-gc.ts,
capture the return value from ChunkPlacementIndex.pruneConfirmedDataRoots inside
the GC flow and record it with a dedicated metric (counter or gauge, similar to
chunkIngestEvictedCounter) so misconfiguration of
CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS is visible. Use the existing chunk
ingest GC path and pruneConfirmedDataRoots symbol to wire the metric in without
changing the pruning behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/workers/chunk-ingest-gc.test.ts`:
- Around line 77-80: The test for chunk-ingest GC is missing an assertion on the
cutoff recorded by the `pruneConfirmedDataRoots` mock. Update the
`rec.prunedBefore` expectations in `chunk-ingest-gc.test.ts` to verify the value
passed into `pruneConfirmedDataRoots` matches `now -
confirmedRootRetentionSeconds`, using the existing `rec` fixture and the
`pruneConfirmedDataRoots` stub.

---

Nitpick comments:
In `@src/workers/chunk-ingest-gc.ts`:
- Around line 151-157: The confirmed-data-root pruning result is currently
ignored in chunk-ingest GC, so add observability for the number of pruned
markers. In chunk-ingest-gc.ts, capture the return value from
ChunkPlacementIndex.pruneConfirmedDataRoots inside the GC flow and record it
with a dedicated metric (counter or gauge, similar to chunkIngestEvictedCounter)
so misconfiguration of CHUNK_INGEST_CONFIRMED_ROOT_RETENTION_SECONDS is visible.
Use the existing chunk ingest GC path and pruneConfirmedDataRoots symbol to wire
the metric in without changing the pruning behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 5277dc15-b95e-4d82-944d-1670384a86e6

📥 Commits

Reviewing files that changed from the base of the PR and between f58c1c5 and 815b701.

📒 Files selected for processing (10)
  • docker-compose.yaml
  • docs/chunk-ingest-cache.md
  • docs/envs.md
  • src/config.ts
  • src/database/chunk-placements.test.ts
  • src/database/sql/chunks/cache.sql
  • src/database/standalone-sqlite.ts
  • src/types.d.ts
  • src/workers/chunk-ingest-gc.test.ts
  • src/workers/chunk-ingest-gc.ts
✅ Files skipped from review due to trivial changes (3)
  • docker-compose.yaml
  • docs/envs.md
  • docs/chunk-ingest-cache.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/database/chunk-placements.test.ts
  • src/database/standalone-sqlite.ts

Comment thread src/workers/chunk-ingest-gc.test.ts
Adds monitoring and documentation for the sticky-confirmation marker introduced
in the prior commits, so operators can verify the marker table stays bounded:

- metrics: chunk_ingest_confirmed_roots (gauge, table size) and
  chunk_ingest_confirmed_roots_pruned_total (counter), set/updated each GC sweep
- countConfirmedDataRoots() DB method (worker + queue wrapper + handler + iface)
- glossary: "Sticky confirmation" entry
- docs/chunk-ingest-cache.md: the two new metrics
- test: count-method assertion

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BFuubmcaMUDBGvRVRyp9BH

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/database/standalone-sqlite.ts (1)

1064-1069: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add a short doc comment to countConfirmedDataRoots.

The sibling pruneConfirmedDataRoots (Line 1056) is documented, but this new marker-count reader has none. A one-liner noting it backs the observability gauge keeps this consistent.

As per coding guidelines: "Add or improve TSDoc comments on code you touch".

📝 Proposed doc comment
+  // Current number of confirmed-data-root markers. Source for the
+  // chunk_ingest_confirmed_roots observability gauge set during the GC sweep.
   countConfirmedDataRoots(): number {
     const row = this.stmts.chunks.countConfirmedDataRoots.get() as {
       count: number;
     };
     return row.count;
   }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/database/standalone-sqlite.ts` around lines 1064 - 1069, Add a short
TSDoc comment for countConfirmedDataRoots that explains it returns the current
confirmed data roots count used by the observability gauge; place it alongside
the existing standalone-sqlite methods near pruneConfirmedDataRoots so the new
reader is documented consistently.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/database/standalone-sqlite.ts`:
- Around line 1064-1069: Add a short TSDoc comment for countConfirmedDataRoots
that explains it returns the current confirmed data roots count used by the
observability gauge; place it alongside the existing standalone-sqlite methods
near pruneConfirmedDataRoots so the new reader is documented consistently.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 088bb161-951f-4604-827a-3931b3a85008

📥 Commits

Reviewing files that changed from the base of the PR and between 815b701 and 52882a9.

📒 Files selected for processing (9)
  • docs/chunk-ingest-cache.md
  • docs/glossary.md
  • src/database/chunk-placements.test.ts
  • src/database/sql/chunks/cache.sql
  • src/database/standalone-sqlite.ts
  • src/metrics.ts
  • src/types.d.ts
  • src/workers/chunk-ingest-gc.test.ts
  • src/workers/chunk-ingest-gc.ts
✅ Files skipped from review due to trivial changes (1)
  • docs/glossary.md
🚧 Files skipped from review as they are similar to previous changes (6)
  • src/types.d.ts
  • docs/chunk-ingest-cache.md
  • src/workers/chunk-ingest-gc.ts
  • src/workers/chunk-ingest-gc.test.ts
  • src/database/sql/chunks/cache.sql
  • src/database/chunk-placements.test.ts

@vilenarios

Copy link
Copy Markdown
Contributor Author

Deterministic confirm-before-any-chunk validation

The earlier E2E runs happened to land the first chunks a few seconds before the confirm event, so they didn't stress the exact ordering that broke the pre-fix code. To force it deterministically, I bumped the bundler's CHUNK_BROADCAST_GATE_MIN_CONFIRMATIONS (1 → 3) so chunk seeding waits ~2 extra blocks after the gateway confirms.

Captured the decisive window (16 MiB / 65-chunk bundle):

  • Gateway set the confirmed_data_roots marker with onDisk: 0 for ~73 seconds — the data root was confirmed while zero chunks existed.
  • When chunks finally seeded: confirmAtIntoWindow: -73s (confirm timestamp is 73 s before the first chunk's cached_at), lateArrivalsConfirmed: 65/65 — every chunk arrived after confirm and was still confirmed.
  • 65/65 complete, offset-0 present, 0 TTL evictions, held >3.7 min past the 180 s TTL.

lateArrivalsConfirmed = 100% is the key: sibling inheritance is impossible here (no confirmed chunk existed at confirm time), so the retention can only come from the unconditional marker. This is precisely the timing that evicted all chunks on the pre-fix code.

Test environment cleaned up

Gateway reverted to the released image + production TTL; bundler MIN_CONFIRMATIONS restored to 1. The one-row-per-confirmed-data-tx marker table is bounded by the age prune and observable via chunk_ingest_confirmed_roots / chunk_ingest_confirmed_roots_pruned_total.

…e (CodeRabbit)

Addresses two CodeRabbit review comments on PR #815:

- Major/perf: the sibling-confirmation fallback in saveChunkPlacement
  (SELECT ... WHERE data_root=? AND confirmed_at IS NOT NULL LIMIT 1) scanned the
  whole data_root range on the primary key whenever the confirmed_data_roots
  marker was absent (i.e. during a bundle's pre-confirmation ingest), making each
  insert O(n) and the window O(n^2) — worst case a multi-GB, thousands-of-chunk
  bundle. Add a PARTIAL index on chunk_placements(data_root) WHERE confirmed_at
  IS NOT NULL: tiny (empty exactly when the fallback runs most) and turns the
  lookup into an O(log n) seek (verified via EXPLAIN QUERY PLAN). Separate
  migration so it also applies where the marker-table migration was already
  recorded. Fallback retained (not redundant): it extends retention for chunks
  arriving after the marker's age-prune, as long as a confirmed sibling is still
  on disk.
- Minor/test: assert the GC sweep calls pruneConfirmedDataRoots exactly once with
  a now-minus-retention cutoff (previously recorded but unasserted).

Also corrects a stale comment in the marker-table migration (the EXISTS gate was
removed in 815b701).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BFuubmcaMUDBGvRVRyp9BH
@vilenarios

Copy link
Copy Markdown
Contributor Author

Both CodeRabbit findings addressed in 279b792:

  1. Sibling-confirmation lookup O(n) → O(log n) — added a partial index chunk_placements_confirmed_sibling_idx ON chunk_placements(data_root) WHERE confirmed_at IS NOT NULL. EXPLAIN QUERY PLAN now shows SEARCH chunk_placements USING INDEX chunk_placements_confirmed_sibling_idx (data_root=?) instead of a range scan. Put in a separate migration so it also applies on gateways where the marker-table migration was already recorded. (Kept the sibling fallback — it's not redundant with the marker: it extends retention for chunks that arrive after the marker's age-prune while a confirmed sibling is still on disk.)

  2. Unasserted rec.prunedBefore — the GC eviction test now asserts pruneConfirmedDataRoots is called exactly once per sweep with a now - retention cutoff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant