fix(EN-1441): propagate initIndexConfig boot errors with self-healing retry#1482
fix(EN-1441): propagate initIndexConfig boot errors with self-healing retry#1482paul-nicolas wants to merge 3 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (72.97%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## release/v3.0 #1482 +/- ##
================================================
- Coverage 73.22% 71.28% -1.94%
================================================
Files 388 388
Lines 39765 40022 +257
================================================
- Hits 29117 28531 -586
- Misses 7900 8106 +206
- Partials 2748 3385 +637
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Builder.initIndexConfigreturnedvoidand logged-and-returned on every boot error branch, soBuilder.loopcould not observe the failure and unconditionally advanced toprocessLogs. A transient Pebble/read-store failure at boot left the in-memory index config empty or partially populated — the write path then silently skipped index writes (byCanonical miss ⇒isMetadataIndexed/isBuiltinIndexedreturnfalse) while the persisted cursor kept advancing, leaving the read-side index permanently incomplete with no error, no retry, and no crash.Fix
initIndexConfig(ctx) error(wasvoid): the four boot read branches now return a wrapped error instead of a swallowed log. A top-of-function reset ofindexConfig/bucketIndexConfig/backfillTasks/schemaRewriteTasks/indexVersionsmakes it idempotent across retries (the two slices would otherwise double-schedule).bootInit(ctx) (cursor, pebbleLast, err)and wrapped it in the existingworker.RetryWithBackoff(100 ms → 10 s, honors the stop channel). Actx.Err()check after the retry distinguishes "init succeeded" from "shutdown requested" soloopnever processes logs against a failed init.LastIndexedSequenceandNewDirectReadHandleboot reads.ReadAppliedProposalProgress/ReadLastSequencekeep their existing best-effort semantics.A transient boot read failure now retries with backoff instead of silently advancing the cursor against an incomplete config.
Tests
TestInitIndexConfig_PropagatesReadError— a closed store makes the boot read fail; the error propagates instead of being swallowed.TestInitIndexConfig_IdempotentAcrossRetries— callinginitIndexConfigtwice does not double-schedule backfill/rewrite tasks.TestBootInit_PropagatesError/TestBootInit_Success— cover the extracted prologue.worker.RetryWithBackoffmechanics are already covered byinternal/pkg/worker/worker_test.go.go build ./...,just pre-commit(0 lint issues), andgo test ./internal/application/indexbuilder/...all pass.Ticket
https://formance-team.atlassian.net/browse/EN-1441
Same silent-swallow class as EN-1440.