Skip to content

fix(EN-1441): propagate initIndexConfig boot errors with self-healing retry#1482

Draft
paul-nicolas wants to merge 3 commits into
release/v3.0from
EN-1441-initindexconfig-swallows-boot-errors
Draft

fix(EN-1441): propagate initIndexConfig boot errors with self-healing retry#1482
paul-nicolas wants to merge 3 commits into
release/v3.0from
EN-1441-initindexconfig-swallows-boot-errors

Conversation

@paul-nicolas

Copy link
Copy Markdown
Contributor

Summary

Builder.initIndexConfig returned void and logged-and-returned on every boot error branch, so Builder.loop could not observe the failure and unconditionally advanced to processLogs. A transient Pebble/read-store failure at boot left the in-memory index config empty or partially populated — the write path then silently skipped index writes (byCanonical miss ⇒ isMetadataIndexed/isBuiltinIndexed return false) while the persisted cursor kept advancing, leaving the read-side index permanently incomplete with no error, no retry, and no crash.

Fix

  • initIndexConfig(ctx) error (was void): the four boot read branches now return a wrapped error instead of a swallowed log. A top-of-function reset of indexConfig/bucketIndexConfig/backfillTasks/schemaRewriteTasks/indexVersions makes it idempotent across retries (the two slices would otherwise double-schedule).
  • Extracted the boot prologue into bootInit(ctx) (cursor, pebbleLast, err) and wrapped it in the existing worker.RetryWithBackoff (100 ms → 10 s, honors the stop channel). A ctx.Err() check after the retry distinguishes "init succeeded" from "shutdown requested" so loop never processes logs against a failed init.
  • This also closes the identical silent-return defects at the sibling LastIndexedSequence and NewDirectReadHandle boot reads. ReadAppliedProposalProgress / ReadLastSequence keep their existing best-effort semantics.

A transient boot read failure now retries with backoff instead of silently advancing the cursor against an incomplete config.

Tests

  • TestInitIndexConfig_PropagatesReadError — a closed store makes the boot read fail; the error propagates instead of being swallowed.
  • TestInitIndexConfig_IdempotentAcrossRetries — calling initIndexConfig twice does not double-schedule backfill/rewrite tasks.
  • TestBootInit_PropagatesError / TestBootInit_Success — cover the extracted prologue.
  • worker.RetryWithBackoff mechanics are already covered by internal/pkg/worker/worker_test.go.

go build ./..., just pre-commit (0 lint issues), and go test ./internal/application/indexbuilder/... all pass.

Ticket

https://formance-team.atlassian.net/browse/EN-1441

Same silent-swallow class as EN-1440.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d67dc7f7-66ee-4960-8133-28e7092b1827

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch EN-1441-initindexconfig-swallows-boot-errors

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.97297% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.28%. Comparing base (5ddbe29) to head (8c355cb).
⚠️ Report is 1 commits behind head on release/v3.0.

Files with missing lines Patch % Lines
internal/application/indexbuilder/builder.go 80.76% 2 Missing and 3 partials ⚠️
internal/application/indexbuilder/index_config.go 54.54% 3 Missing and 2 partials ⚠️

❌ Your patch check has failed because the patch coverage (72.97%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@               Coverage Diff                @@
##           release/v3.0    #1482      +/-   ##
================================================
- Coverage         73.22%   71.28%   -1.94%     
================================================
  Files               388      388              
  Lines             39765    40022     +257     
================================================
- Hits              29117    28531     -586     
- Misses             7900     8106     +206     
- Partials           2748     3385     +637     
Flag Coverage Δ
e2e 71.28% <72.97%> (-1.94%) ⬇️
scenario 71.28% <72.97%> (-1.94%) ⬇️
unit 71.28% <72.97%> (-1.94%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant