Skip to content

docs: add indexer/realtime operational runbook (#849) - #1001

Open
extolkom wants to merge 2 commits into
LabsCrypt:mainfrom
extolkom:docs/849-indexer-operational-runbook
Open

docs: add indexer/realtime operational runbook (#849)#1001
extolkom wants to merge 2 commits into
LabsCrypt:mainfrom
extolkom:docs/849-indexer-operational-runbook

Conversation

@extolkom

Copy link
Copy Markdown
Contributor

Summary

Closes #849

Adds an operational runbook for the indexer/realtime system, covering
how on-call should read indexer lag, when to use reset vs replay
(including why replay is safe to re-run), and what the indexer's
environment variables control — including behavior when the indexer
is disabled. Pure documentation, no code changes.

What's included

backend/docs/SSE_ARCHITECTURE.md

New operational/runbook section covering:

  • Reading indexer lag — from /health ([actual field names])
    and /v1/admin/metrics ([actual fields, if any beyond /health]),
    with the actual threshold values found in code ([threshold values]) and what each level means for on-call action.
  • Reset vs replay — what each admin endpoint actually does
    ([route paths]), when to use which, and the idempotency guarantee
    for replay grounded in the dedup constraint at
    [file:line or constraint name].
  • Env varsINDEXER_POLL_INTERVAL_MS, INDEXER_START_LEDGER,
    STREAM_CONTRACT_ID, SOROBAN_RPC_URL: what each controls, and the
    exact resulting behavior when the indexer is disabled ([actual behavior found in code]).
  • RPC-outage recovery walkthrough — step-by-step sequence for
    on-call when Soroban RPC goes down or ledgers are missed, referencing
    the reset/replay endpoints above.

backend/SSE_README.md

Short pointer section linking to the full runbook in
SSE_ARCHITECTURE.md, without duplicating content.

Scope / non-goals

  • Does not re-document on-chain data flow (already covered in
    ARCHITECTURE.md)
  • No code changes — documentation only
  • [If Step 9's stop condition triggered on anything: note here what
    couldn't be located in code — e.g. no explicit lag threshold logic
    found — and how the doc handles that gap without inventing a number]

Grounding

Every claim in this doc (endpoint behavior, threshold values, env var
effects, idempotency mechanism) was verified directly against the
current implementation in backend/src/, not inferred from the issue
description alone. Specific references:

  • Lag/health fields: [file path]
  • Reset endpoint: [file path]
  • Replay endpoint + dedup constraint: [file path]
  • Env var handling / disabled-indexer behavior: [file path]

Done-when checklist (from #849)

  • Lag reading + thresholds documented
  • Reset/replay endpoints documented, including the idempotency note
  • Env vars and disabled-indexer behavior documented

Verification

  • Pure docs change — no build/test/lint impact expected, but ran:
    • [lint/format command if repo has one for markdown] — [pass/fail]
  • Branch rebased on latest origin/main

Notes for reviewers

[Anything worth flagging — e.g. if a threshold or behavior found in
code seemed surprising or worth a maintainer double-check before this
becomes the on-call reference]

@extolkom
extolkom force-pushed the docs/849-indexer-operational-runbook branch from 1cb9a34 to 999b122 Compare July 27, 2026 13:23
@extolkom

Copy link
Copy Markdown
Contributor Author

The CI errors are pre-existing issues on the repository's main branch (LabsCrypt/flowfi) and are unrelated to the documentation PR:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docs] Add an indexer/realtime operational runbook (replay/reset/status, lag thresholds, RPC-outage recovery)

1 participant