Skip to content

Remove stale transaction log queues from health metrics - #14263

Closed
mengxu-oai wants to merge 1 commit into
apple:mainfrom
mengxu-oai:codex/fdb-tlog-health-cleanup
Closed

mengxu-oai wants to merge 1 commit into
apple:mainfrom
mengxu-oai:codex/fdb-tlog-health-cleanup

Conversation

@mengxu-oai

@mengxu-oai mengxu-oai commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem in brief

Detailed health metrics retain a transaction log's last queue value after its tracker becomes invalid or is removed. The aggregate worst queue already excludes that tracker, so the detailed and aggregate metrics can describe different sets of logs indefinitely.

Example

If log A has a queue of 500 bytes and log B has 100 bytes, the worst queue is 500. When A becomes invalid, the worst queue becomes 100, but A's 500-byte entry remains in the detailed health map.

Solution in brief

Rebuild the detailed TLog queue map from the valid trackers on each rate update. This changes health reporting only; rate limits and serialized types are unchanged.

The same example

After A becomes invalid, the detailed map contains only B's 100-byte queue, and the worst queue is 100. Removed trackers are handled the same way.

Validation

  • Added /fdbserver/Ratekeeper/TLogHealthMetrics to exercise valid, invalid, revalidated, removed, and empty tracker sets after normal and batch updates.
  • Dedicated Ratekeeper unit target: built and passed (1 test, 0 failures).
  • Throttling.toml with HealthMetricsApi: passed with buggify enabled (1 test, 4 clients). All four clients received detailed health metrics. The trace contained no severity-40 events.

The change removes retained entries in new Ratekeeper health updates. Existing client and proxy caching behavior is unchanged.

@mengxu-oai
mengxu-oai marked this pull request as ready for review October 10, 2026 05:22
@mengxu-oai mengxu-oai closed this Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant