Skip to content

eventbroker: fix broker quota release and message validation#5643

Open
lidezhu wants to merge 2 commits into
masterfrom
ldz/fix-event-broker0712
Open

eventbroker: fix broker quota release and message validation#5643
lidezhu wants to merge 2 commits into
masterfrom
ldz/fix-event-broker0712

Conversation

@lidezhu

@lidezhu lidezhu commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #5642

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed changefeed scan quota handling so allocated capacity is properly returned after scan failures, interruptions, and partial scans.
    • Prevented invalid heartbeat and congestion-control messages from causing processing errors.
    • Corrected dispatcher metrics to include table-trigger dispatchers.
  • Tests

    • Added coverage for quota recovery, scan errors, invalid messages, and dispatcher metric tracking.

@ti-chi-bot

ti-chi-bot Bot commented Jul 12, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 12, 2026
@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Eventservice scan quota cleanup now covers early exits and scan failures, table-trigger dispatcher metrics remain balanced, and malformed heartbeat or congestion-control messages are ignored safely after logging.

Changes

Eventservice corrections

Layer / File(s) Summary
Scan quota release and failure coverage
pkg/eventservice/event_broker.go, pkg/eventservice/event_broker_test.go, pkg/eventservice/test_helper.go
doScan releases allocated quota on insufficient quota, scan errors, invalid scans, and partial scans. Tests cover quota restoration and injected table-info failures.
Table-trigger dispatcher gauge accounting
pkg/eventservice/event_broker.go, pkg/eventservice/event_broker_test.go
Table-trigger dispatcher registration increments the dispatcher gauge, and tests verify the gauge returns to baseline after removal.
Invalid message payload handling
pkg/eventservice/event_service.go, pkg/eventservice/event_service_test.go
Heartbeat and congestion-control messages with invalid payload lengths return early, with tests covering nil payloads.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: lgtm, approved

Suggested reviewers: asddongmen, wk989898, flowbehappy

Poem

I’m a rabbit with quotas tucked neat,
No leaked bytes beneath my feet.
Gauges hop in balanced rows,
Bad messages meet quick “no.”
Tests cheer: “The burrow’s bright!” 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description includes the issue number, but the required change summary, tests, questions, and release note are left empty or generic. Fill in what changed, list the tests run, answer the compatibility/documentation questions, and replace the release-note placeholder with a real note or None.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR addresses all three linked objectives: quota release, dispatcher gauge balance, and safe invalid-message handling.
Out of Scope Changes check ✅ Passed The added tests and mock support are directly tied to the stated fixes and no unrelated code changes stand out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title matches the main fixes: broker quota release and message validation.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ldz/fix-event-broker0712

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 12, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces fixes to the event service, including releasing memory quota when skipping scans due to insufficient quota, tracking table trigger dispatchers in metrics, and handling invalid message payloads by returning early. Corresponding unit tests have been added. The review feedback points out a potential quota leak if scanning fails later in the process, and warns about unsafe type assertions on message payloads that could lead to panics, suggesting safer 'comma-ok' type assertions instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

}

if uint64(sl.maxDMLBytes) > task.availableMemoryQuota.Load() {
releaseQuota(available, uint64(sl.maxDMLBytes))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While releasing the allocated quota here is necessary, a similar quota leak can occur if scanner.scan fails further down in this function (around line 728). When scanner.scan returns an error, the remaining scannedBytes of the allocated quota is not fully released, which can lead to temporary quota exhaustion until the next congestion control update.

Consider fully releasing the remaining quota in the error handling block of scanner.scan as well:

if err != nil {
    if scannedBytes > 0 {
        releaseQuota(available, uint64(min(sl.maxDMLBytes, scannedBytes)))
    }
    log.Error("scan events failed", ...)
    return
}

Comment thread pkg/eventservice/event_service.go
Comment thread pkg/eventservice/event_service.go
@ti-chi-bot ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 12, 2026
@lidezhu lidezhu marked this pull request as ready for review July 12, 2026 02:03
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 12, 2026
@lidezhu lidezhu changed the title eventservice: fix broker quota release and message validation eventbroker: fix broker quota release and message validation Jul 13, 2026
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 13, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, hongyunyan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [asddongmen,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 13, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-07-13 03:29:38.662831394 +0000 UTC m=+597964.698926440: ☑️ agreed by hongyunyan.
  • 2026-07-13 09:08:49.884144872 +0000 UTC m=+618315.920239918: ☑️ agreed by asddongmen.

@ti-chi-bot

ti-chi-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

@lidezhu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-unit-test 47a25dc link unknown /test pull-unit-test
pull-unit-test-next-gen 47a25dc link unknown /test pull-unit-test-next-gen

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eventservice: fix quota leak, dispatcher gauge mismatch, and invalid message handling

3 participants