Repository navigation
fix(membership): ignore health monitor stop cancellation - #11421
ReubenBond merged 1 commit into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The narrowly scoped cancellation handling preserves other error logging and has focused regression coverage; tests were not executed.
Review effort: Balanced
Findings: None
What changed in this PR
Prevents expected health-monitor shutdown cancellation from producing error logs, addressing the rolling-upgrade flake tracked in #11408.
Changes:
- Handles cancellation after monitor shutdown without suppressing other exception types.
- Adds regression tests for shutdown cancellation and unexpected failures before and after stopping.
| File | Description |
|---|---|
| test/Orleans.Core.Tests/Membership/SiloHealthMonitorTests.cs | Adds three focused cancellation and error-logging tests. |
| src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs | Filters expected shutdown cancellation in the monitoring loop. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage
Report-only conclusion: current-main baseline stale. The newest successful coverage run tested 961a9bc, not current main 3d64611. Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
Problem
SiloHealthMonitor can be stopped while a timed-out direct probe is awaiting the local stall detector. The monitor-owned cancellation then reaches the generic run-loop handler and is logged as an error, causing rolling-upgrade tests which assert clean runtime logs to fail.
Solution
Classify OperationCanceledException after the monitor's stopping token is canceled as expected lifecycle completion. Preserve error logging for all other failures, including failures which occur after stop is requested.
Add focused coverage for cancellation during stall detection, ordinary unexpected run-loop failures, and unexpected stall-detector failures after stop.
Fixes #11408
Microsoft Reviewers: Open in CodeFlow