Skip to content

fix(membership): ignore health monitor stop cancellation - #11421

Merged
ReubenBond merged 1 commit into
dotnet:mainfrom
ReubenBond:rb-test-track-rolling-upgrade-health-monito
Oct 7, 2026
Merged

ReubenBond merged 1 commit into
dotnet:mainfrom
ReubenBond:rb-test-track-rolling-upgrade-health-monito

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Problem

SiloHealthMonitor can be stopped while a timed-out direct probe is awaiting the local stall detector. The monitor-owned cancellation then reaches the generic run-loop handler and is logged as an error, causing rolling-upgrade tests which assert clean runtime logs to fail.

Solution

Classify OperationCanceledException after the monitor's stopping token is canceled as expected lifecycle completion. Preserve error logging for all other failures, including failures which occur after stop is requested.

Add focused coverage for cancellation during stall detection, ordinary unexpected run-loop failures, and unexpected stall-detector failures after stop.

Fixes #11408

Microsoft Reviewers: Open in CodeFlow

Copilot AI balanced review requested due to automatic review settings October 7, 2026 08:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The narrowly scoped cancellation handling preserves other error logging and has focused regression coverage; tests were not executed.

Review effort: Balanced
Findings: None

What changed in this PR

Prevents expected health-monitor shutdown cancellation from producing error logs, addressing the rolling-upgrade flake tracked in #11408.

Changes:

  • Handles cancellation after monitor shutdown without suppressing other exception types.
  • Adds regression tests for shutdown cancellation and unexpected failures before and after stopping.
File Description
test/​Orleans.Core.Tests/​Membership/​SiloHealthMonitorTests.cs Adds three focused cancellation and error-logging tests.
src/​Orleans.Runtime/​MembershipService/​SiloHealthMonitor.cs Filters expected shutdown cancellation in the monitoring loop.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request
Lines 83.28% (117,894 / 141,566)
Branches 72.76% (34,936 / 48,016)

Report-only conclusion: current-main baseline stale.

The newest successful coverage run tested 961a9bc, not current main 3d64611.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

@ReubenBond
ReubenBond merged commit c50ac22 into dotnet:main Oct 7, 2026
148 of 150 checks passed
@ReubenBond
ReubenBond deleted the rb-test-track-rolling-upgrade-health-monito branch October 7, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: track rolling-upgrade health-monitor cancellation flake

2 participants