Skip to content

test(eventsourcing): capture storage stall boundaries - #11420

Merged
ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:rb-issue-11398-accountwithoutlog-timeout
Oct 8, 2026
Merged

ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:rb-issue-11398-accountwithoutlog-timeout

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Problem

AccountWithoutLog timed out while confirming its first deposit, and the retained scheduler diagnostics show a prolonged synchronous BatchWorker turn. The account grain awaits storage synchronization during activation, so its successful preceding Balance call establishes completion of the initial read. The unresolved boundary is inside deposit/write processing.

Solution

Enable fixture-scoped logging for the boundaries needed to classify the next occurrence:

  • log-consistency worker phases, write/retry outcomes, and confirmation completion at Debug
  • memory storage provider entry at Trace, before storage-grain lookup, serialization, and the RPC
  • memory storage grain receipt and write completion at Debug

Namespace category filters capture the existing events across internal type renames. These diagnostics preserve the account's persistence and confirmation guarantees.

Rationale

The original artifact establishes the stalled worker but leaves its precise execution stage unresolved. The added boundaries distinguish work before provider entry, work before storage-grain receipt, and processing after receipt. Root-cause attribution remains under investigation in #11398.

Relates to #11398

Copilot AI balanced review requested due to automatic review settings October 7, 2026 08:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The namespace filters correctly cover the intended diagnostic boundaries without functional changes.

Review effort: Balanced
Findings: None

What changed in this PR

Adds fixture-scoped logging to diagnose intermittent event-sourcing storage stalls without changing runtime behavior.

Changes:

  • Enables Debug logging for log-consistency phases.
  • Enables Trace logging across memory-storage dispatch and grain receipt.
File Description
test/​Orleans.EventSourcing.Tests/​EventSourcingTests/​EventSourcingClusterFixture.cs Broadens fixture logging to capture stall boundaries.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 08:15
@ReubenBond
ReubenBond force-pushed the rb-issue-11398-accountwithoutlog-timeout branch from 3c27db5 to f727a55 Compare October 8, 2026 08:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The namespace filters correctly capture the intended existing diagnostic events without changing runtime behavior.

0 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@ReubenBond

Copy link
Copy Markdown
Member Author

Rebased onto main 165acd0d33 and published f727a55b69. The exact-head .NET CI run 37748676159 completed successfully, including both builds and all four Functional partitions.

Downloaded exact-head TRX artifacts confirm AccountWithoutLog passes on Windows net10/net8 (0.299s / 0.338s) and Linux net10/net8 (0.044s / 0.055s). Each EventSourcing Functional assembly reports 14 passes and its existing skipped test. Windows net10 logs correlate the same account across worker phases, provider entry, storage-grain receipt/write completion, and confirmation completion; this run also exercises cross-silo storage receipt.

The investigation correction and regression chronology are recorded in #11398. Initial synchronization completes during inherited activation; the historical stall is a prolonged synchronous deposit/write-processing turn whose exact execution stage remains unresolved. Diagnostic visibility is verified, while root-cause repair still requires richer failure evidence or a dump. Self-review and the requested rubber-duck review found no substantive issues.

The remaining failed PR check is the unrelated Documentation sample restore blocked by ImageSharp audit errors, already tracked in #11437. The earlier CodeGen (#11427) and rolling-upgrade (#10904) failures both passed in this run.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 83.29% (117,972 / 141,644) 83.28% (117,956 / 141,644) +0.0113 pp
Branches 72.81% (35,006 / 48,080) 72.79% (34,998 / 48,080) +0.0166 pp

Report-only conclusion: improved.

The current-main baseline is commit 165acd0d33 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

@ReubenBond
ReubenBond merged commit 74d7ae4 into dotnet:main Oct 8, 2026
91 of 92 checks passed
@ReubenBond
ReubenBond deleted the rb-issue-11398-accountwithoutlog-timeout branch October 8, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants