feat(durable-messaging): add durable outbox delivery

[ReubenBond]

Problem

Outgoing durable messages need to become dispatchable with exactly the business and inbox state acknowledged by the journal, while preserving ownership across retries, activation loss and cancellation.

Solution

Add the internal durable outbox above #11284. Synchronous Send(envelope) stages outgoing intent alongside safe business changes; optional prepared batches establish the wakeup prerequisite earlier. Inbox consumers use HandleAsync and synchronous context.Complete() with a final uninterrupted mutation block through method return.

The outbox supplies the journal owner's single final IJournaledStateCaptureHook. Ordinary before hooks run first, then the worker directly awaits the durable self-wakeup prerequisite immediately before synchronous capture. Scheduling covers messages arriving during earlier hooks and its own I/O, and records the exact scheduler-returned job handle.

Six canonical standard durable collections retain messages, attempts, dead letters and ownership data. The existing sequence state seals an immutable message/physical-owner cohort against actual journal acknowledgement. That acknowledgement releases exactly the captured cohort for dispatch; later work remains pending for its next capture. All seven streams, wire identities, selected-format codec bindings and bounded metric dimensions remain stable.

Healthy owners retain their physical handle. Duplicate and orphan wakeups inspect grain-local state; absent ownership is repaired after successful recovery. Optional acquisition tasks and batches stay owned through actual outcomes. Before-hook failures preserve safe pending changes for ordinary persistence retry; acknowledged cohorts survive post-persistence hook errors. Actual storage failures retain terminal owner handling and their original cause.

The owner stops and drains operations before journal deletion and disposal/deactivation. Delivery cancellation retains CTS resources through actual transport outcomes. Bootstrap consumers exercise actual Journaling and Durable Jobs, handler completion, final capture ordering, C/C+1 acknowledgement, replay, recovery and ownership lifetime.

Dependency and scope

Exact inbox parent #11284: d51245a5851cd8c8d572b0efdf5cddc4049f3482; contracts #11282: b3b18dfe330489c6a26a0399bdf224d8abeb39b4. Published outbox head: 0903c255113b826af4471bebb271dc3cddf26f2e. Selected main: 8bc9fd244427351ad24ccc039a9e7642a42c1cc4. Immutable outbox comparison.

The handler-API migration adapts the bootstrap consumer while preserving the production outbox implementation and its actual capture/acknowledgement guarantees. Public hosting, provider cutover, packaging and source-backed site guidance remain in #10693. All merges remain human decisions.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Copilot review overview

🟡 Changes recommended

In-flight inbox and outbox pump turns do not revalidate callback generation and physical ownership after recovery, allowing stale callbacks to mutate current state.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Fence in-flight inbox pumps after recovery · New
• Fence in-flight outbox pumps after recovery · New

What changed in this PR

Adds the durable outbox layer above the journaled inbox, including durable ownership, delivery pumping, retries, dead letters, and extensive contract/functional tests.

Changes:

• Adds journaled outbox scheduling and delivery coordination.
• Extends journaling with observer and participant lifecycle APIs.
• Adds durable messaging routing, serialization, diagnostics, and test infrastructure.

File	Description
src/​Orleans.DurableMessaging/​*	Durable messaging runtime, contracts, routing, ownership, and delivery.
src/​Orleans.DurableMessaging/​README.md	Documents inbox/outbox behavior and layering.
src/​Orleans.DurableMessaging/​Configuration/​DurableInboxOptions.cs	Adds messaging configuration and validation.
src/​Orleans.Journaling/​IJournaledStateObserver.cs	Adds journal boundary observer contract.
src/​Orleans.Journaling/​IJournaledStateManager.cs	Adds observer registration.
src/​Orleans.Journaling/​IJournaledGrainParticipant.cs	Adds feature participant contract.
src/​Orleans.Journaling/​DurableGrain.cs	Initializes journaled participants.
src/​api/​Orleans.Journaling/​Orleans.Journaling.cs	Updates generated journaling API surface.
test/​Orleans.DurableMessaging.Tests/​*	Adds durable messaging contract, component, and functional tests.
Orleans.slnx	Includes durable messaging source and test projects.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Code coverage

Metric	Pull request	Current main	Variance
Lines	83.61% (121,705 / 145,557)	83.28% (117,956 / 141,644)	+0.3369 pp
Branches	73.28% (36,371 / 49,630)	72.79% (34,998 / 48,080)	+0.4931 pp

Report-only conclusion: improved.

The current-main baseline is commit 165acd0d33 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

• Pull request commit: 0903c25511
• View the pull request test run and coverage artifacts
• View the trusted coverage reporting run

[Copilot]

Copilot review overview

🟡 Changes recommended

A recovery race can commit outbox work without scheduling a durable owner.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Pending outbox messages can commit without a durable owner · New
• Fence in-flight inbox pumps after recovery

Resolved since last review (1)

• Fence in-flight outbox pumps after recovery

[Copilot]

Copilot review overview

🔵 Needs a closer look

The change spans journal commit hooks, durable-job ownership, asynchronous delivery, recovery, and broad failure-path testing requiring final human validation.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Pending outbox messages can commit without a durable owner
• Fence in-flight outbox pumps after recovery
• Fence in-flight inbox pumps after recovery

[Copilot]

Copilot review overview

🟡 Changes recommended

Inbox deletion does not block active interleaved pump or gate operations, allowing post-delete work or stale-generation failures.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Reject deletion while an inbox pump or gate is active · New

Resolved since last review (1)

• Pending outbox messages can commit without a durable owner

[Copilot]

Copilot review overview

🟡 Changes recommended

Remote delivery cancellation disposes its token source while in-flight attempts may still use it, creating a concurrency failure during ownership transitions.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Keep batch cancellation sources alive until attempts finish · New

Resolved since last review (1)

• Reject deletion while an inbox pump or gate is active

[Copilot]

Copilot review overview

🔵 Needs a closer look

It changes journaling lifecycle semantics and durable remote-delivery ownership across multiple runtime layers, requiring final human validation.

Review effort: Lite
Findings: 1

Open (1)

• Keep batch cancellation sources alive until attempts finish

[Copilot]

Copilot review overview

🟡 Changes recommended

Inbox owner clearing can race interleaved acceptance preparation and fence the activation instead of persisting the incoming message.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Serialize inbox owner clearing with new admissions · New

Resolved since last review (1)

• Keep batch cancellation sources alive until attempts finish

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

1 open finding

• Prevent handler sends from persisting before completion · New

🧠 Review effort: Lite

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.