Skip to content

test: investigate repeated Windows net10 stream delivery failures #11451

Description

@ReubenBond

Summary

Four October 9 Windows net10 BVT jobs reported managed stream-delivery failures across memory-stream, generator, hosted-client, and grain-call-filter tests. This tracker preserves the observed failures for a focused follow-up. The failures occur on the open durable-messaging stack; their root cause and introducing change are unproven.

Observed CI failures

Source PR Run / Windows net10 BVT job Source commit
#10693 37889691093 / 113687565453 4d13e427fa7a4cb26cde170b1c0375be6c6bb2d9
#11285 37888208842 / 113682903672 24771e88c50e88015554c63cf551ca76c631aa03
#11284 37887220107 / 113679818752 03703d672a2d6374fa08afeb7056e4e03e678dcc
#11282 37885447584 / 113674309966 de7b37caad9d18145e782f80bcb0f2521fceafee

All are attempt 1. The logs report ordinary managed test failures with MTP exit 2. All four explicitly complete Orleans.Serialization.UnitTests successfully.

Exact observed outcomes

  • MemoryProgrammaticSubcribeTests.StreamingTests_Consumer_Producer_SubscribeToStreamsHandledByDifferentStreamProvider: canceled waiting for 80 deliveries from StreamProvider2, observed 0.
  • MemoryProgrammaticSubcribeTests.StreamingTests_Consumer_Producer_Subscribe: canceled waiting for 100 deliveries from StreamProvider1, observed 0.
  • MemoryProgrammaticSubcribeTests.StreamingTests_Consumer_Producer_ConsumerUnsubscribeOnAdd: subscription-removal wait canceled.
  • MemoryStreamBatchingTests.BatchSendSingleConsume: canceled waiting for 30 deliveries from StreamBatchingTest, observed 0, in run 37888208842.
  • StreamGeneratorProviderTests.ValidateGeneratedStreamsTest and ControllableStreamGeneratorProviderTests.Validate2ControllableGeneratedStreamsTest / ValidateControllableGeneratedStreamsTest: expected 4, observed counts between 0 and 3 in the inspected logs.
  • HostedClientTests.HostedClient_StreamTest: expected true, actual false, in run 37889691093.
  • GrainCallFilterTests.GrainCallFilter_Incoming_Stream_Test: expected 86, actual 0, in run 37888208842.

The diagnostic wait failures include StreamingDiagnosticObserver.WaitForItemDeliveryCountAsync. These observations support investigating a shared delivery/readiness or branch-change boundary, while each failure retains its own assertion and outcome until a common cause is demonstrated.

Passing current-main control

Current-main Windows net10 BVT job 113546262383, commit 48f3aaa47fc18123db4109063af591f315e6f9cb, reports:

  • Streaming: 524/524 passed.
  • DefaultCluster: 335/335 passed.
  • Runtime: 307/307 passed.

Its TRX explicitly passes BatchSendSingleConsume, the different-provider subscription test, ValidateControllableGeneratedStreamsTest, HostedClient_StreamTest, and GrainCallFilter_Incoming_Stream_Test. One passing control does not distinguish intermittent exposure from an open-stack regression.

Existing-fix check and investigation boundary

Exact method/diagnostic searches found no existing tracker or demonstrated matching repair. Merged #11061 establishes delivery readiness for Receivers_ShouldMoveCloseTo_PullingAgent; merged #11142 retains local observer implementations in observer call-filter tests. Their described failure mechanisms do not establish a repair for the above outcomes. Open #10588 changes streaming recovery but has not been attributed as their fix.

Follow-up should compare the open-stack revisions against their common base and current main, preserve actual runtime/test-framework/runner configuration, and determine whether the new results reflect an introduced product defect, test readiness/lifetime, or another failure source. No local stress run, skip, retry, or workaround was added during discovery. Found while refreshing #11175; that issue retains its separate native Server GC compaction crash evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions