Skip to content

Deadline-aware cancellation (prototype) - #10018

Open
Jakub Jareš (nohwnd) wants to merge 61 commits into
mainfrom
nohwnd-deadline-aware-cancellation
Open

Deadline-aware cancellation (prototype)#10018
Jakub Jareš (nohwnd) wants to merge 61 commits into
mainfrom
nohwnd-deadline-aware-cancellation

Conversation

@nohwnd

@nohwnd Jakub Jareš (nohwnd) commented Jul 16, 2026

Copy link
Copy Markdown
Member

CI hard-cancels a job at a fixed wall-clock time. When it fires the runner kills the test process, and we lose the TRX/HTML/AzDO reports and get no dump for a hanging test. This is a prototype that tells MTP when that deadline is, so it can react a little early while it still owns its shutdown.

What it does

The deadline arrives as environment variables:

  • TESTINGPLATFORM_DEADLINE: the CI hard-cancel instant in ISO 8601 format, parsed to UTC.
  • TESTINGPLATFORM_DEADLINE_STOP_MARGIN: lead time for the graceful stop. Default 60s.
  • TESTINGPLATFORM_DEADLINE_DUMP_MARGIN: lead time for the hang dump. Default 30s.

Two reactions, armed off the same instant:

  • At deadline - stopMargin an in-process extension asks the framework to gracefully stop scheduling new tests (IGracefulStopTestExecutionCapability). In-flight tests finish, the session ends normally, and every reporter gets to finalize.
  • At deadline - dumpMargin the out-of-process HangDump controller dumps the process tree and kills the host. This is the fallback for a wedged host that never reaches the graceful stop. It reuses the controller that HangDump already runs, so it costs nothing extra when --hangdump is on.

stopMargin > dumpMargin on purpose: try the clean stop first, dump only if that did not happen in time.

It is opt-in. No deadline set, both timers stay unarmed, no behavior change. The graceful stop also degrades to a no-op if the framework does not expose the capability.

Where the deadline comes from

The CI producer must export the actual hard-cancel instant. MTP does not derive the job timeout, and the producer must not subtract the stop or dump margin. MTP applies both margins after reading the deadline.

The preferred version is Arcade or the job orchestrator computing the hard-cancel instant once, before the job starts, and exporting it for every test process.

If the CI system does not expose that instant, the fallback is to compute now + the full job timeout in the first job step, before checkout, restore, build, or test work. For a 60-minute timeout:

GitHub Actions:

- name: Compute deadline
  shell: pwsh
  run: echo "TESTINGPLATFORM_DEADLINE=$((Get-Date).ToUniversalTime().AddMinutes(60).ToString('o'))" >> $env:GITHUB_ENV

Azure DevOps:

- pwsh: echo "##vso[task.setvariable variable=TESTINGPLATFORM_DEADLINE]$((Get-Date).ToUniversalTime().AddMinutes(60).ToString('o'))"
  displayName: Compute deadline

These fallback examples approximate the job start with the first step. Any delay before the first step reduces the real time available, so production CI should prefer the orchestrator-provided instant or subtract a small orchestration buffer. The 60s/30s MTP margins remain separate and provide the graceful-stop and dump lead time.

Verified

build.cmd -pack passes 0/0. New acceptance tests:

  • AbortAtDeadlineTests (5): past deadline stops immediately, future deadline stops when the timer fires, the stop margin is subtracted from the deadline, no deadline stays silent, missing capability is a no-op.
  • HangDumpTests.HangDump_AbsoluteDeadline_CreateDump (3 tfms): 30 minute inactivity timeout so only the deadline path can fire, and it produces a dump. Full HangDumpTests still 30/30.

This is a prototype, so LMK what you think about the shape before I polish it. Open questions:

  • Names for the env vars and the margins.
  • Whether the dump-at-deadline message should say "deadline" instead of reusing the inactivity-timeout text.

🤖

CI systems (AzDO, GitHub Actions) hard-cancel a job at a fixed wall-clock
time. When that happens the runner kills the test process, so we lose the
TRX/HTML/AzDO reports and get no dump for a hanging test.

This teaches MTP about that deadline through an environment variable and lets
it react a bit early, while it still controls its own shutdown:

- TESTINGPLATFORM_DEADLINE is an absolute instant (ISO 8601, parsed to UTC).
- At deadline minus stop margin (default 60s) an in-process extension asks the
  framework to gracefully stop scheduling new tests, so the session ends
  normally and every reporter finalizes.
- At deadline minus dump margin (default 30s) the out-of-process HangDump
  controller takes a dump of the process tree and kills the host, for the case
  where the host is wedged and never reaches the graceful stop.

The deadline comes from the environment, so there is no hardcoded timeout in
MTP. The margins are MTP side policy and are env overridable. Wiring the real
deadline from the CI timeout is a small bit of YAML, left for a follow-up.

It is opt-in: with no deadline set both timers stay unarmed and there is no
behavior change. The graceful stop also degrades to a no-op when the framework
does not expose IGracefulStopTestExecutionCapability.

Verified: build.cmd -pack passes 0/0, and the new acceptance tests pass
(AbortAtDeadlineTests 5/5, HangDumpTests 30/30 including the deadline dump).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings July 16, 2026 15:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a prototype “deadline-aware cancellation” mechanism to Microsoft.Testing.Platform (MTP) so CI can provide an absolute wall-clock deadline and MTP can proactively (a) request a graceful stop before the runner hard-kills the process, and (b) trigger HangDump as a fallback before the deadline.

Changes:

  • Introduces DeadlineHelper + new env vars (TESTINGPLATFORM_DEADLINE, *_STOP_MARGIN, *_DUMP_MARGIN) for parsing an absolute UTC deadline and margins.
  • Registers a new in-proc AbortAtDeadlineExtension that schedules a timer to request IGracefulStopTestExecutionCapability.
  • Extends HangDump to arm an additional one-shot timer for the absolute deadline and adds acceptance coverage for both graceful stop and deadline-driven dump.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs Adds acceptance coverage ensuring HangDump can be triggered via absolute deadline env var.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/AbortAtDeadlineTests.cs New acceptance suite validating the graceful-stop behavior around deadlines/margins and missing capability.
src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt Tracks newly added internal APIs/constants for PublicAPIAnalyzers.
src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.Framework.cs Registers AbortAtDeadlineExtension into the message bus when enabled.
src/Platform/Microsoft.Testing.Platform/Helpers/EnvironmentVariableConstants.cs Adds constants for the new deadline-related environment variables.
src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs New helper to read/parse deadline + margins from environment.
src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs New extension that arms a timer to request graceful stop before the deadline.
src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.MSBuild/InternalAPI.Unshipped.txt Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.HotReload/InternalAPI/InternalAPI.Unshipped.txt Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.HangDump/Microsoft.Testing.Extensions.HangDump.csproj Links DeadlineHelper.cs into HangDump extension for shared deadline parsing.
src/Platform/Microsoft.Testing.Extensions.HangDump/InternalAPI/InternalAPI.Unshipped.txt Tracks DeadlineHelper + env-var constants for this assembly.
src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs Arms a new deadline-driven dump timer and prevents double dump via _dumpTaken.

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs Outdated
Jakub Jareš (nohwnd) and others added 2 commits July 16, 2026 17:51
…-cancellation

# Conflicts:
#	src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt
Fixes from the review of #10018:

- HangDump: the deadline timer and the inactivity timer both wrote
  _activityIndicatorTask, so the losing one could overwrite the winner's
  real dump task with a completed no-op, and Dispose would stop waiting for
  the dump mid-flight. Move the one-shot guard into a TriggerDumpOnce
  trampoline so only the winning timer assigns _activityIndicatorTask.

- HangDump: the deadline path logged "Hang dump timeout expired", which is
  misleading because no inactivity timeout expired. Give the deadline case
  its own reason and its own output message (new HangDumpDeadlineReached
  resource + regenerated xlf).

- AbortAtDeadlineExtension: compute a local non-null stopAt instead of
  dereferencing _stopAt.Value, so there is no nullable deref.

- Add the UTF-8 BOM to the three new source files to satisfy the
  charset=utf-8-bom editorconfig rule.

Verified: build 0/0, AbortAtDeadline 5/5, full HangDump suite 30/30.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 16, 2026 16:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Jakub Jareš (nohwnd) and others added 2 commits July 20, 2026 13:22
…-cancellation

# Conflicts:
#	src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt
More fixes from the review of #10018:

- HangDump: the absolute deadline timer was armed only after the pipe
  handshake in OnTestHostProcessStartedAsync. A test host that wedges during
  startup never connects back over the pipe, so those waits blocked past the
  deadline and the deadline dump/kill was never armed, which is exactly the
  case the deadline is for. Arm the deadline timer right after we have the
  test host process info (before the handshake). The dump path only needs the
  PID; the in-progress-test list needs the consumer pipe, so I make it
  best-effort and skip it when the pipe never connected.

- HangDump: the winning timer published _activityIndicatorTask without any
  ordering against disposal. Disposal could read the field as null, release,
  and tear the pipes down while a dump the timer just started was still
  running. Guard the "take the dump once" gate and the task publish under one
  lock, and have Dispose/DisposeAsync take that lock, claim the gate so no new
  dump can start, and capture the in-flight task to wait on outside the lock.
  The lock is a System.Threading.Lock on net9.0 and an object below it, so it
  still compiles on netstandard2.0.

- AbortAtDeadline and HangDump: deadline - margin on DateTimeOffset throws for
  a very old (but valid) deadline or a large margin. Add a shared
  DeadlineHelper.SubtractSaturating that clamps at DateTimeOffset.MinValue, so
  underflow means "already in the past" -> act immediately.

Verified: build 0/0, AbortAtDeadline 5/5, full HangDump suite 30/30.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 20, 2026 11:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
The automated re-review flagged seven spots after the last push. All are in
the two deadline timers and the hang dump resource text.

AbortAtDeadlineExtension:
- DataTypesConsumed is now empty. The extension only implements IDataConsumer
  so the message bus keeps a live reference to it (which keeps its timer
  alive). Returning [TestNodeUpdateMessage] made the bus route every test
  result to a no-op ConsumeAsync, which is O(test-count) for nothing.
- The timer callback calls HandleDeadlineAsync directly instead of Task.Run.
  On single-threaded runtimes (browser/WASI) Task.Run can queue work that
  never runs; the method is already async and yields at the first await.
- Arming the timer clamps a far-future due time to the Timer maximum
  (~49.7 days) instead of throwing. The run is disposed long before that, so
  the timer never fires early in practice.
- The graceful stop now runs in its own try/catch, separate from the
  best-effort diagnostics. A logging or output-device failure can no longer
  skip the stop, and the diagnostics failure log is itself swallowed so it
  cannot re-throw and skip the stop either.

HangDumpProcessLifetimeHandler:
- Same far-future clamp on the deadline dump timer.
- The in-progress-test query before dumping is wrapped in try/catch. A
  non-null pipe client is not necessarily connected (it is created when the
  host sends its pipe name but connected later), so a deadline dump firing in
  that window could hit an unconnected pipe. Any failure is logged and
  swallowed so it cannot block taking the dump and killing the tree.
- Renamed the resource HangDumpDeadlineReached to HangDumpDeadlineApproaching
  and reworded the text and log reason to "approaching". The dump fires at
  deadline minus the dump margin, so the deadline has not been reached yet.
  Regenerated the xlf files.

Local Debug pack is 0/0. AbortAtDeadline acceptance tests 5/5 and the full
HangDump acceptance suite 30/30.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 22, 2026 06:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:459

  • This request is not actually best-effort when the host is wedged. NamedPipeClient.RequestReplyAsync waits for a response until the supplied run token is canceled (NamedPipeClient.cs:103-105), so a connected host whose control-pipe callback no longer runs can block here indefinitely and prevent both dump creation and process-tree termination. Skip this query for deadline-triggered dumps or bound it to a short deadline-specific budget.
                GetInProgressTestsResponse tests = await _namedPipeClient.RequestReplyAsync<GetInProgressTestsRequest, GetInProgressTestsResponse>(new GetInProgressTestsRequest(), cancellationToken).ConfigureAwait(false);

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs

@Evangelink Amaury Levé (Evangelink) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — deadline-aware cancellation

I ran this through several review passes (expert MTP reviewer + a bug-focused diff reviewer) and then validated every finding against the code on 14d6b88. Skipping everything the earlier Copilot round already covered and you already fixed.

The shape is good and the concurrency work after the last round (the _dumpLock gate + TriggerDumpOnce trampoline, SubtractSaturating, the timer clamp) reads correctly to me. What follows is what survived validation, roughly in priority order. The first three are the ones I'd want settled before this stops being a prototype.

Things I checked and found clean, so you don't have to re-litigate them: the empty DataTypesConsumed is legal (AsynchronousMessageBus.InitAsync just creates no processor, and the consumer is still disposed via CommonTestHost line 401's messageBus.DataConsumerServices loop); the InternalAPI.Unshipped.txt entries are exact across all five projects that link-compile EnvironmentVariableConstants.cs; the .xlf files are genuinely generated (state="new", source==target, alphabetical); [Embedded] on DeadlineHelper matches every neighbour in that folder; and no new test hard-codes a \(\d+ms\) duration pattern.

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.Framework.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs
Comment thread src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs
Jakub Jareš (nohwnd) and others added 3 commits August 4, 2026 13:24
…-cancellation

# Conflicts:
#	src/Platform/Microsoft.Testing.Extensions.HangDump/InternalAPI/InternalAPI.Unshipped.txt
#	src/Platform/Microsoft.Testing.Extensions.MSBuild/InternalAPI.Unshipped.txt
#	src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt
…ocalization, disposal)

This is the review round on the deadline-aware cancellation prototype. The big
behavioural change is the exit code: a run that the deadline cuts short no longer
reports success.

- A deadline-truncated run now returns ExitCode.TestExecutionStoppedAtDeadline (15)
  instead of 0. The stop policy service gained IsDeadlineTriggered and a deadline
  callback, mirroring the max-failed-tests path, and the extension sets the flag
  before it asks for the graceful stop so the exit code cannot be raced by session
  finalization. Without this a suite that never finished would go green on CI, which
  is the opposite of what the feature is for.
- The in-proc extension is only registered for a console run. Server mode builds the
  framework per request, so it would re-arm the timer against the same absolute
  instant on every request and fire immediately once the deadline passed. Discovery
  requests are skipped too.
- The operator-facing description and console message moved into PlatformResources and
  are regenerated into the 13 locales, matching the HangDump side.
- The graceful-stop logging is now wrapped the same way as the rest of the handler so a
  throwing logger cannot escape the timer callback and FailFast the process, and the
  handler task is drained with a bounded wait on disposal so it is not still touching
  the logger and output device after teardown.
- DeadlineHelper now logs the resolved deadline and margins, warns when the deadline is
  set but malformed, warns when the framework has no graceful-stop capability, and warns
  when the dump margin is not smaller than the stop margin. The offset-less footgun is at
  least visible in the log now. I removed the dead non-negative margin guard.
- HangDump disposes both timers on every teardown path (not only the clean exit), takes
  the dump outside the dump lock (claim the gate under the lock, yield before the work),
  and lets the deadline dump be published through the normal exited path by returning
  from the failed handshake when a dump is already in progress.
- Added DeadlineHelperTests covering parsing, timezone handling, margin fallback, and the
  saturating subtraction, which were only exercised indirectly before. Capped the
  acceptance asset's wait so a broken stop path fails the assertions fast instead of
  hanging until the harness times out.

Local Debug pack is 0/0. DeadlineHelper unit tests 28/28 and AbortAtDeadline
acceptance tests 5/5.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b0147c6f-0cfb-4dd6-b420-582b3c7988de
…-cancellation

# Conflicts:
#	src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/InternalAPI/InternalAPI.Unshipped.txt
#	src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md
Copilot AI review requested due to automatic review settings August 4, 2026 12:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 70 out of 70 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

test/UnitTests/Microsoft.Testing.Platform.UnitTests/Helpers/DeadlineHelperTests.cs:1

  • This new C# file is missing the UTF-8 BOM required by .editorconfig:66-67. Add the BOM so repository encoding checks and future rewrites preserve the required format.
    src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:246
  • For the startup-wedge case described above, this still blocks until the five-minute default handshake timeout. Killing a host that never connected does not complete the server's WaitForConnectionAsync, and the controller cannot reach WaitForExitAsync/OnTestHostProcessExitedAsync to publish the dump before the CI hard deadline. Race or cancel the handshake when the deadline dump wins, then await the dump task before continuing.
            await _logger.LogDebugAsync($"Wait for test host connection to the server pipe '{_singleConnectionNamedPipeServer.PipeName.Name}'").ConfigureAwait(false);
            await _waitConnectionTask.TimeoutAfterAsync(TimeoutHelper.DefaultHangTimeSpanTimeout).ConfigureAwait(false);
            using CancellationTokenSource timeout = new(TimeoutHelper.DefaultHangTimeSpanTimeout);
            using var linkedCancellationToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token);
            _waitConsumerPipeName.Wait(linkedCancellationToken.Token);

Comment thread src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs Outdated
The timer callback could pass the _disposed check, then Dispose could run to
completion and read _handleDeadlineTask while it was still null (draining
nothing), and only afterwards would the callback publish the task and run the
handler against torn-down services. Publish the task and set/read _disposed
under a shared lock so the two cannot interleave: either the callback publishes
before Dispose captures it (Dispose then drains it), or Dispose sets _disposed
first and the callback observes it and never starts the handler.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b0147c6f-0cfb-4dd6-b420-582b3c7988de
Copilot AI review requested due to automatic review settings August 4, 2026 13:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 92 out of 92 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:650

  • Each dump is awaited before the next one starts, despite the stated requirement to start the child and parent dumps together. On a multi-process tree this serializes potentially slow dump operations, can consume the remaining deadline margin, and lets a parent exit before its dump begins. Start all dump tasks during enumeration, then await them together.
        foreach (IProcess p in bottomUpTree)
        {
            await dumpProcessAsync(p, inProgressTests, cancellationToken).ConfigureAwait(false);
        }

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dbdb237-9648-4904-bf23-37ee975370b3

🤖
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

Copilot AI commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Build the repository

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 92 out of 92 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dbdb237-9648-4904-bf23-37ee975370b3

🤖
@github-actions

This comment has been minimized.

Copilot AI commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Build the repository

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 92 out of 92 changed files in this pull request and generated no new comments.

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Build Failure Analysis

Summary — All four cross-platform build legs (Linux/macOS × Debug/Release) fail with MSTEST0037 analyzer errors (WarnAsError); the Windows legs are unaffected because these are new test files not yet touched there in the same way, but the underlying issue is identical.

Root cause: MSTEST0037 — use Assert.IsGreaterThan instead of Assert.IsTrue for comparisons

The MSTest analyzer MSTEST0037 flags Assert.IsTrue(a > b)-style comparisons and requires the more diagnostic Assert.IsGreaterThan(b, a) form (it reports a clearer failure message showing both values). Two newly-added test methods in this PR use the disallowed pattern, and since these projects treat analyzer warnings as errors, the build fails on all four Linux/macOS legs.

Affected files / errors

Proposed fix (inline suggestions posted below)

- Assert.IsTrue(deadline - now > firstInterval);
+ Assert.IsGreaterThan(firstInterval, deadline - now);

(and the analogous change in AbortAtDeadlineExtensionTests.cs, using Now instead of now)


Build overview
  • Linux Release, Linux Debug, macOS Release, macOS Debug legs: Build failed. — 4 MSTEST0037 compile errors each (2 distinct call sites × duplicate reporting).
  • Windows legs (Windows_application-model_acceptance, Windows_Release, Windows_Debug, both sub-legs each): no errors reported — clean compile.
All MSBuild errors
Code Project File:Line Message
MSTEST0037 Microsoft.Testing.Extensions.UnitTests HangDumpTests.cs:110 Use 'Assert.IsGreaterThan' instead of 'Assert.IsTrue'
MSTEST0037 Microsoft.Testing.Platform.UnitTests AbortAtDeadlineExtensionTests.cs:207 Use 'Assert.IsGreaterThan' instead of 'Assert.IsTrue'

🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit 44794c1

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 104.5 AIC · ⌖ 1.75 AIC · ⊞ 13.3K · [◷]( · )

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 104.5 AIC · ⌖ 1.75 AIC · ⊞ 13.3K ·

Comment thread test/UnitTests/Microsoft.Testing.Extensions.UnitTests/HangDumpTests.cs Outdated
@Evangelink

Copy link
Copy Markdown
Member

The worked 60-minute CI examples currently export now + 55m, but the implementation then subtracts the 60s/30s margins again. That makes the graceful stop occur roughly six minutes before the advertised hard timeout. Also, “now” approximates job start only if this step runs first; placing it naturally before the test step shifts the computed deadline by all checkout/restore/build time. Please have the producer export the actual hard-cancel instant (preferably computed once by Arcade/job orchestration) and let the margins provide the lead time, or explicitly require this step to run first and document the extra buffer.

Make concurrent dump paths process-unique, keep controller and child deadline settings identical, and surface startup configuration warnings in normal output.

🤖

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@nohwnd

Copy link
Copy Markdown
Member Author

The worked 60-minute CI examples currently export now + 55m, but the implementation then subtracts the 60s/30s margins again.

Fixed the PR description. The fallback examples now use the full 60-minute timeout, must run as the first job step, and leave the 60s/30s lead time to MTP. The description also makes the Arcade/job-orchestrator hard-cancel instant the preferred producer and documents the first-step approximation.

🤖

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 95 changed files in this pull request and generated 1 comment.

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated content by GitHub Copilot. Generated by the Test Reviewer on PR (on open / sync) workflow. · auto · 255.5 AIC · ⌖ 1.01 AIC · ⊞ 16.9K ·


service.NotifyTestExecutionStarting();

Assert.IsFalse(service.IsTestExecutionCompleted);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test review · Grade B (80–89) — Covers restart-after-complete, but never proves completion was true before the restart.

Assert completed before restart, then assert restart makes it false again.

Suggested change
Assert.IsFalse(service.IsTestExecutionCompleted);
[TestMethod]
public void NotifyTestExecutionStarting_AfterPreviousRequestCompleted_ClearsIsTestExecutionCompleted()
{
StopPoliciesService service = new(_cancellationTokenSource.Object);
service.NotifyTestExecutionStarting();
service.NotifyTestExecutionCompleted();
Assert.IsTrue(service.IsTestExecutionCompleted);
service.NotifyTestExecutionStarting();
Assert.IsFalse(service.IsTestExecutionCompleted);
}

Comment on lines +128 to +142

[TestMethod]
public async Task ExecuteDeadlineCallbacksAsync_InvokesRegisteredCallback()
{
StopPoliciesService service = new(_cancellationTokenSource.Object);

int invocationCount = 0;
await service.RegisterOnDeadlineCallbackAsync(() =>
{
invocationCount++;
return Task.CompletedTask;
});

await service.ExecuteDeadlineCallbacksAsync();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test review · Grade B (80–89) — Verifies callback ran, but not that the deadline trigger flag changed too.

Also assert IsDeadlineTriggered after execution to lock the public state change.

Suggested change
[TestMethod]
public async Task ExecuteDeadlineCallbacksAsync_InvokesRegisteredCallback()
{
StopPoliciesService service = new(_cancellationTokenSource.Object);
int invocationCount = 0;
await service.RegisterOnDeadlineCallbackAsync(() =>
{
invocationCount++;
return Task.CompletedTask;
});
await service.ExecuteDeadlineCallbacksAsync();
[TestMethod]
public async Task ExecuteDeadlineCallbacksAsync_InvokesRegisteredCallback()
{
StopPoliciesService service = new(_cancellationTokenSource.Object);
int invocationCount = 0;
await service.RegisterOnDeadlineCallbackAsync(() =>
{
invocationCount++;
return Task.CompletedTask;
});
await service.ExecuteDeadlineCallbacksAsync();
Assert.AreEqual(1, invocationCount);
Assert.IsTrue(service.IsDeadlineTriggered);
}

Comment on lines +168 to +180

[TestMethod]
public async Task WhenPoliciesReportTestExecutionCompleted_TheDeadlineDoesNotTrigger()
{
TaskCompletionSource<bool> triggered = new(TaskCreationOptions.RunContinuationsAsynchronously);
_policiesService.SetupGet(x => x.IsTestExecutionCompleted).Returns(true);
_policiesService.Setup(x => x.ExecuteDeadlineCallbacksAsync()).Callback(() => triggered.TrySetResult(true)).Returns(Task.CompletedTask);

using AbortAtDeadlineExtension extension = CreateExtension(deadlineIn: TimeSpan.Zero);

Task completed = await Task.WhenAny(triggered.Task, Task.Delay(TimeSpan.FromSeconds(2), TestContext.CancellationToken));
Assert.AreNotSame(triggered.Task, completed, "The deadline fired even though the shared policy state reported completed execution.");
_capability.Verify(x => x.TryStopTestExecutionAsync(It.IsAny<CancellationToken>()), Times.Never);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test review · Grade B (80–89) — Covers the policy-state guard, but does not assert callbacks also stayed uncommitted.

Also verify ExecuteDeadlineCallbacksAsync was never called.

Suggested change
[TestMethod]
public async Task WhenPoliciesReportTestExecutionCompleted_TheDeadlineDoesNotTrigger()
{
TaskCompletionSource<bool> triggered = new(TaskCreationOptions.RunContinuationsAsynchronously);
_policiesService.SetupGet(x => x.IsTestExecutionCompleted).Returns(true);
_policiesService.Setup(x => x.ExecuteDeadlineCallbacksAsync()).Callback(() => triggered.TrySetResult(true)).Returns(Task.CompletedTask);
using AbortAtDeadlineExtension extension = CreateExtension(deadlineIn: TimeSpan.Zero);
Task completed = await Task.WhenAny(triggered.Task, Task.Delay(TimeSpan.FromSeconds(2), TestContext.CancellationToken));
Assert.AreNotSame(triggered.Task, completed, "The deadline fired even though the shared policy state reported completed execution.");
_capability.Verify(x => x.TryStopTestExecutionAsync(It.IsAny<CancellationToken>()), Times.Never);
[TestMethod]
public async Task WhenPoliciesReportTestExecutionCompleted_TheDeadlineDoesNotTrigger()
{
TaskCompletionSource<bool> triggered = new(TaskCreationOptions.RunContinuationsAsynchronously);
_policiesService.SetupGet(x => x.IsTestExecutionCompleted).Returns(true);
_policiesService.Setup(x => x.ExecuteDeadlineCallbacksAsync()).Callback(() => triggered.TrySetResult(true)).Returns(Task.CompletedTask);
using AbortAtDeadlineExtension extension = CreateExtension(deadlineIn: TimeSpan.Zero);
Task completed = await Task.WhenAny(triggered.Task, Task.Delay(TimeSpan.FromSeconds(2), TestContext.CancellationToken));
Assert.AreNotSame(triggered.Task, completed, "The deadline fired even though the shared policy state reported completed execution.");
_policiesService.Verify(x => x.ExecuteDeadlineCallbacksAsync(), Times.Never);
_capability.Verify(x => x.TryStopTestExecutionAsync(It.IsAny<CancellationToken>()), Times.Never);
}

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

🤖
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 95 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: aed1b93b-c485-4242-acb5-ffda4d6c7686

🤖
@github-actions

Copy link
Copy Markdown
Contributor

🧵 Parallel-safety audit — PR #10018

Parallelization — assemblies containing the changed test files:

Test assembly Scope Workers Analyzer coverage
Microsoft.Testing.Platform.UnitTests MethodLevel CPU count coverable once parallel-safety analyzers ship (attribute opt-in)
Microsoft.Testing.Extensions.UnitTests MethodLevel CPU count coverable once parallel-safety analyzers ship (attribute opt-in)
Microsoft.Testing.Platform.Acceptance.IntegrationTests MethodLevel CPU count coverable once parallel-safety analyzers ship (attribute opt-in)
MSTestAdapter.UnitTests off (TestContainer engine, no scheduler) n/a n/a — readiness-only

Findings: A (global-state) 0 · B (paths) 0 · C (declaration) 0 · D (over-serialization) 0 — by severity: Critical 0 · High 0 · Warning 0 · Info 0.

Audited all 12 changed test files (deadline/graceful-stop/hang-dump feature tests). None of them touch process-global state, shared filesystem paths, or [ResourceLock]/[DoNotParallelize] declarations in an unsafe way:

  • Environment variables are exercised entirely through mocked IEnvironment/IEnvironmentVariableProvider instances (DeadlineHelperTests, TestConfigurationEnvironmentVariableProviderTests, AbortAtDeadlineExtensionTests) — no calls to the real Environment.SetEnvironmentVariable/GetEnvironmentVariable appear in test bodies.
  • The one real Environment.SetEnvironmentVariable call is inside a generated acceptance-test asset (AbortAtDeadlineTests.cs's embedded Program.cs source, run as a separate out-of-process test host), not in the acceptance test method itself — it mutates that child process's own environment, not the test host process running the audit target, so it is out of scope for in-process races.
  • ServerTests.DeadlineStateIsIsolatedBetweenServerRequests binds its TcpServer on port 0 (ephemeral), so no fixed-port collision.
  • MSTestGracefulStopTestExecutionCapabilityTests mutates the process-global singleton PlatformServiceProvider.Instance.IsGracefulStopRequested, but every test method resets it to false in a finally block, and this suite runs under TestFramework.ForTestingMSTest's TestContainer engine, which has no parallel scheduler — recorded as readiness-only, not a live race.
  • CommonHostTests changes only add mocked IStopPoliciesService service registrations and new assertions; no shared state.
  • No changed file adds, removes, or edits a [ResourceLock], [DoNotParallelize], or assembly-level [Parallelize] attribute, and no .runsettings/testconfig.json/Directory.Build.props parallelization setting changed (the two touched .csproj files only add a <Compile Include> link for DeadlineHelper.cs).

Nothing to flag for parallel-safety.

Advisory only — heuristic, non-blocking. Re-run with /parallel-audit. This audit answers "is it parallel-safe?"; for testability, smells, or flakiness see the detect-static-dependencies / test-smell-detection / test-anti-patterns analyses.

🤖 Automated content by GitHub Copilot. Generated by the Parallel-safety audit on PR (on open / sync) workflow. · auto · 103.9 AIC · ⌖ 2.92 AIC · ⊞ 24.8K · [◷]( · )

@github-actions

Copy link
Copy Markdown
Contributor

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output.

Review the workflow run logs for details.

🧪 Expert test review — PR #10018

This PR adds the deadline-aware cancellation feature (graceful stop at a CI-imposed deadline, hang-dump escalation, new GitHubActionsExitCode reason, and MSTestGracefulStopTestExecutionCapability). The new and modified tests are consistently strong: focused scenarios, meaningful assertions on the actual contract (exit codes, timer scheduling, one-shot callback semantics, race-safety), and correct isolation of per-request/static state via finally blocks or fresh StopPoliciesService instances. No high-confidence actionable findings were identified; the summary below reflects an overview grade per file/area rather than per-line nitpicks.

GradeTestMutationNotesHow to improve
A (90–100) new DeadlineHelperTests.
TryGetDeadline_
WhenUnsetOrMalformed_
ReturnsFalse
4/4 killed Data-driven negative cases plus positive parsing/offset/fraction tests cover the parser's branches.
A (90–100) new DeadlineHelperTests.
SubtractSaturating_
WhenMarginEqualsAvailableRange_
ReturnsMinValue
3/3 killed Exact boundary case for the saturating-subtraction guard is explicitly tested alongside overflow and no-op cases.
A (90–100) new MSTestGracefulStopTestExecutionCapabilityTests.
OverlappingRunCannotClearAnActiveRunsStopRequest
4/4 killed Exercises overlapping run/discovery interleavings against the shared static flag with correct finally reset.
A (90–100) new StopPoliciesServiceTests.
RegisterOnDeadlineCallbackAsync_
RacingTheTrigger_
InvokesEveryCallbackExactlyOnce
2/2 killed Stress test targets the exact registration/trigger race the production comment describes, asserting each callback fires exactly once across 50 attempts.
A (90–100) new AbortAtDeadlineExtensionTests.
WhenGracefulStopFails_
TheDeadlineVerdictIsNeverCommitted
3/3 killed Uses TCS-gated synchronization (no sleeps) to prove the deadline verdict is not committed until the stop attempt resolves.
A (90–100) new AbortAtDeadlineExtensionTests.
FarFutureDeadlineIsScheduledInMultipleTimerIntervals
3/3 killed Verifies the timer-interval-splitting logic for deadlines beyond uint.MaxValue ms, including the exact zero case.
A (90–100) new ServerTests.
DeadlineStateIsIsolatedBetweenServerRequests
3/3 killed End-to-end server-mode test confirms deadline state and exit codes do not leak across concurrent discovery requests.
A (90–100) new TestApplicationResultTests.
GetProcessExitCode_
WithOverlappingRequestDeadline_
IsolatedPerRequest
2/2 killed Confirms exit-code isolation between two independent StopPoliciesService/TestApplicationResult pairs.
A (90–100) mod GitHubActionsExitCodeTests.
GetReason_
ForDeadlineStop_
IsNotTheUnknownFallback
2/2 killed Pins the new deadline exit-code reason text against both the unknown fallback and the neighboring code to catch silent fallback regressions.
A (90–100) mod CommonHostTests.
ExecuteRequestAsync_
WhenSessionIsCancelled_
DisarmsStopPolicyOnlyForRun
2/2 killed Data-driven on discovery vs. run correctly asserts the stop policy is disarmed only for run requests, matching the finally-based disarm contract.
A (90–100) new AbortAtDeadlineTests.
WhenDeadlineIsInThePast_
GracefullyStopsImmediately
2/2 killed Acceptance test asserts exit code, stop message, and summary counts together, not just process success.

This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Suggestions on the Files
changed tab can be applied with one click. Re-run with
/review-tests.

🤖 Automated content by GitHub Copilot. Generated by the Test Reviewer on PR (on open / sync) workflow. · auto · 156.5 AIC · ⌖ 1.01 AIC · ⊞ 16.9K · [◷]( · )

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 95 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs:149

  • The fallback warning is queued inside TryLog, after two logger calls. If either logger provider throws, TryLog swallows the exception before _startupWarnings.Add runs, so the user never sees the invalid margin ordering even though this output warning is meant to survive diagnostic-logger failures. Queue the warning outside the best-effort logging delegate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants