Show failure details in GitHub Actions step-summary collapsible sections by azat-msft · Pull Request #10633 · microsoft/testfx

Azat Mukhametshin (azat-msft) · 2026-08-18T10:15:45Z

What

The GitHub Actions step summary listed only the fully-qualified name of each failed test, so investigating a failure meant leaving the summary page for the Annotations tab (which has no stack trace) or the raw workflow log.

Each failed test is now expanded into a collapsible <details> section:

Namespace.TestClass.TestMethod — 2.40s

Exception: System.InvalidOperationException

Location: src/Calc.cs:42

Expected: 42
Actual:   41

   at Calc.Add() in Calc.cs:line 42

The summary line reuses the test name — duration presentation and duration formatting of the existing "Slowest tests" section, so the two are visually consistent.

--report-gh-failure-details on|off (default on) restores the previous compact list. Existing GitHub error/warning annotations are unchanged.

Bounding the output

GitHub caps a job summary at 1 MiB and drops it entirely when exceeded — it does not truncate. Every reduction is stated in the rendered output rather than applied silently.

Bound	Limit	On overflow
Message length	2,000 chars	clipped, `[... truncated]` appended
Message rows	30 lines	clipped, `[... truncated]` appended
Stack trace length	4,000 chars	clipped, `[... truncated]` appended
Stack trace rows	30 frames	clipped, `[... truncated]` appended
Failure list	20 per project	`Showing the first 20 of N failed tests`
Expanded detail	shared budget	remaining failures degrade to compact lines + a note counting them
Whole project section	shared budget	section condenses to a one-line verdict that says why

The budget is shared, not per-section: the cap applies to the whole GITHUB_STEP_SUMMARY file, which every test project in a job appends to. The aggregate path divides the budget across modules; the direct path measures what sibling projects already wrote and claims only the remainder. Per-project overhead is reserved before dividing, so the bound applies to the rendered file rather than to the diagnostics alone. The final size check is made under the writer lock, in bytes, so two concurrent projects cannot both conclude they fit.

Clipping happens at capture time, not render time, so an enormous stack trace never reaches the aggregation fragment written to disk.

Injection safety

Values in <summary> are HTML-encoded — a generic test name like T.Map<string,int> would otherwise parse as a tag and swallow the rest of the line.
The code fence is chosen longer than the longest backtick run in the body, so a failure message containing a ``` fence cannot terminate our block and leak raw markdown.

Testing

21 unit tests in GitHubActionsSummaryReporterTests covering the rendered section, the off-switch, the no-details fallback, HTML encoding, fence escaping, both row limits, all truncation paths, the budget arithmetic, and a 40-module aggregate asserting the rendered file stays under GitHub's cap.
2 acceptance tests driving a real MTP session with an exception-carrying failure.
HelpInfoAllExtensionsTests --help / --info expectations updated.
End-to-end runs in CI across green, small-detail, oversized-detail, 5,000-failure and 30-project shapes: azat-msft/gh-report-validation.

Docs (PACKAGE.md, docs/glossary.md) and .xlf localization files updated.

Open question before this leaves draft

The limits above are hardcoded constants, chosen rather than measured — including MaxFailures = 20 and the 40%-of-cap target. The 40% figure exists because this extension is not the only writer to the summary file and cannot control what a test framework appends after it. Worth deciding whether any of these should be configurable options before merge.

Each failed test in the GitHub Actions job summary is now expanded into a collapsible <details> section carrying its failure message, exception type, resolved source location and stack trace, instead of only its name. - Capture failure diagnostics in GitHubActionsSummaryReporter, resolving the source location the same way the annotation reporter does (exception call site, falling back to TestFileLocationProperty). - Propagate the diagnostics through the CI summary fragments so aggregated multi-module dotnet test runs render them too. - Bound the output twice (per value and per section) and state every truncation explicitly, so the summary stays well under GitHub's 1 MiB cap. - HTML-encode test-provided values in <summary> and pick a code fence longer than any backtick run in the body, so a hostile message cannot break out. - Add --report-gh-failure-details on|off to keep the previous compact list. Fixes #10591 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 110eb208-0496-4c66-be51-46dc51b16db5

Copilot

Pull request overview

Adds actionable failure diagnostics to GitHub Actions job summaries, including aggregated multi-module runs.

Changes:

Captures and renders failure details in collapsible, injection-safe sections.
Adds --report-gh-failure-details on|off and output-size controls.
Updates tests, documentation, API baselines, and localization resources.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsSummaryReporterTests.cs`	Tests failure-detail rendering and limits.
`test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs`	Updates CLI help expectations.
`test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/GitHubActionsReportTests.cs`	Adds end-to-end summary tests.
`src/Platform/SharedExtensionHelpers/SummaryReporterHelpers.cs`	Adds failure diagnostics to test records.
`src/Platform/SharedExtensionHelpers/CiRunSummaryAggregation.cs`	Persists diagnostics through aggregation.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hant.xlf`	Adds Traditional Chinese localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.zh-Hans.xlf`	Adds Simplified Chinese localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.tr.xlf`	Adds Turkish localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ru.xlf`	Adds Russian localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pt-BR.xlf`	Adds Brazilian Portuguese localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.pl.xlf`	Adds Polish localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ko.xlf`	Adds Korean localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.ja.xlf`	Adds Japanese localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.it.xlf`	Adds Italian localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.fr.xlf`	Adds French localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.es.xlf`	Adds Spanish localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.de.xlf`	Adds German localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/xlf/GitHubActionsResources.cs.xlf`	Adds Czech localization entries.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/Resources/GitHubActionsResources.resx`	Defines new localized messages.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md`	Documents the new option.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/InternalAPI/InternalAPI.Unshipped.txt`	Updates GitHub reporter API baseline.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs`	Captures and renders failure diagnostics.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryArtifactPostProcessor.cs`	Applies the option during aggregation.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsFailureDetails.cs`	Implements bounded collapsible rendering.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineProvider.cs`	Registers and validates the option.
`src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineOptions.cs`	Defines the option name.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/InternalAPI/InternalAPI.Unshipped.txt`	Updates shared internal API baseline.
`docs/glossary.md`	Documents detailed failure summaries.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Azat Mukhametshin (azat-msft) · 2026-08-18T10:39:27Z

Validation in real GitHub Actions runs

Validated end-to-end in azat-msft/gh-report-validation with this build packed into that repo's local feed (extension 1.1.0-dev, platform 2.4.0-dev). Each PR's workflow echoes the summary size and the markers that prove which rendering path was taken, so the evidence is in the run log rather than a manual read of the Summary page.

PR	Pipeline	Summary	Result
#2 green run	✅ green	1,640 B	0 collapsible sections, 0 clips, no truncation notes — the new rendering adds nothing when there is nothing to report
#3 short failure details	❌ red (deliberate)	6,309 B	Every failure fully expanded, 0 clips, no truncation notes
#4 oversized failure details	❌ red (deliberate)	76,355 B	18 values clipped, list capped at 20 of 31, detail budget exhausted after 10 — all reported explicitly

The headline number for the size concern: in #4, 31 failures each carrying a ~6 KB message and a 40-frame stack trace produce a 76 KB summary — roughly 7% of GitHub's 1 MiB job-summary limit — with both truncation notes rendered:

> Showing the first 20 of 31 failed tests. See the workflow log or the test report for the remaining failures.
> Failure details for 10 listed test(s) were omitted because the job summary size limit was reached.

Those validation PRs also fix a pre-existing bug in that repo's workflow, unrelated to this change: it passed --report-gh-slow-test-threshold without the --report-gh master switch, which the reporter correctly rejects as an invalid configuration.

Azat Mukhametshin (azat-msft) · 2026-08-18T13:56:36Z

Fourth validation run: the failure-count axis

Added azat-msft/gh-report-validation#5, which applies the opposite pressure from the oversized-details run: 5,000 failing tests with tiny diagnostics rather than a few with enormous ones.

Summary size: 27749 bytes
Collapsible failure sections: 21
Clipped values: 0
> Showing the first 20 of 5000 failed tests. See the workflow log or the test report for the remaining failures.

5,000 failures produce a 27 KB summary — about 2.6% of GitHub's 1 MiB limit. Varying only the failure count (measured locally):

Failing tests	Summary size
600	17,754 B
5,000	17,809 B

The size is flat; the 55-byte delta is just the wider count in the text.

Notable result: I could not construct a summary that overflows purely from failure count. Both reporters bound their own sections — this one at 20 failures (12,750 B), TUnit's own block at a 50-row table (4,848 B). So failure count cannot push a run past the 1 MiB limit; only per-failure size can, which is exactly what the per-value clips and the per-section budget exist to contain.

The two runs bracket the design: #4 shows the size axis is bounded at runtime, #5 shows the count axis is bounded by construction.

One design question before this leaves draft

MaxFailures is a fixed 20. At 5,000 failures you see 20, with the note pointing at the workflow log and the test report for the rest. That seems like the right default for a 1 MiB page, but it is worth deciding explicitly whether the cap should be configurable — it is a small follow-up on top of this PR if so.

…t by rows The details budget was a per-section constant, but GitHub's 1 MiB cap applies to the whole GITHUB_STEP_SUMMARY file, which every test project in a job appends to. Twelve or so projects could therefore each spend a full budget and push the file past the cap, at which point GitHub drops the summary entirely. - Derive the budget from 80% of the 1 MiB cap and share it. The aggregate path divides it across modules; the direct path measures what sibling projects already wrote and claims only the remainder. - Report at the file level when the shared budget forced projects to render without details -- a per-module note is invisible inside a collapsed section. - Clip messages and stack traces by line count (30 each) as well as by length. A 200-frame trace of one-word frames sits under the character cap while being unreadable, so the character cap alone did not bound readability. Adds unit tests for the row limits, the budget arithmetic (including the unreadable-file fallback and the already-over-budget floor), and a 40-module aggregate that asserts the rendered file stays under GitHub's cap.

Validating with 30 test projects writing to one GITHUB_STEP_SUMMARY showed the budget was measuring the wrong thing. It capped the expanded details, but each project also writes several KB of headings, tables and failure lines, and the test framework appends its own ~5 KB block afterwards. Thirty projects landed at 1,018,161 bytes -- 97% of GitHub's 1 MiB cap, where GitHub drops the summary entirely rather than truncating it. - Reserve each project's non-detail overhead before dividing the budget, so the bound applies to the rendered file rather than to the diagnostics alone. - Condense a project's whole section to a single verdict line once the shared file nears the target, since at that point the per-project overhead is itself what would overflow the cap. The line still states the counts and says why it was condensed, so nothing is dropped silently. - Target 40% of the cap rather than 80%. This extension is not the only writer to the file: a test framework appending ~5 KB per project cannot be prevented by this reporter, only left room for. Thirty projects now render at 550,576 bytes (52.5%), down from 1,018,161 (97%).

Azat Mukhametshin (azat-msft) · 2026-08-18T23:16:06Z

Update: 30-project run found a budgeting bug, now fixed

Added azat-msft/gh-report-validation#6: 30 test projects appending to one GITHUB_STEP_SUMMARY, each contributing 25 failures with multi-line messages and deep stack traces.

The first run produced a 1,018,161 byte summary — 97% of GitHub's 1 MiB cap. A few more projects and GitHub would have discarded the entire summary, since an oversized summary is dropped rather than truncated.

Root cause

The budget capped expanded details, but two other things scaled with project count and were outside it:

Contributor	Per project
This reporter's non-detail content (heading, tables, failure lines)	~6 KB
The test framework's own summary block (TUnit here), appended after us	~5.1 KB

Fixes in this PR

Reserve per-project overhead before dividing the budget, so the bound applies to the rendered file rather than to the diagnostics alone.
Condense a project's whole section to a single verdict line once the shared file nears the target — at that point the per-project overhead is itself what would overflow. The line still reports counts and says why it was condensed, so nothing is dropped silently.
Target 40% of the cap rather than 80%. This reporter is not the only writer to the file: it can account for what earlier projects wrote by measuring the file, but cannot prevent a framework block landing after it. The headroom absorbs roughly 80 further projects of co-writer output.

Result (measured in CI, 30 projects)

	Before	After
Summary size	1,018,161 B	658,451 B
% of 1 MiB cap	97%	62.8%

Bulk projects with failures: 30
Summary size: 658451 bytes
Project sections: 6
Collapsible failure sections: 94
Clipped values: 128
❌ `Bulk05Tests` (net9.0): 25 total, 0 passed, 25 failed, 0 skipped — condensed to one line
because the job summary size limit was reached. See the workflow log or the test report for full results.

Also in this update

Row limits on failure details. A character cap alone does not bound readability: a 200-frame stack trace of one-word frames sits under the 4,000-character cap while being unreadable. Messages and stack traces are now capped at 30 lines each as well, with the same explicit truncation marker.

Validation matrix

PR	Axis	Pipeline	Summary
#2	green run	✅	1,640 B
#3	short details	❌	6,309 B
#4	oversized details	❌	76,355 B
#5	many failures (5,000)	❌	27,749 B
#6	many projects (30)	❌	658,451 B

Unit tests cover the row limits, the budget arithmetic (including the unreadable-file fallback and the already-over-budget floor), and a 40-module aggregate asserting the rendered file stays under the cap. Full suite: 1,108 passing.

…lit reporter main split GitHubActionsSummaryReporter into partial classes (#10562), which moved the markdown builders this branch had changed. Re-applies the failure details work onto the new layout: capture and budget helpers stay with the reporter, the collapsible rendering and the shared-budget arithmetic move to the Markdown partial.

An earlier edit dropped the newline between the new failure-details row and the slow-test-notices row, merging them into one seven-cell row that markdownlint rejected (MD056).

Copilot · 2026-08-18T23:36:02Z

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Build the repository

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:333

A framework can supply an empty or whitespace explanation together with a useful exception message. The null-coalescing expression selects that whitespace value, then Clip turns it into null, so the expanded failure omits the promised exception-message fallback. Treat whitespace explanations as absent.

            GitHubActionsFailureDetails.Clip(failure.Value.Explanation ?? exception?.Message, GitHubActionsFailureDetails.MaxMessageLength, GitHubActionsFailureDetails.MaxMessageRows),

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md:44

This table row also contains the slow-test option, so the package README renders both options as one malformed row and no longer documents --report-gh-slow-test-notices correctly. Split them into separate rows.

| `--report-gh-failure-details on\|off` | Expand each failed test in the job summary into a collapsible section carrying its failure message, exception type, source location and stack trace | on |

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsFailureDetails.cs:226

remainingBudget is based on Stream.Length, which is a byte count, but this comparison and subtraction use UTF-16 character counts. Since the summary is written as UTF-8, non-ASCII diagnostics can consume up to several times the reserved space and cross GitHub's byte limit even though the budget accepts them. Account for the UTF-8 byte count of each rendered block.

            if (detailsBuilder.Length > remainingBudget)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:234

The shared-file size is measured before acquiring the exclusive append handle. Concurrent test-host processes can therefore all observe the same old length, each render up to the full remaining budget, and then serialize multiple oversized sections through AppendStepSummaryWithRetryAsync; three first writers can exceed 1 MiB. Measure and build while holding the same cross-process lock used for the append.

            int detailsBudget = GetRemainingDetailsBudget(_fileSystem, path!, _logger);

            string markdown = detailsBudget <= 0 && IsSummaryNearLimit(_fileSystem, path!, _logger)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryArtifactPostProcessor.cs:65

The aggregate always receives a fresh 40%-of-limit budget without subtracting content already present in GITHUB_STEP_SUMMARY. If one workflow step runs multiple dotnet test commands (or concurrent aggregate processors use different aggregation IDs), every section can consume that budget and the upserts can collectively exceed 1 MiB. Size the step-summary variant against the existing file under the upsert lock; the standalone artifact can retain the full rendering.

        string markdown = GitHubActionsSummaryReporter.BuildAggregateMarkdown(aggregate, _includeFailureDetails);

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs:166

This reserve is only an estimate; it does not bound non-detail output. Once the reserve exhausts the details budget, the loop still emits a full section for every module, including uncapped assembly/test names and compact failure/slow-test lines. A sufficiently large aggregate can therefore exceed 1 MiB even with zero expanded details. Enforce the limit against the actual UTF-8 output and condense remaining modules when the budget is reached.

        int overheadReserve = moduleCount * GitHubActionsFailureDetails.PerProjectOverheadReserve;
        int detailsBudget = Math.Max(0, GitHubActionsFailureDetails.MaxSummaryLength - overheadReserve);
        int perModuleBudget = detailsBudget / moduleCount;

Copilot · 2026-08-18T23:47:10Z

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Build the repository

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsCommandLineProvider.cs:36

The new option is missing from the existing command-line provider test matrix in GitHubActionsCommandLineProviderTests.cs: both sub-option dependency tests enumerate every prior sub-option, and each prior boolean option has invalid-value coverage. Add GitHubActionsFailureDetails cases so the new --report-gh dependency and on|off validation remain protected.

            GitHubActionsCommandLineOptions.GitHubActionsGroups or GitHubActionsCommandLineOptions.GitHubActionsAnnotations or GitHubActionsCommandLineOptions.GitHubActionsStepSummary or GitHubActionsCommandLineOptions.GitHubActionsSlowTestNotices or GitHubActionsCommandLineOptions.GitHubActionsFailureDetails

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsFailureDetails.cs:216

remainingBudget is ultimately derived from Stream.Length and GitHub's byte limit, but StringBuilder.Length counts UTF-16 code units. Non-ASCII failure messages are therefore undercharged (often by 2–4× in UTF-8), so the aggregate can satisfy this check yet produce a file over 1 MiB. Track UTF-8 byte counts consistently and assert Encoding.UTF8.GetByteCount(markdown) in the size tests.

            if (detailsBuilder.Length > remainingBudget)

IDE0008 is enforced as an error in CI. The type was not apparent from the right-hand side because it comes from a LINQ projection, unlike the other 'var' uses here which are all 'new T(...)'.

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:232

The budget is measured before acquiring the exclusive append handle. Parallel test-host processes can therefore all observe the same file length, each render up to the full remaining budget, and only then serialize their appends; the resulting file can exceed GitHub's limit and be dropped. Measure and render while holding the same interprocess lock used for the append, or re-check and re-render after acquiring it.

            int detailsBudget = GetRemainingDetailsBudget(_fileSystem, path!, _logger);

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs:187

When the calculated details budget reaches zero, this still appends the full table, failure list, and slow-test list for every module. PerProjectOverheadReserve is only subtracted from the details allowance; it does not cap actual overhead, so a sufficiently large module count still produces a summary over 1 MiB. Enforce a file-level budget before each module and switch remaining modules to a bounded one-line verdict (with an explicit omission note).

            if (AppendModuleMarkdown(builder, module, headingLevel: 3, includeFailureDetails, ref remainingBudget) > 0)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:368

An I/O error while measuring an existing summary is not equivalent to an empty file. Returning the full budget here can append hundreds of kilobytes to a file that is already near the cap, causing GitHub to drop the entire summary. Distinguish “file absent” from “measurement failed” and use a conservative/minimal rendering fallback for the latter.

            return GitHubActionsFailureDetails.MaxTotalDetailsLength;

Copilot · 2026-08-19T00:05:23Z

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Build the repository

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

MSTEST0037 is enforced as an error in CI, which builds MSTest.Analyzers from source; the analyzer package restored locally predates the rule, so the local build did not flag it.

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:346

When the minimal verdict is refused, this project is omitted entirely, but BuildTruncationNotice only says that projects “reported below as a single line” lost their details. There is no line, name, or count for this dropped project, so the rendered summary is still silently incomplete despite the PR’s stated guarantee that every reduction is reported. Add a distinct omitted-project count/notice or reserve room for the minimal verdict.

        // This project's section is dropped entirely, which is exactly what the note at the top of the summary
        // describes, so make sure it is there even if this project was not condensed. The note is a few hundred
        // bytes and dropping the section frees far more room than it takes; if even that does not fit, the writer
        // refuses it and the summary is genuinely full, where silence is what keeps the rest rendered.
        await TryAppendNoticeOnlyAsync(writer, testSessionContext).ConfigureAwait(false);

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs:368

Full aggregate module sections do not emit ProjectSectionMarker, so a later direct writer in the same step cannot count them when building the truncation notice. For example, ten fully rendered aggregate modules followed by one condensed direct project produces a notice claiming that zero projects have full results. Mark aggregate full sections too so the reported count reflects the file.

            bool needsDiscriminator = HasDuplicateModuleIdentity(aggregate.Modules, module);
            builder.Append("<details>\n<summary>")
                .Append(HtmlEncode(module.AssemblyName))

Every failed test paid for its diagnostics at capture time - formatting Exception.StackTrace, walking it for a source location, and clipping the result - but only MaxFailures of them are ever rendered. A 5,000-failure run did that work thousands of times and retained tens of megabytes of clipped text it immediately discarded. Capture now keeps only the explanation and the exception reference, both already allocated, and the expensive part runs at session end for the selected failures alone. The snapshot is also sorted now, which makes the choice of which failures get expanded reproducible - dictionary order is arbitrary, so a run could previously expand a different twenty each time - and makes the direct path's selection agree with the aggregated path's. CountProjectSections also counted the section marker when a failure body printed it on a line of its own inside a fenced block, letting a test inflate the project count the truncation note reports. It now skips fenced content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryArtifactPostProcessor.cs:106

This length includes any existing section for the same aggregationId, even though UpsertStepSummaryWithRetryAsync removes that section before writing the replacement. Reprocessing the same inputs therefore budgets the new rendering against its own previous output; for example, an existing section above the 60% threshold makes an identical rerun condense all modules and add a truncation warning although replacing it with the original full section would fit. Compute the occupied bytes after excluding the matching start/end block (ideally while holding the upsert lock) and render against that replacement-aware length.

        long alreadyWritten = writer.GetSummaryLength() ?? 0;
        GitHubActionsSummaryReporter.AggregateRenderResult rendered = alreadyWritten == 0
            ? artifact
            : GitHubActionsSummaryReporter.BuildAggregateMarkdown(aggregate, _includeFailureDetails, condenseAllModules: false, alreadyWritten);

The aggregate now carries rendered failure messages, so anything used as a structural marker can also appear inside a fenced diagnostic as user-controlled text. Marker scanning is now fence-aware and whole-line: CountProjectSections no longer inflates the truncation note's project count, and the upsert no longer ends a section early on an end marker a test printed - which would have left the stale tail behind on the next upsert. The notice rewrite keeps its atomic temp-and-swap, but now captures the file length under its exclusive handle and re-checks it before the swap, retrying when the file moved. The extension's lock does not cover other summary producers, so an append landing during staging is carried into the next attempt instead of being overwritten by a snapshot taken before it. The aggregate budget also counted the block the upsert is about to replace, so reprocessing could condense or drop modules that fit fine once the old block was gone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs:16

This marker is emitted by direct project sections only; full aggregate module sections still begin directly with <details> and are never marked. If an aggregate run is followed in the same step-summary file by direct runs that eventually trigger condensation, CountProjectSections omits all aggregate modules and the warning's “first N test projects” count is incorrect. Emit this marker for each full aggregate module as well.

    /// <summary>
    /// Marks a full test project section, so the truncation note can state how many test projects got their
    /// results into the summary without depending on the localized heading text.
    /// </summary>
    internal const string ProjectSectionMarker = "<!-- microsoft-testing-platform:github:project-section -->";

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/StepSummaryWriter.cs:323

On the final retry, a detected co-writer append is still overwritten: when the length changed and attempt == _maxAttempts, this condition falls through to ReplaceFile with the stale payload. The last attempt should fail instead of knowingly discarding content appended by another summary producer.

                if (GetSummaryLength() is long lengthBeforeSwap
                    && lengthBeforeSwap != lengthAtCapture
                    && attempt < _maxAttempts)
                {
                    continue;
                }

Holding the TestNode for every failure kept its whole property bag alive, and that bag can carry unbounded captured standard output and error - so a high-failure run retained the run's entire output to render at most twenty failures. Only the declared file and line are read out of the bag now, at capture time, and the node is dropped. Retention is also bounded outright: the pending map keeps at most MaxFailures entries, ordered the same way the summary renders, so the diagnostics held are exactly the ones that will be shown. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Rebuilt from microsoft/testfx#10633 after the merge with main and the review rounds that followed: notice strength upgrading, deferred and bounded failure-diagnostic capture, fence-aware structural markers, and the budget no longer counting the block it is about to replace. The extension package moved to 2.4.0-dev alongside the platform, so the project references move with it.

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs:367

Full aggregate module sections are not tagged with ProjectSectionMarker, although CountProjectSections relies exclusively on that marker. If an aggregate with full modules is followed in the same step summary by a direct project that must condense (or upgrades the aggregate's details-only warning), the leading notice reports that zero projects received full results. Emit the marker before each full aggregate module, just as BuildMarkdownCore does for direct sections.

            bool needsDiscriminator = HasDuplicateModuleIdentity(aggregate.Modules, module);
            builder.Append("<details>\n<summary>")

The strength token was appended to the marker's own line, which silently broke every consumer that matches the marker as a whole line - including this project's validation workflow, whose end-to-end run reported seven condensed projects and no notice explaining them. The token now sits on the following line, inside the same delimited block. Covered by a test asserting the marker is the notice's first line. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:441

The direct path drops the project when this full rendering exceeds the writer's hard limit, instead of retrying the existing one-line rendering as the aggregate path does. This can occur even below the 60% threshold because failure identities are not length-bounded (and can fall back to an arbitrary display name), so a large section is refused and no project verdict reaches the summary. Re-render minimally when the projected UTF-8 size does not fit; the writer can still reject that fallback if the file truly has no room.

                    var budget = SummaryBudget.ForProject(currentLength);
                    bool condense = budget.Stage is SummaryStage.Condensed or SummaryStage.Unlisted;
                    string markdown = condense
                        ? BuildMinimalMarkdown(snapshot, assemblyName, _targetFrameworkMoniker.Value, exitCode)
                        : BuildMarkdown(snapshot, assemblyName, _targetFrameworkMoniker.Value, exitCode, coverage, _sections, _includeFailureDetails, budget);

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:190

A later terminal update for the same UID only replaces _records; it never clears an earlier pending failure. If that test later passes or is skipped, its stale entry can remain among the bounded first 20 candidates and evict diagnostics for a test that is actually failed in the final snapshot. Remove the UID before conditionally adding the current failure so the pending map mirrors _records.

                _records[uid] = new TestRecord(displayName, fullyQualifiedName, kind, duration);
                if (pendingFailure is { } failure)

An in-process retry reuses the test's UID, so a test that failed and then passed left its diagnostics behind in the bounded retention set. That entry could then evict a test that was still failing, whose failure would render as a bare line with nothing explaining why its details were missing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Every other consumer of test node updates skips superseded attempts; the step-summary reporter did not. A [Retry] test that failed and then passed was therefore held in the bounded failure set while its earlier attempt was live, where it could permanently evict the diagnostics of a test that really was failing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs:475

The direct path only chooses the condensed form from the pre-render threshold. If a full section below that threshold is itself too large for the remaining file space (for example, long failure names or a large coverage section), the writer refuses it and this project is dropped instead of retrying the one-line verdict. Recheck the rendered UTF-8 size in this locked callback and switch to BuildMinimalMarkdown before returning it.

                    var budget = SummaryBudget.ForProject(currentLength);
                    bool condense = budget.Stage is SummaryStage.Condensed or SummaryStage.Unlisted;
                    string markdown = condense
                        ? BuildMinimalMarkdown(snapshot, assemblyName, _targetFrameworkMoniker.Value, exitCode)
                        : BuildMarkdown(snapshot, assemblyName, _targetFrameworkMoniker.Value, exitCode, coverage, _sections, _includeFailureDetails, budget);

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/StepSummaryWriter.cs:732

This aggregate upsert stages a whole-file snapshot and replaces the destination without checking whether a non-MTP producer appended after ReadToEndAsync. Such producers do not honor this extension's lock file, so their block can be silently overwritten during the staging window. Capture the source length and recheck it immediately before replacement, retrying or aborting on a mismatch as the leading-notice rewrite does.

                    cancellationToken.ThrowIfCancellationRequested();
                    _fileSystem.ReplaceFile(tempPath, Path);

Hoisting or upgrading the leading notice stages a replacement and swaps it in, and the summary handle has to be released for that swap. Another writer can append in the gap. Once retries were exhausted the swap went ahead anyway against a stale snapshot, deleting whatever had landed. Fall back to a plain append instead: an understated notice is recoverable by the next writer, a deleted block is not. Also drops a duplicated comment block left behind by an earlier edit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa963c37-6214-46b9-9f7b-b084b293b544

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/InternalAPI/InternalAPI.Unshipped.txt:122

Neither of these old static methods exists after the writer/rendering refactor. Stale unshipped API declarations trigger RS0017; remove both entries rather than marking them *REMOVED*.

static Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.AppendStepSummaryWithRetryAsync(Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, string! path, string! content, int maxAttempts, System.TimeSpan retryDelay, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!
static Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.BuildAggregateMarkdown(Microsoft.Testing.Extensions.CiRunSummaryAggregate! aggregate) -> string!

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/InternalAPI/InternalAPI.Unshipped.txt:128

This static upsert helper was replaced by the instance method on StepSummaryWriter, so this declaration is now stale and will trigger RS0017. Remove the unshipped entry.

static Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.UpsertStepSummaryWithRetryAsync(Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, string! path, string! aggregationId, string! content, int maxAttempts, System.TimeSpan retryDelay, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/StepSummaryWriter.cs:723

The private lock only coordinates StepSummaryWriter instances. If another GITHUB_STEP_SUMMARY producer appends after ReadExistingSummary() but before this replacement, ReplaceSummary() overwrites that append. The leading-notice path already rechecks the original file length after staging; apply the same change-detection/retry pattern here before replacing the summary.


                    using (IFileStream tempStream = _fileSystem.NewFileStream(tempPath, FileMode.CreateNew, FileAccess.Write, FileShare.Read))

+    /// <summary>
+    /// What rendering a failure's diagnostics needs, without retaining the test node it came from.
+    /// </summary>
+    internal readonly struct PendingFailure


+Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.AggregateRenderResult.UnlistedModules.get -> int
 Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.DataTypesProduced.get -> System.Type![]!
-static Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.AppendStepSummaryWithRetryAsync(Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, string! path, string! content, int maxAttempts, System.TimeSpan retryDelay, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!
+Microsoft.Testing.Extensions.GitHubActionsReport.GitHubActionsSummaryReporter.GitHubActionsSummaryReporter(Microsoft.Testing.Platform.CommandLine.ICommandLineOptions! commandLineOptions, Microsoft.Testing.Platform.Configurations.IConfiguration! configuration, Microsoft.Testing.Platform.Helpers.IEnvironment! environment, Microsoft.Testing.Platform.Helpers.IFileSystem! fileSystem, Microsoft.Testing.Platform.Messages.IMessageBus! messageBus, Microsoft.Testing.Platform.OutputDevice.IOutputDevice! outputDevice, Microsoft.Testing.Platform.Services.ITestApplicationModuleInfo! testApplicationModuleInfo, Microsoft.Testing.Platform.Services.ITestApplicationProcessExitCode! testApplicationProcessExitCode, Microsoft.Testing.Platform.Logging.ILoggerFactory! loggerFactory, System.Func<bool>! shouldDeferToArtifactPostProcessing) -> void


Copilot AI balanced review requested due to automatic review settings August 18, 2026 10:15

Copilot started reviewing on behalf of Azat Mukhametshin (azat-msft) August 18, 2026 10:21 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Azat Mukhametshin (azat-msft) mentioned this pull request Aug 18, 2026

Validation: high failure count leaves the summary size flat azat-msft/gh-report-validation#5

Closed

Copilot AI added 2 commits August 19, 2026 00:43

Azat Mukhametshin (azat-msft) mentioned this pull request Aug 18, 2026

Validation: 30 test projects sharing one job summary azat-msft/gh-report-validation#6

Closed

Copilot AI review requested due to automatic review settings August 18, 2026 23:24

Copilot started reviewing on behalf of Azat Mukhametshin (azat-msft) August 18, 2026 23:24 View session

Fix the malformed PACKAGE.md options table row

e31c4f6

An earlier edit dropped the newline between the new failure-details row and the slow-test-notices row, merging them into one seven-cell row that markdownlint rejected (MD056).

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 18, 2026 23:36

Copilot started reviewing on behalf of Azat Mukhametshin (azat-msft) August 18, 2026 23:37 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Comment thread src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs Outdated

Comment thread ...rm/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.Markdown.cs Outdated

Use an explicit type for the module array in the aggregate test

5db0ed2

IDE0008 is enforced as an error in CI. The type was not apparent from the right-hand side because it comes from a LINQ projection, unlike the other 'var' uses here which are all 'new T(...)'.

Copilot AI review requested due to automatic review settings August 18, 2026 23:54

Copilot started reviewing on behalf of Azat Mukhametshin (azat-msft) August 18, 2026 23:55 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Comment thread src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsFailureDetails.cs Outdated

Use Assert.HasCount for the clipped-row count

f548af3

MSTEST0037 is enforced as an error in CI, which builds MSTest.Analyzers from source; the analyzer package restored locally predates the rule, so the local build did not flag it.

Copilot started reviewing on behalf of Azat Mukhametshin (azat-msft) August 25, 2026 12:44 View session