Repository navigation
[Testing] Fixed Test case failure in PR 38782- Candidate - 1 - #38843
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
Ahamed-Ali
added
community ✨
This comment has been minimized.
This comment has been minimized.
github-actions
Bot
added
the
s/agent-review-in-progress
|
/azp run |
|
Note 🔍
|
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
MauiBot
added
s/agent-changes-requested
This comment has been minimized.
This comment has been minimized.
|
/azp run |
github-actions
Bot
added
the
s/agent-review-in-progress
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
|
Note 🔍
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
MauiBot
removed
the
s/agent-review-in-progress
This comment has been minimized.
This comment has been minimized.
|
/azp run |
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
github-actions
Bot
added
the
s/agent-review-in-progress
|
Note 🔍
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
MauiBot
added
s/agent-approved
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
/azp run maui-pr-uitests , maui-pr-devicetests |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
This comment has been minimized.
This comment has been minimized.
github-actions
Bot
added
the
s/agent-review-in-progress
|
Note 🔍
|
MauiBot
left a comment
MauiBot
left a comment
There was a problem hiding this comment.
AI Review Summary
@NafeelaNazhir — new AI review results are available based on commit
7d76bff.
🗂️ Review Sessions — click to expand
🚦 Gate — Test Before & After Fix
Gate Result: ⚠️ SKIPPED
No tests were detected in this PR.
Recommendation: Add tests to verify the fix using the write-tests-agent.
📋 Pre-Flight — Context & Validation
Pre-Flight Context
Scope: Context gathering only for draft PR #38843 at head 7d76bff03375f8e4ac1e78812484f615b7d7dfc0, targeting inflight/candidate. This is not an exhaustive code review or a recommendation.
Reported problem
The author reports that the .NET 10 SR12 candidate PR #38782 exposed cross-platform device/UI test failures caused by expected rendering no longer matching existing screenshot baselines. #38782 is a large candidate snapshot, so the underlying product changes responsible for each rendering difference were not isolated by this PR.
A prior CI analysis for an earlier #38843 commit (6381ac8) specifically attributed seven Windows TitleBar screenshot failures to this PR’s removal of vertical margins. That analysis also reported unrelated or unresolved device-test crashes and other UI mismatches. Subsequent commits resaved images, including the Windows baselines, but validation of the current head was still pending when queried.
Submitted changes and mechanism
GitHub reports 41 changed files: 39 PNG baselines and two C# files, with no shipping framework source changed.
ScrollViewTests.iOS.cs: modifies the existingContentSizeExpandsToViewportdevice test. WithSafeAreaEdges.Container, it now compares the nativeUIScrollView.ContentSizedirectly againstAdjustedContentInset.InsetRect(Bounds).Size. Previously it manually insetContentSizeusingSafeAreaInsetsbefore comparison. Height and width assertions remain.TitleBarViewModel.cs: changes the test-host leadingImageand trailingImageButtonmargins from uniform10toThickness(10, 0), retaining horizontal margins while removing top and bottom margins. This deliberately changes rendered TitleBar test output.- Screenshot baselines:
- Android, 6 files: one
Issue37440...baseline is moved from genericandroidtoandroid-notch-36; existing RTL CollectionView, max-lines Label, and two ScrollView gradient baselines are refreshed; aLabelsShouldNotBeTruncatedAfterItemsSourceIsReplacedbaseline is added. - Mac, 6 files: refreshes ItemsWrapGrid, NoScrollbars, Shadows, and two CollectionView preselection baselines; adds the replaced-items-source Label baseline.
- Windows, 15 files: refreshes DataTrigger, seven TitleBar variants, five BindableLayout variants, and one CarouselView baseline; adds the replaced-items-source Label baseline.
- iOS, 12 files: the same six baselines are added/refreshed in both
iosandios-26: replaced-items-source Label, NoScrollbars, ScrollView bounds, Shadows, and two CollectionView preselection cases.
- Android, 6 files: one
The binary patch records only that the PNGs differ; it does not expose the pixel-level changes or establish that each new image represents intended output.
Platform-sensitive paths
ScrollViewTests.iOS.csuses UIKit and, by repository suffix convention, is relevant to both iOS and Mac Catalyst builds.- Its changed test depends on nonzero safe-area/adjusted content insets, including system bars, notch, or home-indicator conditions.
ScrollViewShouldRenderWithinBoundsis declared only for Android and iOS; this PR refreshes its generic and iOS 26 baselines, but no Android baseline for that test.- TitleBar screenshot tests compile on Windows and Mac Catalyst. The margin change affects shared host-app construction, while this PR updates seven Windows TitleBar baselines and no Mac TitleBar baselines. Whether Mac rendering is unaffected or requires corresponding baselines is not established.
- Android baselines distinguish generic rendering from the
android-notch-36profile; movingIssue37440...changes which environment supplies that expectation. - iOS maintains both generic and
ios-26expectations, so both image sets require independent validation. - Mac snapshots are under
TestCases.Mac.Tests; Windows snapshots are underTestCases.WinUI.Tests.
Validation requirements
The supplied Gate result is skipped because no tests were detected. It must not be interpreted as evidence that tests passed: the PR modifies an existing device-test assertion and many UI-test expectations, but adds no new test method.
For the requested Windows validation lane, the current head needs the Windows UI screenshot suite covering the 15 changed baselines, especially the seven TitleBar cases affected directly by the margin change. Generated images should be compared with the submitted baselines to distinguish intentional layout changes from incidental font, scaling, antialiasing, device, or timing differences.
Complete cross-platform validation additionally requires:
- Android runs for the generic and
android-notch-36baseline profiles. - iOS runs for generic and iOS 26 baselines, plus
ContentSizeExpandsToViewporton a configuration where adjusted safe-area insets are meaningful. - Mac Catalyst verification for the six changed Mac baselines, the iOS-suffixed device test, and the shared TitleBar margin behavior.
- Confirmation that the newly added
LabelsShouldNotBeTruncatedAfterItemsSourceIsReplacedexpectations correspond to an existing executed test on every represented platform. - Evidence that each broad baseline refresh comes from the canonical CI configuration and reflects intended candidate output rather than environmental drift.
At the current head, GitHub’s combined status was pending. The latest comments show maui-pr-uitests and maui-pr-devicetests reruns started after the final image refresh, and a separate Windows expert-review build started; no completed result for those current-head runs was available in the retrieved context.
Remaining uncertainties
- The exact rendering cause and intended pixel difference for each of the 39 binary files remain unknown.
- The linked candidate PR contains hundreds of files; its underlying functional changes were not reviewed here, so attribution beyond the reported failures is unknown.
- Available code search did not locate the declaration of
LabelsShouldNotBeTruncatedAfterItemsSourceIsReplacedon the indexed branches; this is unavailable context, not evidence that the test is absent. - It is unknown whether the shared TitleBar margin change intentionally applies to Mac Catalyst without Mac TitleBar baseline updates.
- The current
inflight/candidatehead reported by #38782 differs from #38843’s recorded base SHA, so final behavior after integrating with the latest candidate state is not established. - Current-head rerun outcomes, pixel-diff artifacts, device configurations, and failure logs were not yet available.
🔬 Code Review — Deep Analysis
Code Review — PR #38843
Independent Assessment
What this changes: The submitted change updates test infrastructure rather than shipping framework code: it refreshes 39 cross-platform screenshot baselines, removes vertical margins from the shared TitleBar test-host leading/trailing images, and changes the iOS/Mac Catalyst ScrollView safe-area assertion to compare native content size with the adjusted viewport.
Inferred motivation: The candidate branch changed rendered output and UIKit safe-area behavior, so the PR aligns existing visual and device-test expectations with that candidate output.
Reconciliation with PR Narrative
Author claims: The PR addresses failures exposed by #38782 by refreshing expected images across platforms and correcting the iOS ScrollView viewport comparison.
Agreement/disagreement: The submitted diff supports those claims. The expert review found no concrete line-level correctness defect, but binary baseline replacement alone cannot establish that every pixel change is intentional.
Prior Review Reconciliation
No unresolved prior error-level finding was validated against the submitted head. The earlier Windows TitleBar failures described in the trusted context predated the final baseline refresh.
Blast Radius Assessment
- Runs for all instances: The C# changes run only in test code, but the shared TitleBar host change affects every corresponding Windows and Mac Catalyst TitleBar screenshot case.
- Startup impact: No shipping application startup impact.
- Static/shared state: No new static or shared state.
- Platform scope:
ScrollViewTests.iOS.cscompiles for both iOS and Mac Catalyst; the baseline changes span Android, iOS, Mac Catalyst, and Windows.
CI Status
- Required-check result: Undetermined for the submitted head; the trusted context recorded current-head reruns as pending.
- Gate result: SKIPPED — no tests detected.
- Classification: Neither the pending checks nor the skipped Gate provide passing validation evidence.
- Action taken: Confidence is capped at low. Gate verification was not rerun, as required by the caller.
Findings
The expert reviewer wrote a validated empty JSON array to inline-findings.json; no concrete actionable source defect survived its dimension review.
The remaining concern is evidentiary rather than a line-level defect: 39 binary expectations cannot be judged for fidelity from the Git patch, and the supplied Gate did not execute the affected screenshot/device tests.
Failure-Mode Probing
- Shared TitleBar host change on Mac Catalyst without corresponding Mac TitleBar baseline updates: no failure was established, but the supplied evidence does not show that the Mac cases ran or remained unchanged.
- iOS/Mac Catalyst adjusted-inset assertion with nonzero safe-area insets: the new comparison is coherent with the adjusted viewport, but no completed current-head device-test result was supplied.
- Broad screenshot refresh under a different scale, font, antialiasing, device, or timing environment: the patch cannot distinguish environmental drift from intended rendering; canonical UI-test artifacts are required.
External Output Contract
Not applicable.
Verdict: NEEDS_DISCUSSION
Confidence: low
Summary: The one permitted expert pass found no blocking code defect and produced no inline findings. Approval is nevertheless unsupported because the Gate was skipped and the submitted binary baselines and platform-sensitive test changes lack completed, isolated validation evidence.
Refinement Threshold
Not met. There is no concrete actionable expert finding or source/test-backed unresolved mechanism to justify a pr-plus-reviewer patch.
🛠️ Try-Fix — Analysis & Comparison
Alternative generation: not requested
Evidence-first mode intentionally omits routine Try-Fix attempts. This is not a failed
attempt, a completed alternative search, or evidence that alternatives have no value.
The expert review may still produce one evidence-backed PR refinement.
📝 PR Finalize — Recommended Title & Description
Assessment: ✏️ Recommend updating — the current title is vague and the description omits the shared TitleBar margin change and the precise cross-platform baseline scope.
Recommended title
[All Platforms] UI tests: Refresh candidate baselines and iOS ScrollView assertion
Recommended description
### Description of Change
This PR addresses test failures exposed by the inflight candidate PR https://github.com/dotnet/maui/pull/38782 and aligns the affected test expectations with the candidate output.
The submitted changes:
- Add or refresh 39 expected screenshot baseline files across Android, iOS, Mac Catalyst, and Windows.
- Move `Issue37440EmptyAutoSizeEditorDoesNotSnapToMaximumHeight` from the generic Android baseline set to the `android-notch-36` profile.
- Update the shared TitleBar test-host leading `Image` and trailing `ImageButton` margins from uniform `10` to `Thickness(10, 0)`, retaining horizontal spacing while removing vertical spacing.
- Update the iOS/Mac Catalyst `ContentSizeExpandsToViewport` ScrollView device test to compare `ContentSize` directly with the viewport after UIKit applies `AdjustedContentInset`.
#### Updated image cases
- DataTrigger_ButtonDisabled
- Issue37440EmptyAutoSizeEditorDoesNotSnapToMaximumHeight
- ItemsWrapGridWithDefaultWidth
- LabelNotTruncatedWithMaxLines
- LabelsShouldNotBeTruncatedAfterItemsSourceIsReplaced
- NoScrollbarsTest
- RtlCollectionViewShouldNotBeTruncatedAfterLandscapeRotation
- ScrollViewShouldRenderWithinBounds
- ShadowsDontRespectControlShape
- VerifyBindableLayoutWithBasicItemTemplate
- VerifyBindableLayoutWithGridItemTemplate
- VerifyBindableLayoutWithItemTemplateSelector
- VerifyBindableLayoutWithItemsSourceNone
- VerifyBindableLayoutWithItemsSourceObservableCollection
- VerifyCarouselViewKeepScrollOffsetAdd
- VerifyModelItemsObservableCollectionWhenMultipleModePreSelection
- VerifyModelItemsObservableCollectionWhenSingleModePreSelection
- ScrollViewGradientBackground
- ScrollViewWithoutGradientBackground
- TitleBar_Window
- TitleBar_Icon_WithTrailingContentAndLeadingContent
- TitleBar_TrailingContentAndLeadingContent_WithTitleAndSubtitle
- TitleBar_TrailingContentAndLeadingContent_WithSearchBar
- TitleBar_TrailingContentAndLeadingContent_WithGrid
- TitleBar_TrailingContentAndLeadingContent_WithHorizontalStackLayout
- TitleBar_TrailingContentAndLeadingContent_WithBackgroundColor
### Issues Fixed
Related candidate PR: https://github.com/dotnet/maui/pull/38782
🏁 Report — Final Recommendation
⚠️ Final Recommendation: REQUEST CHANGES
Phase Status
| Phase | Status | Notes |
|---|---|---|
| Pre-Flight | ✅ COMPLETE | Trusted read-only context consumed for submitted head 7d76bff03375f8e4ac1e78812484f615b7d7dfc0 |
| Code Review | NEEDS_DISCUSSION (low) | One expert pass completed; 0 validated inline findings, but required validation evidence is missing |
| Gate | No tests detected; this is not a passing result | |
| Try-Fix | NOT REQUESTED | Evidence-first mode intentionally omitted routine alternative generation |
| Refinement | NOT GENERATED | Expert evidence did not meet the threshold for a pr-plus-reviewer patch |
| Report | ✅ COMPLETE | Raw PR assessed as the sole real candidate |
Code Review and Candidate Comparison
| Candidate | Implementation | Expert Findings | Validation | Ranking |
|---|---|---|---|---|
pr |
Submitted test-only change: 39 screenshot baselines, shared TitleBar test-host margins, and iOS/Mac Catalyst ScrollView assertion | No concrete actionable defect survived expert validation | Gate skipped; current-head platform reruns and binary fidelity were not demonstrated in the supplied evidence | 1 |
Alternative generation: not requested. No try-fix-N candidates were created. No pr-plus-reviewer candidate was implemented because the expert review did not identify a concrete actionable defect or source/test-backed unresolved behavioral mechanism.
Selected Fix: PR — it is the only implemented candidate and the expert pass found no line-level defect. Selection does not establish merge readiness: the skipped Gate independently prevents approval, and the broad binary baseline refresh remains unvalidated.
Summary
Request changes pending isolated current-head validation of the affected Windows UI screenshot cases and the relevant cross-platform screenshot/device-test lanes. The submitted implementation is narrowly scoped to tests and has no validated inline code defect, but neither artifact presence nor a skipped Gate proves that the 39 replacement images are canonical or that the platform-sensitive assertion behaves correctly.
Root Cause
The candidate branch changed rendered output relative to existing UI-test expectations, and UIKit applies safe-area adjustments through AdjustedContentInset, making the prior ScrollView comparison stale. The submitted evidence does not isolate the product-side cause of each image delta, so attribution for the full baseline set remains uncertain.
Fix Quality
The source changes are focused: TitleBar test content retains horizontal margins while removing vertical margins, and the ScrollView test compares content size with the adjusted viewport. The expert reviewer produced a valid empty inline-findings.json. However, the fix changes expectations rather than proving the expected output; approval requires completed canonical runs for the affected Windows baselines and applicable Android, iOS, and Mac Catalyst paths, including meaningful safe-area coverage.
📱 UI Tests — Button,Label,Layout
Detected UI test categories: Button,Label,Layout
❌ Deep UI tests — 401 passed, 2 failed, 8 skipped across 3 categories on platform-pool agent (replaces in-process counts above).
🧪 UI Test Execution Results (deep, platform pool)
| Category | Tests | Snapshot diffs |
|---|---|---|
Button |
100/103 (2 ❌, 1 skipped) | 2 diff PNGs |
Label |
115/118 (3 skipped) ✓ | — |
Layout |
186/190 (4 skipped) ✓ | — |
🔍 AI analysis of failures — PR-related vs unrelated
🔍 AI-generated triage (GitHub Copilot CLI) — a heuristic judgement of whether each deep UI test failure is connected to this PR's changes. Verify before relying on it.
Likely unrelated: the failures appear pre-existing, flaky, or infrastructure.
- ● Unrelated — Windows Button visual snapshots (2 tests): both are baseline-image mismatches, while the PR changes neither Button rendering code nor these Windows Button baselines; its shared source edit is confined to TitleBar test content, and the iOS-only test change cannot affect Windows.
Strongest signal: the PR updates several Windows baselines, but neither failing Button baseline is among them.
❌ Button — 2 failed tests
VerifyButton_ImageWithShadow
VisualTestUtils.VisualTestFailedException :
Snapshot different than baseline: VerifyButton_ImageWithShadow.png (0.60% difference)
If the correct baseline has changed (this isn't a a bug), then update the baseline image.
See test attachment or download the build artifacts to get the new snapshot file.
More info: https://aka.ms/visual-test-workflow
at Microsoft.Maui.TestCases.Tests.UITest.VerifyScreenshot(String name, Nullable`1 retryDelay, Nullable`1 retryTimeout, Int32 cropLeft, Int32 cropRight, Int32 cropTop, Int32 cropBottom, Double tolerance, Boolean includeTitleBar) in /_/src/Controls/tests/TestCases.Shared.Tests/UITest.cs:line 299
at Microsoft.Maui.TestCases.Tests.ButtonFeatureTests.VerifyButton_ImageWithShadow() in /_/src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/ButtonFeatureTests.cs:line 684
at System.RuntimeMethodHandle.InvokeMethod(ObjectHandleOnStack target, Void** arguments, ObjectHandleOnStack sig, BOOL isConstructor, ObjectHandleOnStack resu
...
VerifyButton_ContentLayoutRightWithImage
VisualTestUtils.VisualTestFailedException :
Snapshot different than baseline: VerifyButton_ContentLayoutRightWithImage.png (1.76% difference)
If the correct baseline has changed (this isn't a a bug), then update the baseline image.
See test attachment or download the build artifacts to get the new snapshot file.
More info: https://aka.ms/visual-test-workflow
at Microsoft.Maui.TestCases.Tests.UITest.VerifyScreenshot(String name, Nullable`1 retryDelay, Nullable`1 retryTimeout, Int32 cropLeft, Int32 cropRight, Int32 cropTop, Int32 cropBottom, Double tolerance, Boolean includeTitleBar) in /_/src/Controls/tests/TestCases.Shared.Tests/UITest.cs:line 299
at Microsoft.Maui.TestCases.Tests.ButtonFeatureTests.VerifyButton_ContentLayoutRightWithImage() in /_/src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/ButtonFeatureTests.cs:line 564
at System.RuntimeMethodHandle.InvokeMethod(ObjectHandleOnStack target, Void** arguments, ObjectHandleOnStack sig, BOOL isConstructor,
...
📎 Download drop-deep-uitests artifact (TRX + snapshot diffs)
🧭 Next Steps — review latest findings
No alternative fix was selected for this run. Review the session findings and CI results before merging.
MauiBot
added
s/agent-changes-requested
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Tests Failure Analysis
🧪 CI Analysis — click to expand📊 maui-prNo failures found. 🧪 maui-pr-devicetests
🧪 maui-pr-uitests
🧭 Follow-up — actions and refreshNext action: Restore complete results and variant metadata; compare crash stacks and snapshots against equivalent target runs.
|
6 participants
Description
This PR addresses the test failures that occurred in the inflight/candidate branch: #38782, and includes updates to improve rendering and test stability across platforms.
This PR adds and updates the expected baseline images for all platforms so the tests validate the current intended output.
Fixes: #38782
It also updates the iOS ScrollView safe-area test to compare the content size with the viewport after UIKit applies the adjusted content inset.
Images