Skip to content

Fix existing WebView automation, errors, and screenshots - #660

Open
mattleibow wants to merge 1 commit into
mainfrom
mattleibow-webview-automation-fixes
Open

mattleibow wants to merge 1 commit into
mainfrom
mattleibow-webview-automation-fixes

Conversation

@mattleibow

Copy link
Copy Markdown
Member

Scope

First of five approved replacement implementation layers above #652. Parent: b404eb2247ea2ad10272e7a8088bdca9b5338ab5; implementation source inspected at the pinned #655 head 889c5039ba09e50a167dcf052402655a4d424fd3. This PR owns only existing WebView automation correctness, its focused regressions, and related documentation. The existing native stack and #655 remain untouched; final restacking/supersession is coordinator-owned.

  • Serialize typed action results in the original evaluation so click/fill mutations execute once. Preserve structured results and report CDP/JavaScript failures explicitly. For stack-only WebKit exceptions, read the original exception object's message with Runtime.getProperties, without replaying the expression.
  • Propagate HTTP/CDP/JavaScript failures through portable AgentClient, CLI, and MCP. CLI query/query-all/snapshot forward the selected C# WebView argument to the request, never an undefined JavaScript variable. Click/fill/text use the existing action endpoints. Batch fails the process when any command fails, including --continue-on-error, and stops immediately by default.
  • Add AgentClient.GetWebViewScreenshotAsync over the existing native-first screenshot endpoint; use it in CLI/MCP and verify the PNG signature.
  • Return honest 501 webview.network for unsupported browser fetch/XHR capture and remove the advertised WebView network feature. Native .NET HTTP capture endpoints are unchanged.

Breaking behavior

AgentClient.SendCdpCommandAsync and typed WebView action methods now throw with HTTP/CDP/JavaScript details instead of returning error payloads or false. /api/v1/webview/network changes from a misleading HTTP 200 native-store alias to 501 unsupported. Existing signatures, namespaces/type forwards, selectors, --webview/-w, webview/contextId, readiness aliases, and Blazor registration APIs are preserved.

Issues and reproduction evidence

Fixes #621: focused process regressions cover the undefined-variable generation, selected-context forwarding, action errors, useful JavaScript exceptions, nonzero exits, failure-fast stopping, and aggregate batch failure. On the existing Mac Catalyst Blazor sample, a single nonverbose CLI process completed 30 commands: 10 evaluations, query/query-all, HTML source, snapshot, fill/readback (unit1-value), insert/readback (unit1-value-typed), actual Add-button mutation (3 → 4 items with the expected title), deletion restoring three items, checkbox toggle (0 → 1 → 0 completed), and a valid PNG. A subsequent ReferenceError exited 1 and did not execute its sentinel. A separate single-process missing-selector click returned useful HTTP 404 details, exited 1, and did not execute its sentinel. The existing multi-Blazor test verifies left/right fill isolation. The Windows cause is source-proven and covered by platform-independent regressions; Windows runtime was not exercised. maui_query_css remains a native-tree query, not a browser DOM tool.

Related #232, not Fixes: repeated nonverbose evaluation and snapshot work on the tested run, but bridge readiness, wire-ID correlation, and broader timing reproduction belong to later layers.

Related #407, not Fixes: screenshot passed in the full 20-test WebView class, and CLI PNG verification checked signature, chunk CRCs, IEND, and IDAT decompression (374,398 bytes; 1250×1366). This is not proof that the existing in-suite flake is resolved. One later repeat on the long-lived app failed two readiness checks after a native WebKit SIGTRAP in WebKit::ProcessThrottler::sendPrepareToSuspendIPC; a fresh owned app rerun passed 20/20. No native crash/bridge lifetime fix is claimed here.

Related #654: only this layer's browser-network capability/spec correction and automation/error documentation. The broad tracker remains open. No claim of #257 onboarding fixes or #113's four new interaction tools.

Validation

  • Focused agent/action/navigation/protocol: 24 passed.
  • Portable Client suite: 77 passed; netstandard2.0 build succeeded with zero warnings/errors.
  • Focused CLI/WebView/MCP regressions: 23 passed.
  • Existing sample Mac Catalyst build succeeded; final fresh-app full WebViewTests class: 20 passed.
  • Real stdio MCP returned HTML and PNG and preserved ReferenceError: Can't find variable: __unit1McpMissing in an isError result.
  • Native MAUI CLI query/fill/property readback/clear remained functional in a seven-command single process.

SDK 10.0.401; per-process Xcode 26.6 selection, no global Xcode or sample-project edits. A cross-process handoff initially encountered the already-known mutation lease (#620); validation waited for expiry instead of changing lease behavior. Owned app/helpers stopped; shared broker/unrelated apps untouched. No dependencies changed. No full-suite-green, Windows, GTK/WPF, physical mobile-device, relative browser-navigation, or flake-resolution claim.

Intentionally deferred

Generic WebView package/assets/platform adapters and atomic owner/registry/Blazor/GTK/shipping migration; canonical context-only CLI/MCP/Inspector contracts and ready-only fields; generic DOM layout diagnostics; maintained standard/Hybrid/mixed sample and new cross-host matrix. Parent schema corrections outside the WebView network block remain intact.

@mattleibow mattleibow added the breaking-change This PR contains breaking changes label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

✅ Expert Code Review: 2 independent reviewers found no issues. Methodology: 2-model adversarial consensus. (1 reviewer failed to complete — reduced coverage, no tiebreaker)

Generated by Expert Code Review (auto) for #660 · copilot · gpt56 · 251 AIC · ⌖ 21.6 AIC · ⊞ 37.6K · ◷

Execute typed WebView evaluations once, propagate browser errors, forward selected contexts, and use native-first PNG screenshots. Report browser-network capture as unsupported without changing native HTTP capture.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dbcaf214-7d31-463c-a36a-e8e6aaada82b
@mattleibow
mattleibow force-pushed the mattleibow-webview-automation-fixes branch from 3fed69e to b2a420e Compare October 9, 2026 19:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change This PR contains breaking changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant