Drive VM Chrome with opencua - #51
Open
aleksei-okatiev wants to merge 18 commits into
Open
Conversation
aleksei-okatiev
added a commit
that referenced
this pull request
Sep 7, 2026
Code review of PR #51 surfaced several correctness gaps: - socketio-client.ts: gate the get_desktop_thumbnail interception on serverName too, not just tool name, so a same-named tool on another configured server isn't silently answered from the VM framebuffer; wrap capture() in try/catch so a throw doesn't hang the caller; reject non-positive max_width instead of coercing 0/negative; add the isError field the wire contract expects. - vnc-frame-source.ts: stop falling back to "any running VM" when the configured ssh target names a host that isn't found running, since that can silently serve a different VM's screen; recheck `stopped` after the discover() await so a stop() during discovery doesn't resurrect a connection; retry discovery when no VM is found yet instead of giving up until a full service restart; downscale by the longest edge as documented, not just width. - mcp-client.ts: don't spread the full process.env into a child MCP server just because conf.env is a (possibly empty) object. - ConfigPage.svelte: disable Install when sudo is required but no password was entered, instead of silently running unelevated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aleksei-okatiev
changed the title
aleksei-okatiev
changed the title
Netflix's cloud games (e.g. title 81677232) will not start under Playwright. The click lands, the page acknowledges it, and the game never launches — the input layer is the variable, not the gesture shape. Four combinations were tried (plain click, human-like pointer path, bringToFront, fullscreen dedup) and none worked. DRM, an attached debugger, bot detection and a fullscreen race were each chased and disproved along the way. What does work is a real Chrome driven by real OS-level input with no debugger attached anywhere. MacOS_VM is a cua-driver MCP server reached over SSH into a Lume-managed macOS VM, configured like any other stdio server, so nothing in the transport layer needed changing. docs/vm-native-input.md has the setup and the traps. The Netflix recipe is: one tab only (a second fails with web-cg-5003, which presents as a screen that never advances), pass the profile gate, then click "Resume game" — never navigate to /play-game/<id> directly, as it carries no user activation and drops to a "Start game" screen that stalls. docs/vm-agent-allowlist.md covers exposing the tools to an agent, including why coordinates need care: macOS in the VM always runs a 2x backing scale, so screenshots are 2560x1600 for a 1280x800 point desktop, and no lume resolution avoids it. Also fixes serializeToolResult (mcp-client.ts) to mirror a tool's structuredContent into readable text: cua-driver's list_windows sends its payload only there, so without this an agent driving the VM could see a window count but never the windows themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aleksei-okatiev
force-pushed
the
feat/vm-native-input-and-browser-install
branch
from
September 7, 2026 15:40
e244c71 to
5f674c8
Compare
…scrub product-specific content Neither doc previously said how cua-driver itself gets onto the VM — they started from "once cua-driver is installed and running" without covering that step. Verified directly against the live VM rather than guessed: - docs/macos-vm.mcp-config.json: the exact MacOS_VM server entry for testinator-connect's config.json (ssh + BatchMode=yes + cua-driver mcp), with the IP templated out — pulled from a real, working config. - docs/vm-native-input.md: new step 5 covering the install command (curl -fsSL https://cua.ai/driver/install.sh | bash, confirmed via the tool's own check-update output), granting Accessibility/Screen Recording via `cua-driver permissions grant` over VNC (a GUI session is required — SSH cannot satisfy the TCC prompts), verifying with `doctor`, and copying an SSH key so the BatchMode invocation doesn't need a password. Reordered so the pre-existing VNC bootstrapping section comes first, since the new step depends on it for a GUI session. Also scrubbed both docs of the specific site/game that originally motivated this feature — VM name, example URLs, and a walkthrough tied to one site's UI flow — none of which is needed to explain the VM or cua-driver setup generically. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not needed for testinator-connect's own setup — it was config/system-prompt guidance for a different repo's browser_tool_selector, not anything this repo consumes or enforces.
Renamed serializeStructuredContent to mirrorStructuredContentIntoTextBlock and its variables (structuredContentJson, jsonAlreadyInContent) so the function reads without the prose that used to carry that meaning.
vncdotool was only ever a manual bootstrapping tool from before cua-driver existed — the doc kept documenting it (a whole step with four traps) even though the final, recommended path never touches it: Chrome installs over plain SSH, and the one-time TCC permission grant is done interactively through Screen Sharing, no scripting needed. Removed: - the vncdotool step and its "four traps" (per-gesture VNC sessions, shift handling, no clipboard paste, wait syntax) - the vncenv/pip prerequisite - the Coordinates section, which was entirely about verifying vncdotool clicks by capture-and-compare Also renamed vm-native-input.md -> vm-opencua.md and reworded the title/ intro to name OpenCUA (github.com/trycua/cua), the project cua-driver belongs to, instead of the generic "native input" framing.
Replaced the motivating-case paragraph and other after-the-fact framing with a plain purpose statement, so this reads as setup instructions for a clean machine rather than a record of how the feature was built.
Gives the macOS VM its own egress on the host's physical uplink instead of a full-tunnel VPN (e.g. GlobalProtect), which otherwise breaks the VM's DNS and makes its traffic look like it's coming from the VPN's network instead of the host's own. Verified against a live setup today: proxy up, tunnel up, VM and host egress on their expected, separate paths. - docs/vm-novpn/README.md: architecture, setup, verification, teardown, and a supervisor for surviving reboots/restarts. Scrubbed of the specific product/site this was built against and of machine-identifying details (real IPs, username, absolute paths) in favor of placeholders and the same example VM name/IP used in vm-opencua.md. - docs/vm-novpn/vm_socks.py: the SOCKS5 proxy that pins outbound sockets to the physical interface via IP_BOUND_IF. - docs/vm-novpn/lume-vm-novpn.sh: the supervisor script the doc's "day to day" section references — previously undocumented how to obtain it, so a fresh reader had no way to follow that section. Locates vm_socks.py next to itself rather than at a hardcoded path, so the pair works whether run from the repo checkout or copied to ~/.local/bin. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…up before use Checked whether this doc actually works for someone setting up a lume VM fresh, not just on this already-configured machine: it did not. The reverse tunnel (and the supervisor script) use `ssh -o BatchMode=yes`, which refuses a password prompt — unlike `lume ssh`, used elsewhere in the doc, which authenticates with lume/lume automatically. A fresh VM has no authorized key yet, so that step would silently fail with nothing to explain why. Also moved the "find your VM's IP / physical interface" lookups before the commands that use them, instead of after.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cua-driver, from OpenCUA, via SSH) for pages that block Playwright automation — seedocs/vm-opencua.mdfor setup: VM creation, Chrome/Widevine install, and installing and connecting cua-driver.docs/macos-vm.mcp-config.json: a copy-pasteableMacOS_VMserver entry for testinator-connect'sconfig.json, pulled from a real working config.docs/vm-novpn/: how to give the VM its own internet egress on the host's physical uplink instead of a full-tunnel corporate VPN (e.g. GlobalProtect), which otherwise breaks the VM's DNS and makes its traffic look like it's coming from the VPN's network. Includes the SOCKS5 proxy script and a supervisor script for surviving host reboots / VM restarts.