Skip to content

Implement the FaceTime helper's real Accessibility traversal so probe/answer/hangup work #5388

Description

@atomantic

Problem

The FaceTime Audio epic (#5306) shipped all four phases — control plane (#5307), audio bridge (#5308), the Mind voice.call-user capability (#5309), and inbound answer (#5310) — against a native helper that has never actually driven the FaceTime UI. server/native/facetime-ax/main.swift is 53 lines and fails closed on every command that needs AX traversal:

  • probe → unconditionally returns state: "idle", regardless of what FaceTime is doing.
  • call → opens facetime-audio://<handle> via NSWorkspace, then returns ok: false / ambiguous-surface.
  • answer, hangup → fall straight through to the terminal emit(... "No unambiguous FaceTime surface naming \(identity) was found", errorCode: "ambiguous-surface", exit: 1).

That was correct for phase 1 — the comment in the file says so explicitly ("This phase deliberately fails closed. Future phases provide the verified AX element traversal; no coordinate click is ever acceptable here."). But every phase since has been built and tested against that stub, so the feature cannot work end to end on any install today:

  • callSession.pollCall() reads state === 'idle' as remote-hangup, so a placed call ends on the first 2-second poll even if FaceTime connected.
  • The phase-4 incoming watcher can never observe a ring, so autoAnswer is inert.
  • The Mind's voice.call-user grant can pass every gate and still reach a dial that reports failure.

The JavaScript side is complete, tested, and fail-closed by design — nothing here is a defect in it. The gap is that the one component that has to touch the real UI was never written.

What to do

Implement the verified AX element traversal in server/native/facetime-ax/main.swift, keeping the existing strict JSON contract (facetimeControlResultSchema in server/services/voice/facetimeBridge.js) unchanged so no JS changes are needed:

  • probe — report the real state (idle / dialing / connected / ended) by inspecting the FaceTime window and Notification Center, plus whether an incoming call is ringing and whether its caller matches the configured identity. Phase 4's watcher needs the incoming signal; today it has no way to see one.
  • answer — press only the Notification Center action whose accessibility label names the configured identity. Never answer, decline, or touch any other caller. Ambiguity fails closed.
  • hangup — same rule against the active call surface.
  • Semantic label matching only. No coordinate clicks, ever — this is stated in both the epic and the existing file comment.
  • An unauthorized caller must produce no output naming them[repo-study-facetime-bridge-inbound-answer] FaceTime Audio phase 4: answer the user's own call and run a Mind conversation over it #5310 has a test asserting zero log output on that path; keep the native side equally silent.

Constraints

  • Accessibility permission is already checked (AXIsProcessTrusted()); keep that gate.
  • Compiled by npm run setup:facetime via swiftc; macOS-only.
  • Real-device validation is required and cannot be done headlessly — this needs a Mac with FaceTime, a second device to call from, and Accessibility granted to the compiled helper. That is why it was deferred rather than guessed at.

Verification

Manual, on a real Mac, with BlackHole installed and the call host attached:

  1. probe reports idle with no call, dialing while ringing out, connected once picked up.
  2. A Test call from Settings → Voice connects and stays connected past the first poll.
  3. With autoAnswer on, a call from the configured handle is answered within 5 s and the caller hears the greeting.
  4. A call from any other handle is left ringing, is never answered, and produces no log line naming it.
  5. Hang up from the Mind tab ends the call.

Part of #5306.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:voiceVoice stack: STT/TTS pipeline, proactive speech, voice tools, call bridgeeffort:highDispatch effort: highmodel:heavyDispatch capability: heavyplanTracked by /do:replan

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions