Notify users when an agent finishes or needs permission

[ndisidore]

When an agent turn finishes, or stops to ask for permission, the person who started it now hears about it: a toast in their open Workshop tabs, or a push to each of their devices through the Cloudflare OS app if no tab picks it up within 3 seconds. Replaces #650 and #676; the first six commits are @deregtd's from #650, and the rest rework them based on review there.

How it works:

• When a turn ends, the overseer sends a UserNotification (completed / needs permission) to the initiator's User DO.
• The User DO offers it to every visible tab, and each shows a toast with "Open task". If no tab accepts within 3s, the DO signs a request to the central notification service for each registered device, and the service pushes through APNs.
• The native app gives the SPA a one-time registration id; the User DO exchanges it for that device's subscription and keeps one per device.
• docs/notifications.md has the full flows and what crosses the boundary.

agent turn ends: completed / needs permission
        |
        v
     User DO ---notify---> visible Workshop tabs ---> toast
        |
        | no tab accepted within 3s
        v
     signed POST /v1/deliveries, one per device ---> notification service ---> APNs ---> devices

device app --one-time id--> SPA --> User DO --signed POST /v1/subscriptions--> service
                                      (keeps device key -> subscription id)

Design decisions:

• Browser first, push as fallback. No buzzing your phone while you're at your desk.
• Backend signs with the install key, no proxy worker. Same footing as CF_AI_GATEWAY_API_TOKEN, and gadget code never sees backend env.
• Fixed templates only. Event type, ids, chat title (trimmed to 96 UTF-16 units) and a same-origin link leave the install. No chat content, no free text.
• Every device the service lets register gets push; this side stores any number. The service returns a stable, install-scoped device key with each subscription, so a phone that registers on every app launch replaces its own entry instead of piling up ids that would each get a copy.
• A subscription is dropped only when the service says it is dead (device_gone, invalid_subscription). Any other failure keeps it, so a signing misconfiguration can't wipe everyone's devices.
• Completion only for turns a person started, including one their approval resumed. Schedules, callbacks and spawned agents run unattended, but still notify when they need permission.
• A pending connection request or awaited action means "needs permission"; a denied or rejected one means "completed". This is read when the notification is sent, after the turn has finished persisting, so a decision made while the turn was ending counts.
• Every notification a tab accepts shows a toast. A chat in the URL isn't proof it's on screen (mobile hides it behind Preview), so "Open task" also switches a phone from Preview to the chat.
• Push is optional. Missing config means push off; browser notifications still work.

Push needs the internal deploy-service and notification-service change (MR !416): the deploy service injects NOTIFICATION_SERVICE_URL, CFOS_INSTALL_ID, CFOS_INSTALL_KEY_ID and the secret CFOS_INSTALL_PRIVATE_KEY into the backend, and POST /v1/subscriptions has to return deviceKey (64 lowercase hex, stable per installation and device) next to subscriptionId. Neither side has shipped, so no migration; they land together.

Open with !416: the service caps registration at 10 devices per Cloudflare user (MAX_DEVICES_PER_OWNER); an 11th device's app registration fails with "Device limit reached" before it reaches this install. That's central policy and nothing here changes it, but it conflicts with "any number of devices", so it needs a decision there.

Follow-ups (deferred): retry native registration after a transient failure, unregister on sign-out, and shared contract fixtures with the service.

Known limitation: in a shared workspace, when a collaborator approves an action, the completion goes to the approver, not the person who started the task.

---

[github-actions]

Preview: pr687-feat-notifications

https://pr687-feat-notifications-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[devin-ai-integration]

Devin Review found 4 potential issues.

[devin-ai-integration]

🟡 Resumed turns send false completion alerts

When an approval resumes an agent, initiator.type is user, so notification announces another completed task. #resumeSuspendedAgent starts that continuation with the approver's profile, not a new user request.

Learn more

Agent turns can begin from a new human request or from a permission decision on an existing task. The notification gate uses the author type alone to decide whether to announce a completed task. #resumeSuspendedAgent starts permission-decision continuations with the approving user's profile, so they pass that gate. A completion alert then appears for a continuation the recipient did not initiate as a fresh task.

Example: Alice requests a connection, then Bob accepts it. Bob's acceptance resumes the agent with Bob as its initiator; when that turn finishes, Bob receives a completed-task alert even though his action was only an approval.

Recommended fix: Carry a separate user-request-versus-continuation signal through startAgent and its persistent ActiveAgentRecord, and gate completion notifications on that signal. Preserve permission alerts for unattended continuations.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[deregtd]

This also breaks the PR’s stated recipient ownership in a collaborative workspace: the original requester gets the permission notification, but if another collaborator approves, the continuation is attributed to the approver and its completion goes there instead. The persisted state should distinguish a fresh user request from a continuation and retain the task’s notification recipient across the suspension.

[ask-bonk]

Review: 1 findings.

Posted 1 actionable inline comment.

github run

[ask-bonk]

Review: 1 findings.

Posted 1 additional actionable inline comment.

github run

[deregtd]

Two lower-priority hardening/documentation follow-ups from my review:

• Validate the incoming deviceRegistrationId against its expected 64-lowercase-hex contract before creating a signed central-service request, mirroring the validation already applied to returned service IDs.
• Expand the deployment contract in docs/notifications.md to describe signing-key generation, encrypted custody, rotation, and injection boundaries. “The trusted deploy service injects it” does not yet establish where the private key exists or which systems/operators can recover it.

[ask-bonk]

Review: 0 findings.

No additional actionable findings beyond the existing review threads. The current changes address the earlier browser-navigation and pending-decision race findings.

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	90% → 100%	+10 pp	p = 1.00	86% → 87% +1 pp	3.8 → 3.4	25.3 → 27.3
chess	100% → 90%	−10 pp	p = 1.00	97% 0 pp	8.8 → 7.7	59.4 → 51.1
incident-desk	100% → 90%	−10 pp	p = 1.00	95% → 94% −1 pp	4.7 → 3.5	36.6 → 26.8
worker-logs	100% → 100% (1 run error)	candidate run errors	—	94%	4.1 → 3.8	23.6 → 22.2

Failed checks

Task	Check	Failed
change-calendar	t1 schedules-and-lists-windows	1/10 → 0/10
change-calendar	t1 rejects-invalid-windows-without-changing-anything	1/10 → 0/10
change-calendar	t1 overlap-is-per-service-and-touching-is-allowed	1/10 → 0/10
chess	t1 starts-from-the-standard-position	0/10 → 1/10
chess	t1 agrees-with-the-oracle-on-the-hard-positions	0/10 → 1/10
chess	t1 agrees-with-the-oracle-on-perft-positions	0/10 → 1/10
chess	t1 agrees-with-the-oracle-through-random-games	0/10 → 1/10
chess	t1 the-game-is-shared-across-connections	0/10 → 1/10
incident-desk	t1 opens-acknowledges-and-resolves-in-order	0/10 → 1/10
incident-desk	t1 simultaneous-acknowledges-yield-exactly-one-owner	0/10 → 1/10
incident-desk	t1 simultaneous-opens-of-one-id-admit-exactly-one	0/10 → 1/10
incident-desk	t1 board-lists-by-severity-then-age	0/10 → 1/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Pass-rate changes stayed within noise: change-calendar went 90% → 100%, while chess and incident-desk went 100% → 90%; worker-logs passed every completed run but had one candidate infrastructure error, preventing comparison. Mean costs were approximately $0.0280 → $0.0286, $0.0552 → $0.0475 and $0.0288 → $0.0217 respectively, but run costs overlap and neither cache hits nor cache breaks changed significantly. Chess remained the largest step consumer at 59.4 → 51.1 model steps; untested generated code cost the candidate two runs, while malformed edits and repeated ambiguous replacements wasted steps on both sides.

🟡 VERDICT: INCONCLUSIVE

The scored failures are unrelated generated-code mistakes, and the diff leaves agent prompts, tools and evals unchanged, but the candidate-only verification disconnect lacks enough diagnostic evidence to establish whether the new turn-end notification path contributed.

• Verification connection loss · caused by this PR: unclear — worker-logs lost its connection after a successful agent smoke test; the artifacts do not identify the disconnect’s source.

Triage

Failure modes

• Generated gadget cannot start · chess 1/10, incident-desk 1/10 · model error · this PR: no — in turn 1, chess trial 9’s writeFile(server.js) exported the string START_FEN, which Worker startup rejected; incident-desk trial 4’s writeFile(client.js) put unescaped double quotes inside a double-quoted HTML string. Both agents announced completion without executing a smoke test. Main passed these tasks, and a main chess run caught and repaired another startup syntax error through executeCode.
• Verification interrupted with partially loaded data · worker-logs 1/10 · harness bug · this PR: unclear — trial 6’s turn-1 executeCode successfully exercised ingestion, summaries and hourly buckets, then removed its test events. Verification subsequently reported WebSocket connection failed, incomplete dataset counts and a dropped Workshop connection; comparison.json classifies this as an infrastructure error, not failed task checks. Main had no such interruption.

Tool errors

• editFile: Validation failed · change-calendar 2 → 4, chess 9 → 3, incident-desk 5 → 4, worker-logs 1 → 2 · model error — agents omitted required fields or emitted malformed arguments; candidate calendar trial 5 supplied replacement text as a property name, while chess trial 7 omitted filename.
• editFile: No matching text was found · change-calendar 0 → 3, chess 10 → 5, incident-desk 1 → 1, worker-logs 1 → 2 · model error — replacements targeted text that differed from the file, including double-escaped regexes. Candidate calendar trial 4 repeated the same failed match after reading the file; worker-logs trial 6 corrected its escaping and recovered.
• editFile: Multiple matches were found · change-calendar 0 → 0, chess 2 → 3, incident-desk 3 → 6, worker-logs 1 → 1 · model error — agents supplied non-unique snippets despite the tool’s explicit uniqueness requirement. Candidate chess trial 7 retried the same ambiguous snippet three times before adding surrounding context.
• readFile: File does not exist · change-calendar 0 → 4, worker-logs 0 → 4 · model error — agents read server.js and client.js immediately after creating empty gadgets, although the system prompt explicitly says new gadgets have no files; they recovered by writing them.
• executeCode: Failed to start Worker · change-calendar 1 → 0 · model error — main trial 6, turn 2 generated a malformed nested template expression for the maintenance document. The next call rewrote it and succeeded.

What to do

• Re-run packages/workshop-evals/evals/worker-logs.eval.ts with connection-close and backend logs captured, correlating the disconnect with src/overseer.ts’s notification publication before changing that path.
• Optional: add a completion-time smoke-test instruction in packages/workshop-backend/src/agent.ts, requiring agents to exercise a gadget RPC after their final edits and fix startup errors before claiming completion. The generated-code and tool mistakes do not require a notification-PR fix.

github run