Add launchd watchdog for silent LaunchAgent failure detection

[ns408]

Problem

When a LaunchAgent fails silently (e.g., script path breaks after a repo rename), there's no notification — the job just stops running and nobody notices for days. Currently the only visibility is manually checking log files in ~/.local/log/.

Proposed Solution: Hybrid Approach

1. Wrapper script with failure notification (scripts/run-with-notify.sh)

• Wraps all scheduled jobs; on non-zero exit, sends a macOS notification via osascript
• Plist templates updated to run jobs through the wrapper
• Handles the common case: script ran but failed

2. Lightweight watchdog (scripts/launchd-watchdog.sh)

• Runs every 4 hours via its own LaunchAgent
• Checks if each monitored job's log file was modified in the last 36 hours
• Sends macOS notification for stale (never-ran) jobs
• Handles the edge case the wrapper can't: job never executed (broken path, plist unloaded)

Scope

• Monitor ns-bootstrap's own agents only (com.ns-bootstrap.update-daily, com.ns-bootstrap.update-interactive)
• Config file at ~/.config/ns-bootstrap/watchdog-jobs.conf allows adding labels later, but don't design a multi-project framework upfront
• macOS only initially; Ubuntu can defer since systemd has built-in OnFailure= support

Files to Add/Modify

scripts/
├── run-with-notify.sh                              # NEW: wrapper with failure notification
├── launchd-watchdog.sh                             # NEW: checks jobs ran recently
├── launchd/
│   ├── com.ns-bootstrap.update-daily.plist.template        # MODIFY: use wrapper
│   ├── com.ns-bootstrap.update-interactive.plist.template  # MODIFY: use wrapper
│   └── com.ns-bootstrap.watchdog.plist.template            # NEW: runs watchdog every 4h
install/
│   └── bootstrap.sh                                        # MODIFY: generate default watchdog config

Why hybrid over pure watchdog?

• The wrapper catches 90% of failures (runtime errors) with near-zero complexity (~15 lines)
• The watchdog catches the remaining edge case ("job never ran") by checking log freshness — no launchctl list parsing needed
• A pure watchdog that only polls launchctl list has its own silent-failure problem (turtles all the way down)

Alternatives Considered

Approach	Catches runtime failures	Catches "never ran"	Complexity
Watchdog only (polling)	Yes	Yes	Medium — extra plist, config file, launchctl parsing
Wrapper only	Yes	No	Low — ~15 lines
Hybrid (recommended)	Yes	Yes	Low — two simple scripts

[ns408]

Scope update — architecture changed (PR #3), re-scope before implementing

The update/scheduling design this issue was written against has changed. Some of the proposed files no longer exist, and the notification approach needs rethinking for the new model. Updating scope so whoever picks this up builds the right thing.

What changed

• The interactive tier is no longer scheduled. com.ns-bootstrap.update-interactive (plist + systemd units) was removed; casks/mas/softwareupdate now run on-demand via update-my-system. So don't monitor an interactive agent — there isn't one.
• The daily job can run as a LaunchDaemon, not a LaunchAgent. For multi-account setups (Homebrew owner != GUI login user), the daily update runs from scripts/launchd-daemon/com.ns-bootstrap.update-daily.daemon.plist.template (UserName=<brew owner>, system domain). Single-account installs still use the per-user LaunchAgent at scripts/launchd/com.ns-bootstrap.update-daily.plist.template. The watchdog must handle whichever is present.
• The "script path breaks after a rename" case is now partly handled. The installer removes orphaned update agents whose backing script is missing, and warns when the installing admin isn't the console/GUI user. This mitigates the originating scenario but does not notify at runtime — the core of this issue still stands.

Important nuance for the wrapper (run-with-notify.sh)

The original plan has the wrapper call osascript display notification on non-zero exit. That worked when the job was a LaunchAgent (runs inside the user's Aqua session). A LaunchDaemon has no GUI session, so osascript display notification from it won't surface to the desktop. Options:

• Route the notification to the console user: launchctl asuser "$(stat -f %u /dev/console)" osascript -e '...'.
• Or lean on the watchdog as the notifier instead, and run the watchdog itself as a per-user LaunchAgent under the GUI login account so its notifications actually display. The watchdog already checks log freshness, which covers both "ran but failed" (stale-ish log + error) and "never ran" (stale log) — so a watchdog-only notifier may be simpler than wrapper + watchdog now.

Re-scoped file list

scripts/
├── launchd-watchdog.sh                                  # NEW: check ~/.local/log/launchd-update-daily.log freshness, notify if stale
├── run-with-notify.sh                                   # NEW (optional): wrapper; if kept, route osascript via console user
└── launchd/
    └── com.ns-bootstrap.watchdog.plist.template         # NEW: LaunchAgent (GUI login user) running the watchdog every 4h
install/bootstrap.sh                                     # MODIFY: install watchdog agent + default watchdog-jobs.conf

Drop the two update-interactive modifications from the original list. Monitor only com.ns-bootstrap.update-daily (and its log), and run the watchdog in the GUI session so notifications are visible.