Jetmon 2 — Site health platform

.agents/skills/handoff/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: handoff
+description: Create a self-contained Jetmon handoff for another agent.
+---
+
+# Jetmon Handoff
+
+Use this when Chris asks for a handoff doc or wants another agent to continue a
+Jetmon thread.
+
+## Include
+
+- Repo path, branch, and relevant commit IDs.
+- Whether the work affects Jetmon v1, Jetmon v2, Veriflier, bridge, support
+  services, or uptime-bench.
+- Active test locks and what must not be changed.
+- Problem statement, evidence, and current hypothesis.
+- Relevant logs, reports, metrics, PRs, and file paths.
+- Commands already run and their outcome.
+- Next recommended actions and approvals needed.
+
+## Placement
+
+During active tests, prefer `.agents` or global memory for agent-only handoffs.
+Ask before editing non-agent project docs.
+
+## Secrets
+
+Do not include tokens, passwords, private keys, or unredacted service configs.

.agents/skills/jetmon-test-fleet/SKILL.md

@@ -0,0 +1,44 @@
+---
+name: jetmon-test-fleet
+description: Work safely with Jetmon services used by uptime-bench capacity tests.
+---
+
+# Jetmon Test Fleet
+
+Use this when Chris asks about Jetmon v1/v2 test services, Verifliers, support
+services, Prometheus capacity data, or whether a Jetmon branch is ready for
+uptime-bench tests.
+
+## Safety First
+
+- If tests are running, do not restart services, change config, move support
+  services, deploy binaries, mutate databases, or alter target/provider state
+  without explicit permission.
+- Prefer read-only inspection and report analysis during active tests.
+- State which repo is being acted on before making changes.
+
+## Common Context
+
+- Uptime-bench canonical repo:
+  `/home/gaarai/code/uptime-bench`.
+- Current Prometheus for Jetmon capacity work:
+  `http://10.0.0.67:9091`.
+- Service hosts:
+  `jetmon-service-host-1`/`jetmon-v1`,
+  `jetmon-service-host-2`/`jetmon-v2`,
+  `jetmon-service-host-3`,
+  `jetmon-service-host-4`.
+- Support/monitoring hosts:
+  `jetmon-vm-host-1`,
+  `jetmon-vm-host-2`,
+  `jetmon-vm-host-3`.
+
+## Output Expectations
+
+When answering readiness or risk questions, include:
+
+- Branch and commit under discussion.
+- What is deployed versus only local.
+- Which checks were read-only.
+- Whether changes are safe during an active uptime-bench run.
+- Recommended next action and any approval needed.

.agents/skills/safe-background-work/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: safe-background-work
+description: Pick useful Jetmon work that cannot affect active uptime-bench or Jetmon tests.
+---
+
+# Safe Background Work
+
+Use this when tests are running and Chris asks what can be done without
+interrupting them.
+
+## Allowed By Default
+
+- Local code review and static analysis.
+- Agent-specific files.
+- Branch inspection and commit comparison.
+- Handoff writing.
+- Local-only planning for changes that will not be deployed.
+
+## Ask First
+
+- Deploying binaries or configs.
+- Restarting `jetmon2`, Jetmon v1, bridge, Veriflier, database, StatsD, or
+  monitoring services.
+- Moving support services between hosts.
+- Changing bucket ownership, pinned bucket ranges, or test fleet data.
+- Running smoke tests that create, delete, or modify sites/providers.
+
+## Blocker Policy
+
+If a safe task becomes blocked on approval, record the blocker and move to the
+next safe task.

.claude/commands/debug-memory.md

@@ -1,97 +1,66 @@
 # Debug Memory Issues
 
-Debug memory issues in Jetmon workers and identify leaks.
+Debug memory growth and goroutine leaks in the Jetmon 2 Go binary.
 
 ## Instructions
 
-Help the user diagnose memory problems in Jetmon workers. Memory leaks are a known pitfall because workers are long-running processes.
+Help the user diagnose memory problems in Jetmon 2. Unlike the old Node.js/worker architecture,
+Jetmon 2 is a single Go binary. Memory pressure does not cause worker crashes — instead the
+orchestrator drains the goroutine pool when RSS exceeds `WORKER_MAX_MEM_MB`.
 
 ### 1. Check Current Memory Status
 
-First, see current memory usage of all Jetmon processes:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon ps aux --sort=-%mem | grep -E '(node|PID)' | head -20
+cd docker && docker compose exec jetmon ps aux
 ```
 
-Check worker memory limits in config:
+Check memory config:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon cat config/config.json | grep -E '(WORKER_MAX_MEM|WORKER_MAX_CHECK)'
+docker compose exec jetmon cat config/config.json | grep -E '(WORKER_MAX_MEM|NUM_WORKERS)'
 ```
 
-### 2. Monitor Memory Over Time
+### 2. Use pprof for Deep Analysis
 
-Watch memory growth in real-time:
-```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon bash -c 'while true; do echo "=== $(date) ==="; ps aux --sort=-%mem | grep node | head -10; sleep 10; done'
-```
-
-Let this run for a few minutes to observe trends. Look for:
-- Workers steadily increasing memory without recycling
-- Workers approaching or exceeding `WORKER_MAX_MEM_MB` (default 53MB)
-- Memory not dropping after worker recycle
-
-### 3. Check Worker Recycling
+The operator dashboard exposes pprof endpoints at http://localhost:8080/debug/pprof/
 
-Verify workers are being recycled when hitting limits:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose logs jetmon 2>&1 | grep -E '(memory|recycle|spawn|die|limit)' | tail -30
-```
-
-### 4. Force Aggressive Recycling (Testing)
+# Count goroutines
+curl http://localhost:8080/debug/pprof/goroutine?debug=1 | grep -c "^goroutine"
 
-To test worker recycling behavior, temporarily set low limits:
-
-```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon bash -c 'cat > /tmp/test-config.json << EOF
-{
-    "WORKER_MAX_CHECKS": 50,
-    "WORKER_MAX_MEM_MB": 20
-}
-EOF
-cat /tmp/test-config.json'
+# Heap profile
+curl http://localhost:8080/debug/pprof/heap > heap.prof
+go tool pprof heap.prof
 ```
 
-Tell the user to manually update `config/config.json` with these values, then reload:
-```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon sh -c 'kill -HUP $(pgrep -f "node lib/jetmon.js" | head -1)'
-```
+### 3. Monitor Memory Over Time
 
-Watch for recycling:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose logs -f jetmon 2>&1 | grep -E '(spawn|die|recycle|memory|limit)'
+docker compose exec jetmon bash -c 'while true; do ps -o pid,rss,vsz,comm -p $(pgrep jetmon2); sleep 10; done'
 ```
 
-### 5. Check for Known Memory Issues
+Enable detailed StatsD metrics by setting `STATSD_SEND_MEM_USAGE: true` in `config/config.json`,
+then reload config: `docker compose exec jetmon ./jetmon2 reload`
 
-**Retry queue growth:** If retry queues aren't being processed, they can grow unbounded:
-```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose logs jetmon 2>&1 | grep -i retry | tail -20
-```
+### 4. Check Retry Queue Size
+
+Large retry queues indicate many sites are down and being tracked. This is expected behaviour.
 
-**StatsD buffer:** Check if metrics buffer is growing:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon bash -c 'cat stats/* 2>/dev/null'
+curl http://localhost:8080/api/state | python3 -m json.tool
 ```
 
-### 6. Analyze with Node.js Tools (Advanced)
-
-If deeper analysis is needed, suggest:
-
-1. **Heap snapshots:** Would require code changes to expose `v8.writeHeapSnapshot()`
-2. **--inspect flag:** Could attach Chrome DevTools, but requires exposing debug port
-3. **Process stats:** Check `/proc/<pid>/status` for detailed memory breakdown
+Look at `RetryQueueSize`.
 
-### 7. Common Memory Issues in Jetmon
+### 5. Common Issues
 
 | Symptom | Likely Cause | Fix |
 |---------|--------------|-----|
-| Workers never recycle | `WORKER_MAX_MEM_MB` set to 0 or very high | Set reasonable limit (53MB default) |
-| Memory spikes during rounds | Too many concurrent checks | Reduce `NUM_TO_PROCESS` |
-| Gradual leak over hours | Retry queue not draining | Check Veriflier connectivity |
-| Sudden OOM | Node.js version regression | Test with previous Node version |
+| Goroutine count grows | Context not cancelled on shutdown | Verify `orch.Stop()` called |
+| Memory never drops | Pool drain not triggered | Check `WORKER_MAX_MEM_MB` value |
+| Retry queue unbounded | Veriflier unreachable | Check veriflier connectivity |
+| High allocations | Keyword-check body reads | Reduce `NUM_WORKERS` |
 
-### 8. Restore Normal Settings
+### 6. Restore Normal Settings
 
-Remind user to restore normal config values after testing:
-- `WORKER_MAX_MEM_MB`: 53
-- `WORKER_MAX_CHECKS`: 10000
+After testing, remind user to restore:
+- `STATSD_SEND_MEM_USAGE`: false (avoid extra StatsD traffic in production)

.claude/commands/docker-test.md

@@ -1,79 +1,91 @@
 # Docker Test Environment
 
-Run, debug, and test Jetmon using the Docker development environment.
+Run, debug, and test Jetmon 2 using the Docker development environment.
 
 ## Instructions
 
-Help the user test Jetmon in the Docker environment. Follow these steps:
+Help the user test Jetmon 2 in the Docker environment. Follow these steps:
 
 ### 1. Check Docker Status
 First, check if the Docker environment is already running:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose ps
+cd docker && docker compose ps
 ```
 
 ### 2. Start Services (if needed)
 If services aren't running, start them:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose up -d
+cd docker && docker compose up -d
 ```
 
 Wait a few seconds for services to initialize, then verify:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose ps
+docker compose ps
 ```
 
 ### 3. Ask User What They Want to Test
 
 Present these options:
 - **View logs** - Watch Jetmon or Veriflier logs in real-time
-- **Check worker status** - See worker activity and stats
+- **Operator dashboard** - Open http://localhost:8080 in a browser
 - **Test with sample sites** - Insert test URLs into database
-- **Test configuration reload** - Send SIGHUP to master process
-- **Test graceful shutdown** - Verify shutdown behavior
+- **Test configuration reload** - Send SIGHUP to reload config
+- **Test graceful drain** - Verify drain/shutdown behaviour
 - **Test Veriflier connectivity** - Check Veriflier is responding
+- **View audit log** - Query the audit log for a specific blog
 - **View metrics** - Check StatsD/Graphite dashboard
 
 ### 4. Execute Based on Selection
 
 **View logs:**
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose logs -f jetmon
+docker compose logs -f jetmon
 ```
 
-**Check worker status:**
+**Check process and stats:**
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon ps auxf
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon cat stats/sitespersec
+docker compose exec jetmon ps aux
+docker compose exec jetmon cat stats/sitespersec
+docker compose exec jetmon cat stats/sitesqueue
 ```
 
 **Test with sample sites:**
 First check if table exists and has data:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec mysqldb mysql -u root -p123456 jetmon_db -e "SELECT COUNT(*) as count FROM jetpack_monitor_sites;" 2>/dev/null
+docker compose exec mysqldb mysql -u root -p123456 jetmon_db -e "SELECT COUNT(*) as count FROM jetpack_monitor_sites;" 2>/dev/null
 ```
 
 If empty or table doesn't exist, offer to create test data per `running-tests.md`.
 
 **Test configuration reload:**
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon sh -c 'kill -HUP $(pgrep -f "node lib/jetmon.js" | head -1)'
+docker compose exec jetmon ./jetmon2 reload
+```
+
+**Test drain/graceful shutdown:**
+```bash
+docker compose exec jetmon ./jetmon2 drain
 ```
 
 **Test Veriflier connectivity:**
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose exec jetmon curl -k https://veriflier:7801/get/status
+docker compose exec jetmon curl http://veriflier:7803/status
+```
+
+**View audit log:**
+```bash
+docker compose exec jetmon ./jetmon2 audit --blog-id 1 --since 1h
 ```
 
 **View metrics:**
 Tell user to open http://localhost:8088 and navigate to `Metrics > stats > com > jetpack > jetmon > docker > jetmon`
 
 ### 5. Cleanup (if requested)
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose down
+docker compose down
 ```
 
 Or to fully reset with fresh database:
 ```bash
-cd /Users/rdcoll/Code/a8c/jetmon/docker && docker compose down -v
+docker compose down -v
 ```