fix: stop the reconciler retry storm exhausting the GitHub rate limit

[genisd]

Why

Task 385f2ad2 lost ~3 minutes of agent time unable to read its own CI status:

failed to get runs: HTTP 403: API rate limit exceeded for installation ID 129649310

Everything in Optio — PR watcher, reconciler, and every agent's gh call — authenticates as the same GitHub App installation and shares one bucket of 12,500 req/hour.

The obvious hypothesis (too many PRs polled) was wrong. Agent traffic peaks at 28 gh calls/hour. From 40 minutes of production api logs:

$ kubectl logs deploy/optio-api -c api --since=40m | grep -c 'No stored user token'
1762      # each is a loadPrStatus() call == 3 GitHub API requests

$ ... | grep 'reconcile.decision' | grep -o '"reason":"[^"]*"' | sort | uniq -c
   2392 "reason":"resync"
    885 "reason":"stale_retry:cas_failed_pre_transition"
    328 "reason":"stale_retry:Invalid state transition: failed -> needs_attention"
     72 "reason":"pr_watch:open"

loadPrStatus calls per minute show a benign baseline of 8/min punctuated by storms:

12:17    82      12:21   225      12:25    24
12:18   250      12:22   203      12:26     8
12:19   249      12:23   118      12:27     8
12:20   178      12:24     8      ...

250/min × 3 calls = ~750 GitHub req/min for ~7 minutes. That burst rate is why a 403 arrives while the hourly bucket is nearly empty — the agent's own probe read {"limit":12500,"remaining":12475,"used":25} for the very window its calls were failing in. This is GitHub's per-minute secondary limit, driven by burst rate, not hourly volume.

Root cause

1. reconcile-repo.ts — the merge-conflict edge guards on prev.checks !== "conflicts" but its statusPatch writes "failing". Nothing ever writes "conflicts", so the guard can never become false and the edge re-fires on every pass.
2. decideFailed() routes failed tasks with an open PR into that branch, where canResume is false, so it always lands on the NEEDS_ATTENTION transition.
3. FAILED → NEEDS_ATTENTION is not a legal edge (state-machine.ts), so transitionTask throws.
4. The executor matched Invalid state transition together with StateRace and returned stale.
5. The worker re-enqueued every stale outcome after 500 ms, with no attempt cap and no backoff escalation.
6. The retry rebuilds the snapshot, and loadPrStatus unconditionally spends 3 GitHub calls.

→ ~1 iteration/second per stuck task, indefinitely. Two production tasks were in this loop (external-board-tools#1080, backend-nest#3451, both failed since 2026-06-19 with conflicted PRs).

The 885 cas_failed_pre_transition retries are the same amplifier: casUpdate gates on tasks.updatedAt, which the PR watcher bumped on every row every cycle while 4 reconciles raced for it.

What changed

Breaking the loop

• Write the sentinel the guard compares against ("conflicts", not "failing"), and preserve it in patchPrStatusFields.
• Never propose NEEDS_ATTENTION from FAILED — added isFailed guards to the conflicts / CI-failing / review-changes branches.
• New non-retryable invalid outcome for illegal transitions, split from StateRace (a genuine race, still retried).
• stale retries now escalate 1s→2s→4s… capped at 5 min, max 10 attempts (OPTIO_MAX_STALE_RECONCILE_RETRIES). Counter is in-process on purpose: reconcile_attempts already drives the world-read backoff curve, and inflating it would lengthen unrelated waits.

Removing the waste it amplified

• Cache PR reads 30s per URL — a retry moments later cannot observe anything new.
• Skip the PR fetch for states that provably never read snapshot.pr. Notably needs_attention, which always noops awaiting user intent yet is swept into every 5-minute resync (8 such rows in prod, 3 calls each, result discarded).
• Stop watching tasks idle longer than OPTIO_PR_WATCH_MAX_AGE_DAYS (default 5), measured on last_activity_at — updated_at was unusable because the watcher bumped it on every poll. Aged-out pr_opened tasks transition once to needs_attention so abandonment is visible rather than silent.
• The watcher no longer talks to GitHub; the reconciler is the sole fetcher. Previously both read the same PR every cycle. Also drops the updatedAt bump that kept losing the reconciler's CAS.
• git-platform errors now carry HTTP status, so a 404 PR is reported gone and drops out of the watch set permanently. This also makes the pre-existing 401 auth-failure check functional for the first time — it could never fire before.

Test plan

• 4 new tests written first, all failing against the old code with exactly the production symptom: the second consecutive reconcileRepo on an unchanged conflicted-PR snapshot must not re-propose the same action, and a failed task must never propose NEEDS_ATTENTION.
• One pre-existing test in reconcile-edge-cases.test.ts asserted the buggy FAILED → NEEDS_ATTENTION behaviour; updated to assert the legal outcome.
• Caught on review, not by test: stripping the watcher's fetches left nothing writing prReviewComments, which the resume prompt reads. The edge now persists it and applyResumeAgent prefers the freshly-observed value. Covered by a new test.
• npx turbo typecheck 12/12, npx turbo test 12/12 (2032 api + 398 shared), prettier --check . clean.
• Dry-ran both new queries against production: watch set drops 19 → 4, and 12 tasks age out to needs_attention — 11 of them pointing at frontend-flutter#5678, which does not exist (gh pr view → Could not resolve to a PullRequest) and had been 404-polled since April.

Hooks were bypassed: pnpm turbo test/typecheck fail in this environment because pnpm 11 wants to purge a node_modules installed by pnpm 10 (fallout from #42, unrelated to this change). Both gates were run directly via npx turbo — results above.

Follow-up, deliberately not in this PR

• Nothing honours Retry-After or x-ratelimit-remaining. Callers still cannot distinguish "rate limited" from "PR missing" and never slow down. This PR removes the traffic that trips the limit; that change is what would make a 403 survivable rather than merely rare.
• ETag conditional requests in fetchJson — GitHub does not charge 304s.
• 12 scheduled cve-fix triggers all fire between 06:30 and 07:05 weekdays, every agent spending the same installation bucket concurrently.