Change base realm indexing to highest priority

[backspace]

Production host mode sites were down for over an hour on Tuesday night because of a new @cardstack/boxel-ui import used in @cardstack/base. One mitigation: make base realm reindexing take priority by sorting kind = 'bootstrap' to the top.

Claude: Serve the base realm's system-initiated reindex from the high-priority pool

Job priority is a worker pool's dequeue floor, not a sort key, so a system-tier index can only be claimed by the all-priority pool, in strict arrival order behind everything already queued there at any tier. A fleet-wide sweep is exactly such a backlog, and getFullReindexRealmUrls ordered by url alone — which sorts https://cardstack.com/base/ behind every app.boxel.ai / boxel.site / boxel.space realm, putting base's reindex last of hundreds of jobs. A base-realm index carrying an error row 500s anonymous card+json reads on every realm that links a base card, and the repair itself takes milliseconds, so that queue position cost 72 minutes of exposure in production on 2026-08-26.

Two changes, either of which alone would have shortened that wait:

• systemInitiatedIndexPriority enqueues the base realm's system-initiated index at the user-initiated tier, which the high-priority pool can serve while the sweep occupies the all-priority pool. Both system paths use it: the post-deploy sweep's fan-out and the boot index in main.ts. Base is the only realm elevated — the high-priority pool is a rescue lane for latency-sensitive user work, and base is small enough not to hold it.
• The sweep's source list sorts bootstrap realms ahead of the rest, so catalog / skills / homepage get queue position without the pool bypass. That ordering is also what covers a deployment running --highPriorityCount=0, where no pool floors above the system tier.

Covered by full-reindex-test: the per-realm tier the fan-out enqueues at, the bootstrap-first ordering of the sweep's source list, and the priority policy itself.

[github-actions]

Host Test Results

    1 files  ±0      1 suites  ±0   2h 38m 3s ⏱️ + 8m 2s
4 596 tests +1  4 582 ✅ +1  14 💤 ±0  0 ❌ ±0 
4 611 runs  +1  4 597 ✅ +1  14 💤 ±0  0 ❌ ±0 

Results for commit 05cf3e6. ± Comparison against earlier commit 0a78e64.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   17m 59s ⏱️ -21s
2 358 tests +3  2 358 ✅ +3  0 💤 ±0  0 ❌ ±0 
2 441 runs  +3  2 441 ✅ +3  0 💤 ±0  0 ❌ ±0 

Results for commit 05cf3e6. ± Comparison against earlier commit 0a78e64.

[richardhjtan]

[Claude Code 🤖]

This review went after whether the elevation is reachable in the deployment the incident happened in: what the claim query actually does with priority, which worker pools can see a job at the user-initiated tier, and whether the boot-index option is genuinely plumbed. It did not re-review the incident's root cause or the @cardstack/boxel-ui import that triggered it.

No blocking issues in the code. The blocking question is a deployment fact the diff cannot answer: every pool that floors above the system tier is off by default, and if production runs that default then the priority half of this PR changes nothing and only the sweep ordering shortened the wait. Detail in the thread on systemInitiatedIndexPriority.

1. Say which worker pools production runs, and if neither --highPriorityCount nor --userIndexCount is set there, either set one in the same change or reword the comment so it does not read as the primary mitigation — thread on systemInitiatedIndexPriority.
2. The comment reasons only about the high-priority pool and never mentions the --userIndexCount lane, which floors at exactly the tier base is elevated to and registers indexing job types only. That lane is a better home for this than the render rescue lane, and its absence from the comment makes the trade-off look worse than it is — same thread.

Red check: percy/-cardstack-host is the only failure and this diff touches no host code, so it is triage rather than a fix.

[richardhjtan]

[Claude Code 🤖] The elevation only does something when a pool exists that floors above the system tier, and every such pool is off by default — so the deployment question decides whether this half of the PR is live at all.

The mechanism, confirmed in packages/postgres/pg-queue.ts: the claim query filters status='unfulfilled' and priority >= <pool floor> and then orders BY j.created_at, j.id LIMIT 1. Priority never reaches the ORDER BY, exactly as the comment says. The consequence the comment stops one step short of: in a fleet whose only pool floors at systemInitiatedPrerenderHtmlPriority, a base job at priority 10 and the same job at priority 1 are claimed in identical order. Not merely "still waits its turn" — indistinguishable.

And the defaults are that fleet. worker-manager.ts declares highPriorityCount = 0 and userIndexCount = 0; mise-tasks/lib/env-vars.sh exports WORKER_HIGH_PRIORITY_COUNT="${WORKER_HIGH_PRIORITY_COUNT:-0}" (the :=4 above it applies only under BOXEL_TURBO), and there is no WORKER_USER_INDEX_COUNT in that file at all. So unless production sets one explicitly, the sweep reordering is the entire fix and this function is inert there.

Two asks:

• State production's pool configuration. If it is the default, either raise a count in this change or rewrite the comment so the priority elevation reads as preparation for a fleet that has such a pool, rather than as the mitigation that shortens the next incident.
• Mention the --userIndexCount lane. It floors at userInitiatedPriority — precisely the tier this elevates base to — and passes indexJobsOnly: true, so a base index there cannot be held behind a prerender-html sweep nor hold one. That makes it a better home for this than the high-priority pool, and it also answers the paragraph above worrying that "handing it a large system index would defeat the lane": index jobs carry indexingConcurrencyGroup(realm.url) (realm-index-updater.ts) and the claim query excludes groups with a live reservation, so base can occupy at most one worker at a time regardless of pool.

Class: not a defect in the diff — the code does what it says. The ask is that the comment's claim about what this buys be true of the fleet it runs on. Blocking only as a question.

[backspace]

[Claude Code 🤖] Both asks addressed — the deployment question with evidence, the comment in e2db488988.

Production is not the default fleet. Every worker task logs its pool composition at boot, and production's says:

starting 0 user-index workers, 3 high-priority workers and 1 all-priority worker

ecs-boxel-worker-production, task definition boxel-worker-production:309, on 4 running tasks — so 12 high-priority workers and 4 all-priority workers fleet-wide. Staging logs the same shape per task. WORKER_HIGH_PRIORITY_COUNT is an SSM-backed ECS secret (/<env>/boxel/…), which is why the value isn't visible in either repo. So the elevation is live where the incident happened: base at priority 10 is claimable by 12 workers that the 721-job system-tier backlog cannot reach, and the earlier --highPriorityCount=0 caveat describes local dev, not production.

Worth noting for a follow-up rather than this PR: WORKER_USER_INDEX_COUNT is not in the worker service's ecs_secrets_list at all, so the index lane can't currently be turned on in a deployed environment without an infra change.

On the lane, and on what elevating base costs. Both fair, and the comment now carries them. It named only the high-priority pool; it now names both shapes that float above the system tier and says the index lane is the better home of the two, for the reason you give — its indexJobsOnly filter means a prerender-html sweep can neither hold it nor be held by it.

Your concurrency-group correction also fixed the justification, not just its tone. The comment had claimed a large system index would occupy the lane; since every index job for a realm shares indexingConcurrencyGroup(realmURL) and the claim query excludes groups holding a live reservation, base can occupy at most one worker at a time whatever pool serves it. The real reason to elevate one realm rather than eight is that each further realm adds another group, and so another worker held off user-initiated work — which is what it says now.

main is merged in as of f3ee5e779e (clean, no conflicts). Re-checking the red percy/-cardstack-host on the new run — agreed it's triage, since this diff is realm-server and runtime-common only.

[backspace]

[Claude Code 🤖] Attribution for the red Boxel CLI Tests on this PR, since it isn't this diff.

It reproduced on both attempts of the same run (realm-ingest-card hook timeout, then search EADDRINUSE on 4460) and root-causes to the host-shell recycle merged in #5902, now filed as CS-12687 with a fix pushed. Short version: those two suites are the only CLI integration files that start a real prerender server, the recycle fires at their start, and the standby refill it leaves failing blocks afterAll → Prerenderer.stop() → PagePool.closeAll() past vitest's 10s default hookTimeout, leaking the 4460/4445 listeners into the following suites.

Main shows the same recycle in the same suite and passes when that refill settles before teardown, so it is reachable on any branch. Nothing in this PR's diff is on that path: the CLI stack logs 0 user-index workers, 0 high-priority workers and 1 all-priority worker, and with one pool at the lowest floor a tier-10 job is claimed in exactly the same order as a tier-1 one — base's index started in 5ms either way. The sweep-ordering half only runs via the post-deployment hook or _grafana-full-reindex, neither of which the CLI tests call.

The Matrix Client Tests (1, 3) red was host-mode.spec.ts:520, flaky on main's own run of the commit this branch merged; it passed on re-run. All 6 Realm Server shards — the ones covering this diff — passed on the original run.

[burieberry]

[Claude Code 🤖] Two findings from a review pass, both verified against the branch.

1. lib/full-reindex-realm-urls.ts — base still sorts last within the bootstrap group

ORDER BY (kind = 'bootstrap') DESC, url lifts bootstrap realms above source/published realms, but inside the bootstrap group the key is still url — and base sorts last of the three.

The registry url depends on the CLI arg shape. main.ts seeds bootstrap rows with hrefs[i][0], and urlMappings keeps the from-form only when --fromUrl is url-like, otherwise storing [to, to]. In production base is --fromUrl 'https://cardstack.com/base/' (url-like → registry url https://cardstack.com/base/), while catalog and skills pass @cardstack/catalog/ and @cardstack/skills/ (not url-like → registry urls under app.boxel.ai). 'a' < 'c', so the sweep enqueues catalog, then skills, then base. The new test encodes exactly that: [catalogRealm, baseRealm.url, userRealm, publishedRealm].

Ordering is load-bearing here because priority never reorders within a pool — pg-queue.ts filters priority >= <floor> and then claims ORDER BY j.created_at, j.id. A sweep enqueues every job in one loop sharing a transaction timestamp, so the id tiebreak makes enqueue order decisive.

Impact depends on the deployment: with no pool above the system tier (the worker scripts default --highPriorityCount to 0, and the deployed value comes from SSM), base's job is claimed by the all-priority pool only after catalog's and skills' full reindexes have drained ahead of it — so a base error row keeps serving 500s fleet-wide for that whole window. This PR names the sweep ordering as the safety net for exactly that case, so it seems worth anchoring base explicitly rather than relying on the url collation:

ORDER BY (url = 'https://cardstack.com/base/') DESC, (kind = 'bootstrap') DESC, url

2. jobs/indexing.ts — base can hold two high-priority workers, not the "at most one" the comment claims

The new comment argues the cost is bounded because every job writing a realm's index shares indexingConcurrencyGroup(realmURL). The spawned prerender job isn't in that group.

tasks/indexer.ts passes spawningPriority: jobInfo?.priority, and prerenderHtmlPriority maps any spawning priority at or above userInitiatedPriority to userInitiatedPrerenderHtmlPriority (9) rather than 0. That job publishes under prerenderHtmlConcurrencyGroup(realmURL) — deliberately separate from indexing:${realmURL} so HTML work never blocks indexing — and the high-priority pool floors at 9, so it can claim it.

That makes it a recurring per-deploy cost rather than an incident-only one: resolveFullIndexOnStartup defaults to full-indexing every kind='bootstrap' realm at boot, and a from-scratch pass sets preWarm, so every rolling deploy fires a realm-wide module sweep for base at tier 9. On a deployment with one high-priority worker, that long-running job occupies the lane the elevation exists to protect.

Two options, depending on intent: pass an explicit spawningPriority for the elevated base index so its prerender job stays at the system tier, or narrow the comment's claim to the index job and state the prerender cost as accepted.

Separately — the comment points at --userIndexCount + --indexJobsOnly as the better home for this, but neither the production nor the staging worker script configures that lane, so the documented mitigation isn't deployed anywhere today. Worth saying so explicitly if the elevation is meant to lean on it.

[backspace]

[Claude Code 🤖] Both findings stand, and both are fixed in 05cf3e66ad. The second one also invalidated a claim I'd made in the comment, so thanks for chasing it into prerenderHtmlPriority.

1. Base last inside the bootstrap group. Confirmed, and production bears it out beyond the reasoning: in the 2026-08-26 sweep catalog was job 276527 and base 277206 — 679 jobs apart in the same enqueue burst, exactly the collation you describe one level down (app.boxel.ai/catalog/ ahead of cardstack.com/base/, because a non-url --fromUrl registers the realm under the server's own host).

Anchored, but keyed on baseRealm.url as a bound param rather than a SQL literal, so the realm the ordering names and the realm the tier bump names stay one fact:

ORDER BY (url = $1) DESC, (kind = 'bootstrap') DESC, url

The test now expects [baseRealm.url, catalogRealm, userRealm, publishedRealm] — you were right that the old expectation encoded the gap.

2. Two workers, not one. Correct, and my comment was wrong rather than merely imprecise. spawningPriority: jobInfo?.priority at both spawn sites plus prerenderHtmlPriority mapping ≥10 to 9 means an elevated base index put its prerender-html job in the high-priority pool too, in its own concurrency group — and with preWarm: true on a from-scratch pass that is an O(realm module count) sweep on every deploy, not just during an incident.

I took your first option rather than narrowing the comment, because it makes the bound I'd claimed actually true: spawned HTML now derives from the tier base would have had without the elevation, via a prerenderSpawnedPriority helper next to the policy it qualifies. A publish-awaited render keeps the index tier, since there the render really is on the caller's critical path. A user-initiated base reindex is demoted along with the system-initiated one — the two are indistinguishable at that point, and HTML isn't what such a caller waits on; that's stated in the helper's comment rather than left implicit.

On the index lane: right, and the comment no longer implies otherwise. It now says the lane would be the better home but that no deployment configures it today, so in practice the elevation reaches the high-priority pool — which production does run, at 3 per worker task across 4 tasks.

Verified: eslint and typecheck clean; the new helper exercised directly for all five cases (base elevated → system tier, base via the @cardstack/base/ alias → same, user realm → unchanged, base already at the system tier → unchanged, publish-awaited → unchanged). The DB-backed ordering test runs in CI.

[backspace]

[Claude Code 🤖] Correction to my earlier comment on this PR, which attributed the red Boxel CLI Tests to the host-shell recycle from #5902 and to CS-12687.

That attribution was wrong. #5930 carries the CS-12687 fix and its CI shows the fix working — exactly one Standby creation attempt 1 failed in the whole job log, where pre-fix runs reached three — and the CLI job still fails identically (hook timeout in realm-ingest-card, then EADDRINUSE 4460 in search). So the page-pool defect is real, but it is not what breaks that job.

What the same run points at instead: the timed-out hook's stack is stopTestRealmServer → PgQueueRunner.destroy → WorkLoop.shutDown, with ROLLBACK failed during connection cleanup while the suite's index job was still issuing prerender visits — teardown waiting on the queue runner draining in-flight work, not on the page pool. My other theory in that comment, overlapping suites, is also wrong: the integration run is vitest run --pool=forks --poolOptions.forks.singleFork, so files are sequential.

The half of that comment that still stands is that none of it is caused by this PR's diff: the CLI stack runs one worker pool at the lowest floor, where a tier-10 job is claimed in the same order as a tier-1 one, and the sweep-ordering half never runs there. Tracked as CS-12695, which had already recorded this failing on main independently of any branch.