Repository navigation
feat(live): production — one poller across tasks, a fraction of the bytes, on by default - #168
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 4 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| /** Join the other tasks: listen, then ask the leaders for what they hold. */ | ||
| async connect(): Promise<void> { | ||
| if (!this.bus) return; | ||
| try { | ||
| await this.bus.subscribe((message) => this.receive(message)); | ||
| await this.bus.publish({ type: 'sync', from: this.bus.taskId }); | ||
| } catch (error) { | ||
| this.busFailed(error); | ||
| } |
There was a problem hiding this comment.
🔴 Recovered bus strands live followers
When the first connect() fails, a later successful coordinate() stops local polling without restoring the subscription. Followers receive no published positions until their processes restart.
Learn more
The hub subscribes to the shared Valkey channel only in connect(), which runs once from start. If that call fails, busDown enables local polling. Later coordinate can succeed and clear busDown, although the subscriber was never connected. Followers stop polling and never receive leader updates.
Example: Valkey is offline when task B starts, then returns. B's next coordination succeeds, task A gets the lease, and B's existing map viewers receive no vehicles from A.
Recommended fix: Track subscription health separately from command health. Retry connect() on recovery, confirm the subscription and request a fresh sync before disabling local polling; also handle subsequent subscriber disconnects.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let valkeyUrl: string | null = null; | ||
| const rawValkey = emptyAsUndefined(source.LIVE_VALKEY_URL) ?? emptyAsUndefined(source.REDIS_URL); | ||
| if (rawValkey !== undefined) { | ||
| try { | ||
| const url = new URL(String(rawValkey).trim()); | ||
| if (url.protocol !== 'redis:' && url.protocol !== 'rediss:') throw new Error('protocol'); | ||
| valkeyUrl = url.toString(); | ||
| } catch { | ||
| // Never echo the value: it may carry a password. | ||
| problems.push(' - LIVE_VALKEY_URL / REDIS_URL: must be a redis:// or rediss:// URL'); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
🔴 Disabled live configuration blocks backend startup
With LIVE_ENABLED off and a malformed REDIS_URL, parseLiveConfig() still throws at server import. The entire backend fails to start, although live polling is disabled.
Learn more
The liveConfig export is evaluated when server bootstrap imports it. The new Redis URL parse runs even when LIVE_ENABLED is unset. A bad optional bus setting now prevents Places, search, and routing from booting, although none of them requires the live service.
Example: Start the backend with LIVE_ENABLED=0 and REDIS_URL=not-a-url. The module throws during import instead of starting with live disabled.
Recommended fix: Validate bus settings only when the live service is enabled and has configured sources, or isolate the live parse so an invalid bus setting disables live without aborting core startup.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const [available, setAvailable] = useState<'checking' | 'yes' | 'no'>('checking'); | ||
| useEffect(() => { | ||
| let current = true; | ||
| void import('@/lib/live/client') | ||
| .then(({ liveApiClient }) => liveApiClient.live.sources()) | ||
| .then( | ||
| () => current && setAvailable('yes'), | ||
| (error: unknown) => | ||
| // Anything else (offline, a blip) is the socket's to report and retry. | ||
| current && setAvailable(error instanceof GoWayUnavailableError ? 'no' : 'yes'), | ||
| ); | ||
| return () => { | ||
| current = false; | ||
| }; | ||
| }, []); |
There was a problem hiding this comment.
🟡 Existing maps miss newly enabled live feeds
If live.sources() returns 503 once, LiveHost remains unavailable after the backend enables live. Open maps never reconnect or show vehicles until remounted.
Learn more
Availability is checked once per mount. On a 503, the state becomes no and the socket effect never runs. Backend rollout enables LIVE_ENABLED independently of existing frontend sessions, so those sessions cannot discover the change.
Example: A map stays open while /live/sources changes from 503 to 200 during deployment. The map keeps showing no Live panel and opens no socket until the user reloads it.
Recommended fix: Retry the availability request after a bounded delay while the feature is enabled, cancel retries on unmount, and open the socket when a retry returns 200.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const bus = this.bus; | ||
| if (bus && !this.busDown) { | ||
| void Promise.all( | ||
| this.states | ||
| .filter((state) => state.leader) | ||
| .map((state) => bus.release(state.definition.id).catch(() => {})), | ||
| ).finally(() => void bus.close().catch(() => {})); | ||
| } |
There was a problem hiding this comment.
🟡 Failed bus connections survive live shutdown
When stop() runs during a bus outage, busDown skips bus.close(). The command and subscriber connections remain open after live has stopped.
Learn more
ValkeyLiveBus owns two Redis clients, and close closes both. An outage sets busDown; the shutdown path then bypasses close altogether. Graceful shutdown may exit soon afterward, but a stopped service held in-process retains these connections and their reconnect activity.
Example: A task loses Valkey, then calls live.stop() during a drain. Both Redis clients remain open until process termination rather than closing with the service.
Recommended fix: Always close the bus during stop, regardless of whether releasing leases is possible; attempt lease release only when the bus is reachable.
Was this helpful? React with 👍 or 👎 to provide feedback.
02061d0 to
2a70549
Compare
…ver blocks boot From the review of #168: - A task whose subscription never happened (Valkey away at start) or was lost with its connection (Bun's client reconnects without restoring subscriptions — measured on a Redis restart) now subscribes again on a fresh connection, with a sync, before it stops polling locally. The bus reports `isSubscribed()`; a reconnect marks it lost. - With LIVE_ENABLED off, parseLiveConfig reads nothing else and cannot throw: no live setting (a shared REDIS_URL, a stale source list) can stop the backend from booting. - An open map asks again whether the backend serves live (30 s, 1, 2, then every 5 min), so it finds live when LIVE_ENABLED is turned on. - Stopping live closes the bus's connections even during an outage. Each has a test that fails without its fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ytes, on by default GoWay Live ready for goway.to: - One poller across API tasks (live/cluster.ts): tasks share demand (room keys with a TTL, never a viewport) through Valkey (REDIS_URL, the fleet's /oxy/_shared/REDIS_URL); a per-source lease (SET NX PX, compare-and-renew) picks the one task that polls; it publishes each answer (gzip) on pub/sub and every task applies it and serves its own sockets. A dead leader's lease expires in 15 s and another task polls at once; a starting task is synced; with Valkey down every task polls locally, and a follower stops polling only once it is subscribed again (Bun's client reconnects without restoring subscriptions). Positions are never stored in Valkey. Tested in memory and against a real Valkey (CI service valkey/valkey:8.0); failover measured with two real backend processes. - Bandwidth: rooms at a zoom per kind (aircraft/ships z7, transit z11, bike share z13; LIVE_ROOM_ZOOMS), bike share asked for only from z15, vehicles that only moved sent as compact LiveMove tuples, and messages over 2 KB as gzip bytes (Bun's WebSocket server does not negotiate permessage-deflate). New York at the same moment: first snapshots 1,225 KB → 25 KB, deltas 18 KB/s → 0.7 KB/s (whole city) and 0.3 KB/s (Midtown). - The registry offers live everywhere on the web, on by default: the backend is the gate (GET /live/sources 503 → nothing drawn, no socket; asked again while open), the panel starts as its pill, live.maxEntities caps drawing. WebSocket-only transport, so no ALB stickiness is needed. - docs/MAP_LIVE.md → Production: the cluster, the measurements, the exact environment and secrets, the owner's oxy-infra checklist (2 GB tasks), a per-source licence checklist, the adsb.lol email and the MTA key steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b9c3ccc to
95527c5
Compare
# Conflicts: # docs/MAP_FEATURES.md # packages/frontend/features/explore/ExploreScreen.tsx
…only coarse demand From the independent review of #168, each with a test that fails without it: - After an outage of 8 s or more Bun 1.4.3's Redis client stays failed (ERR_REDIS_CONNECTION_CLOSED) even once Valkey is back, so every task kept polling every source. A failed call now replaces any connection that is no longer connected (LiveBus.recover), a subscriber that closed or is not connected counts as lost, and a real-server test stops redis-server for 10 s and asserts one leader again and the follower hearing it. - Every bus call is bounded at a third of the lease: unanswered, the bus counts as down and the task polls on its own, instead of a follower going dark for ~10.7 s per command (minutes on a silent partition). - A malformed shared REDIS_URL logs a warning (never the value) and live runs without a bus; it no longer stops the backend booting. LIVE_VALKEY_URL is gone: REDIS_URL is the one name. - Shared demand is z6 tiles × kinds (~470 km), never the rooms a viewer is in; MAP_LIVE.md says exactly what is stored, for how long and why it is not a viewport. - CI installs redis-server (kept from starting on the service's port) for the outage test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… replace a connecting client From the second review of #168, each with a test that fails on 9a07838: - Demand: each task decides from its exact rooms (never shared) which sources it wants and shares only their ids — plus z6 regions for the one regional feed (aircraft), polled per z6 region anyway. A viewer in Philadelphia no longer turns on the MTA's feeds (they shared a z6 tile with New York). Only the leader, and a task whose own viewers watch a source, read its static GTFS. - Lease epochs: the lease carries Valkey's clock at the time it was taken; every message is stamped with it and followers drop older ones, so a publish abandoned at its deadline and delivered late is never applied. - recover() replaces a client only when a call failed on its connection or it connected and then dropped — never one still making its first connection. - Reverts an unrelated .pyc change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery vehicle With transit rooms only at z11 (~15 km) and at most 20 rooms per subscription, transit vanished from any view wider than ~z10.5: all of Catalonia showed no Rodalies train at all. Rooms now come at a few levels per kind (transit z7, z9, z11); every entity is indexed in one room per level, and a subscription takes, per kind, the finest level that covers its view in at most 20 rooms — a borough in a city, the z7 rooms for a region. Street views keep their small rooms and bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y stop areas, counter epochs From the independent review of db334c8; each with a test that fails without it. - The subscription's answer lists its rooms PER KIND (`tiles: { transit, aircraft, ... }`) and the client keeps exactly those (tile, kind) pairs: an aircraft z7 tile no longer keeps the transit z7 room of the same tile, whose buses froze there after zooming into Midtown. - Only rail (`LIVE_REGIONAL_TRANSIT_MODES`: rail, subway) is in the regional transit rooms; buses and every street-level mode are in z11 rooms alone (`liveRoomZoomsOf`, the one source of truth in the contract). A regional view of New York: 3.4 KiB of snapshots (77 trains) instead of 100.2 KiB (2,344 vehicles). A source of street modes only (`transitModes`) is not woken by a regional view. - Static GTFS is read for local demand only from the rooms that can use shapes (the finest transit level; `LIVE_SHAPES_MIN_ZOOM`, now shared). - Demand is matched against each source's areas, derived from its operator's stops (`areas.ts`, `live:areas` → `areas.generated.ts`), and its coverage is the box round them: a street view of Toulouse no longer wakes Renfe. - Lease epochs come from a per-source `INCR` counter, not Valkey's clock; a follower takes a lower epoch once the newer holder has been silent a whole lease, so a failover that lost the counter cannot leave it deaf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
#194 and #195 are rebased onto this commit. #195 derives areas for 51 of its 73 feeds. 🤖 Generated with Claude Code |
1 participant
GoWay Live on goway.to
This makes the live layer ready for production. Production still serves nothing until the owner lists
LIVE_SOURCESon the task definition, along with the other values below. On currentmainthere is noLIVE_ENABLED: live runs when a source is listed. Design, measurements and the full checklists are indocs/MAP_LIVE.md→ Production.1. One poller across API tasks (
packages/backend/src/live/cluster.ts)The tasks share GoWay's Valkey (
REDIS_URL, i.e./oxy/_shared/REDIS_URL).SET NX PX 15000, renewed with a compare-and-renew script). Only the holder polls. It publishes each answer (enriched entities, gzip-compressed) on pub/sub, and every other task applies it to its own index and serves its own sockets.sync). If Valkey is down, every task polls on its own and the outage is logged once.cluster.test.tscovers, in memory: three tasks making one upstream request, every task serving the answer, failover after the lease, a late task being synced, local polling while the bus is down, and compact moves.TEST_VALKEY_URLis set. CI now runs avalkey/valkey:8.0service for this.SIGKILLat 11:24:58, and the lease moved at 11:25:14.GET goway:live:lease:*named exactly one holder per source.2. Bandwidth
Per-kind rooms (
LIVE_ROOM_ZOOMS): aircraft and ships use z7, transit z11 (~15 km), bike share z13 (~3.7 km). The client asks for bike share only from z15, where it is drawn.Moves. A vehicle that only moved travels as a
LiveMovetuple (~80 B) instead of the whole entity (~450 B).Packing. Any payload over 2 KB is sent as gzip bytes and inflated by
unpackLivePayload, in arrival order. This is done in the app because Bun's WebSocket server does not negotiate permessage-deflate:socket.iounder Node answerspermessage-deflate, under Bun nothing.Measured on New York at the same moment, old protocol vs new:
This was at 04:20 local time, ~550 vehicles; daytime has about 3× as many.
3. Settings registry
liveis now stable, offered everywhere on the web, and on by default. It is safe to be on by default because the backend is the gate:LiveHostasksGET /live/sourcesfirst, and a 503 means nothing is drawn and no socket is opened.LIVE_SOURCES.live.maxEntitiesquality cap still applies.4. Production config
These are the owner's actions; this PR writes no SSM and changes no AWS.
aws ssm put-parameter --name /oxy/goway/LIVE_SOURCE_MTA_BUS_KEY --type SecureString --value <key>once the key exists.oxy-infraapp-goway.tf:LIVE_SOURCES=renfe-cercanias,fgc-geotren,mta-lirr,mta-mnr,mta-subway,amb-bus,aisa-bus,emt-fuenlabrada,digitraffic-aisLIVE_POLICY), not configuration.REDIS_URL←/oxy/_shared/REDIS_URLLIVE_SOURCE_MTA_BUS_KEY←/oxy/goway/LIVE_SOURCE_MTA_BUS_KEYmemory = "2048". A task holding every source's static GTFS measured 592 MB resident and a 902 MB peak, against today's 1,024 MB.iam-goway-deploy.tf, givegoway_executionssm:GetParameterson/oxy/_shared/REDIS_URLand/oxy/goway/LIVE_SOURCE_MTA_BUS_KEY. It can read onlyDATABASE_URLtoday.GET https://api.goway.to/api/v1/live/sourcesanswers 200.5. Licence readiness
Review fixes (Devin)
isSubscribed(), and a reconnect marks the subscription lost. Before giving up local polling, a task subscribes again on a fresh connection and sends async.LIVE_SOURCESlisted, no other live setting is read, so a badREDIS_URLcan't stop the backend from booting.LiveHostasks again while the backend says 503: after 30 s, 1 min, 2 min, then every 5 min.stop()now always closes the bus, and releases leases only when Valkey answers.Each fix has a test that fails without it.
Independent review fixes
LiveBus.recover).redis-serverfor 10 s and checks that there is one leader again and the follower hears it. It fails on the previous code, which left 2 leaders. CI installsredis-serverand the test passes there in 21.9 s.REDIS_URLstopped the backend booting (fixed). It now logs a warning, never the value, and live runs without a bus.LIVE_VALKEY_URLis removed.Tests
check:openapi,check:contracts,test:gates,check:lockfileandvalidate:agents-mdpass.?live=1: Midtown drew live vehicles from the new protocol, with packed payloads inflated and the panel starting as its pill.🤖 Generated with Claude Code