feat(live): worldwide providers as data — 73 feeds, 14 countries, licence-checked

[NateIsern]

Stacked on #194 (Renfe regional/long-distance), which is stacked on #168 (production readiness). Merge those first. Don't merge before them.

What

There are now 73 real-time feeds in 14 countries. Each one is a data entry (LiveProvider in packages/backend/src/live/providers.ts) rather than per-city code. One adapter per standard builds them:

Format	Adapter	Entries
GTFS-RT VehiclePositions	adapters/gtfsRealtime.ts	69
GTFS-RT TripUpdates (estimated along shapes)	adapters/arrivalEstimates.ts	0 new (the only candidate's licence was refused)
GBFS	adapters/gbfs.ts	4
SIRI-VM, XML and JSON	adapters/siriVm.ts (new)	0 (every SIRI feed found also publishes GTFS-RT)

• 51 feeds need no key. The other 22 stay off until LIVE_SOURCE_<ID>_KEY is set. Where to register for each key is in docs/MAP_LIVE.md under "Keys to register".
• The catalogue was seeded from the Mobility Database (feeds_v2.csv) and from the operators' own portals.
• A feed gets in only with a licence that was read. Every source now has a licenceCheck: the date, the page and the verbatim words that allow public commercial redistribution. licence.test.ts fails a source that:
  • is missing them;
  • has a credit, licence, name or id the public contract would refuse;
  • or has a key in its URL.
• 34 candidates were excluded for their licence, either refused or not verifiable. 18 more have no usable feed. All are listed with reasons in the doc.
• The single-poller and demand-driven rules are unchanged. A feed is polled only while someone watches its box, by the lease holder, at an interval inside the provider's published limit:
  • TransLink allows 1,000 requests a day, so it is polled every 90 s.
  • The NTA allows one request a minute.
  • Trafiklab's quota is shared by four operators, so each is polled every 60 s.
• LIVE_SOURCES=all takes the whole licence-checked catalogue. It skips:
  • keyed sources that have no key (one log line);
  • sources still awaiting a permission (adsb-lol, nyc-bikeshare, york-yrt). Name one beside all once its permission is in, e.g. all,adsb-lol.

Correctness found along the way

• Modes come from the operator's routes.txt. A position names a route, not a mode, and Helsinki or Prague put every mode in one feed. Static GTFS is read at the depth each network can afford, measured with the hub's own parser:
  • full where it is small;
  • only: 'routes' or only: 'trips' where shapes would cost GBs (Prague 2.2 GB, Helsinki 1.1 GB, Austin 1.7 GB resident while parsing);
  • big archives by HTTP range, fetching only the directory and routes.txt. The Netherlands' 259 MB file costs 236 KB and Norway's 534 MB file 250 KB. Range requests send accept-encoding: identity because TransLink Queensland gzips on the fly and would otherwise range over its gzip stream.
• A static GTFS that has gone unneeded for 15 minutes is released, so memory stays bounded by what people are actually watching.
• ZIP64 sizes are now read (Nice's GTFS).
• A feed that carries several modes defaults to mode other. Such a vehicle is drawn as a neutral marker with no 3D body, never guessed into a bus (kinds.ts, points.ts).
• A route id is shown as the line only when riders use it (routeIdIsLine). Before, lineOf: () => null fell back to the route id, which put ids like 100001 and OST:Line:1_400 on the map. That includes feat(live): Renfe's regional and long-distance trains beside Rodalies #194's Renfe LD trains, fixed here.
• GTFS route types 1100–1199 are air services. They were mapped to ferries: Entur's 101 Avinor routes would have been drawn as boats.
• Route ids are now looked up as ids before short names, where another route's short name could shadow them.
• One Accept header that every protobuf server answers. Trapeze servers (Edmonton, Halifax, Mississauga) returned 406 to the old one, and Adelaide returned JSON.

Evidence

• Every keyless entry was polled once through its own adapter on 2026-10-12 around 21:00 UTC. Counts are in the coverage table, e.g.:

  • Toronto 1,282, Brisbane 1,264, Rio 1,152, the Netherlands 1,478, Prague 834;
  • Oslo bike share 267 stations.
  • The exception is Hamilton: its host did not resolve from here, though its licence is verified.

• End-to-end hub smoke with LIVE_SOURCES=all (60 sources configured) and viewers in six cities:

  City	Vehicles	Modes
  Amsterdam	367	240 bus, 126 tram, 1 metro
  Oslo	27	9 bus, 18 rail
  Boston	450	338 bus, 48 tram, 28 rail, 26 subway, 10 ferry
  Brisbane	1,273	1,184 bus, 75 rail, 14 ferry
  Helsinki	145	107 bus, 26 tram, 8 rail, 1 ferry, 3 not placed by the static file

  Lines and colours came from each operator's static file. The process was 97 MB resident.

• Tests. Each fix's test was checked to fail without the fix:

  • licence.test.ts: every source's licence evidence, contract shapes, and no keys in URLs.
  • providers.test.ts:
    • GTFS-RT with provider headers and the new Accept header;
    • keyed sources by header or query;
    • GBFS with Client-Identifier;
    • format → adapter;
    • route id as line;
    • all versus awaiting.
  • siriVm.test.ts: XML and JSON.
  • staticGtfs.test.ts: ZIP64, ranged reads (the whole archive is never fetched; falls back when the server ignores ranges), routes-only and trips-only depth, route ids before short names.
  • hub.test.ts: ranged static read through the hub, unencoded, giving line and mode; static GTFS released after 15 minutes idle and re-read on return.
  • combined.test.ts: route types, including air ≠ ferry.
  • Frontend points.test.ts: other is a neutral marker with no 3D body.

• Backend 1,197 pass with a real Valkey (TEST_VALKEY_URL); SDK 207; frontend 1,639. bun run lint, bun run typecheck, check:openapi and check:contracts are clean.

Owner actions (nothing here writes SSM or touches AWS)

1. Set LIVE_SOURCES=all (see docs/MAP_LIVE.md → Configuration). This is in place of feat(live): production — one poller across tasks, a fraction of the bytes, on by default #168's list. The REDIS_URL and 2,048 MB memory changes from feat(live): production — one poller across tasks, a fraction of the bytes, on by default #168 still apply.
2. Register the keys listed under "Keys to register", then store each as SSM /oxy/goway/LIVE_SOURCE_<ID>_KEY in the task's secrets and the execution role's ssm:GetParameters:
   • NTA Ireland;
   • Trafiklab (ask for Silver);
   • TransLink Vancouver;
   • STM;
   • OC Transpo;
   • Metra;
   • TriMet;
   • TfNSW;
   • Victoria DTP;
   • Auckland Transport;
   • Metlink;
   • Christchurch Metro.
3. File York Region's developer form (https://forms.yrt.ca/YRT-GTFS-Data), then add york-yrt beside all.
4. Optional: confirm that the UK Bus Open Data Service's data licence (OGL v3) covers its feeds. Its licence page refused our probe. It would become one entry.

🤖 Generated with Claude Code