Stop pinning the Synapse container's address on the boxel Docker network

[habdelra]

The matrix Playwright suite runs four containers on a shared Docker network called boxel: smtp4dev, the mock OIDC upstream, a Caddy proxy in front of it, and Synapse. Global setup starts them in that order, and only Synapse asked for a specific address on the network — 172.20.0.5. The other three took whatever Docker handed out.

Docker allocates addresses on a user-defined network in the order containers join it. On an otherwise empty network that works out exactly: smtp .2, upstream .3, proxy .4, and .5 is still free when Synapse asks for it. One more tenant anywhere on that network shifts every allocation up by one, the Caddy proxy takes .5, and Synapse cannot start:

docker: Error response from daemon: failed to set up container networking: Address already in use

That reads like a host port conflict. It is the container's address, and the suite dies before a single test runs.

The observability stack is what makes this routine locally — its Prometheus joins boxel to scrape Synapse's metrics, sits at 172.20.0.2, and restarts on its own, so a reboot does not clear it. The same mechanism hits mise run start-synapse, which takes the same fixed-address path as the suite.

The shape of the fix

Nothing reaches Synapse by address. The host reaches it through the published port (localhost:8008); containers on the same network reach it by container name through Docker's embedded DNS, which is exactly how the Prometheus config above names its scrape target — boxel-synapse:9001. The pinned address bought nothing and cost the suite its independence from whatever else was on the network, so the request is gone.

With it goes the last difference between the two start paths. The fixed-port path and the dynamic-host-port path (used where several harnesses share a host) now build identical flags — publish the port, join the network, let Docker allocate — so the branch collapses into one synapseDockerParams function.

The network's --subnet=172.20.0.0/16 was pinned to make 172.20.0.5 meaningful and goes with it. Requesting a subnet is not free: Docker refuses to create a network whose range another network already holds (Pool overlaps with other one on this address space), and a boxel network created any other way — docker network create boxel, which packages/observability/docker-compose.yml tells you to run when bringing that stack up first — gets a different range, in which the pinned address is not merely taken but invalid.

Two adjacent repairs

Both are part of the same reported reproduction.

An abandoned container no longer poisons the next run. Synapse containers are named after the temp config directory they are given, so a run killed before its teardown leaves one behind under a name no later run can predict — still holding the host port the next run needs.

Nothing about the container itself identifies it as debris. An abandoned Synapse is running and healthy, on the same port and under the same name shape as one whose suite is mid-run — and both the software-factory harness and the Playwright suite start containers under that shape. Neither name nor port can tell them apart.

The owning process can. Each container carries its owner's pid and the host that issued it, and only those whose owner has exited are swept. The host travels with the pid because a pid is only answerable from the machine that issued it — two processes in different namespaces against the same Docker daemon number theirs independently, so a foreign pid read locally names an unrelated process or none. Every unreadable answer leaves the container alone — an owner that does not parse, or a pid that is alive only because it was reused — because declining to sweep costs a legible port-conflict message on the next start, while sweeping a live run destroys it. The dev Synapse is excluded by name: it outlives the process that starts it, and callers meaning to replace it stop it explicitly.

The realm test harness kept a second sweep of its own — every sf-test-synapse-* container, removed by name with no liveness test — so starting the factory support services destroyed a running Playwright suite's homeserver, and a concurrent harness's, over containers neither contended for. It now calls the same function rather than carrying a second definition that can drift from this one.

A refused bind now names what it collided with. Docker's message declines to say which address is in use. A bind failure is now reported as the host port and whatever publishes it:

Could not start Synapse: Host port 8008 is already published by: boxel-synapse (matrixdotorg/synapse:v1.126.0). Docker reported: ...

The three answers stay distinct: containers holding it, nobody holding it (so a host process is), and Docker not answerable at all — a failed query is never reported as a host process.

Verification

Reproduced and re-checked against the real containers with Prometheus holding 172.20.0.2, running the container-startup half of global setup:

• Before: RESULT: FAILED — ... failed to set up container networking: Address already in use
• After: Synapse starts at 172.20.0.3, passes its health check, and serves http://localhost:8008

And the three behaviors the change rests on, checked directly:

• A peer container on boxel resolves the Synapse container by name and gets OK from /health on port 8008 — the path Prometheus uses.
• With a suite mid-run holding port 8008 and debris from a killed run alongside it, a dev-stack Synapse start sweeps the debris, leaves the live suite running, and fails with Host port 8008 is already published by: sf-test-synapse-… (matrixdotorg/synapse:v1.126.0) — the loud failure that path has always had, now naming what it collided with.
• A squatter on host port 8008 produces the message above instead of the unattributed one.

packages/matrix/tests/synapse-container-networking.spec.ts covers the flag construction directly, so a reintroduced address pin fails a test rather than waiting for a machine that happens to have a neighbour on the network. pnpm lint and the shard-assignment test pass.

Scope

The ticket asks whether this is also behind the Synapse container flakes seen in CI shards. It does not appear to be: no workflow puts anything else on the boxel network, so a fresh runner allocates the addresses in exactly the order the pinned one assumed. This removes the dependency regardless, but it should not be expected to move those shards.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T13:18:50.903202Z	d2ba0c4	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2ba0c4347

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

🟡 Changes recommended

The new bind-conflict messaging path can misdiagnose failures when docker ps itself errors (currently indistinguishable from “no container publishes the port”), and should handle/report Docker query errors explicitly.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR removes the hard-coded Synapse container IP (and the fixed boxel Docker network subnet) from the Matrix Playwright harness, relying instead on published host ports and Docker DNS by container name to avoid address-allocation collisions on shared networks.

Changes:

• Refactors Synapse container startup to build a single set of docker run flags via synapseDockerParams, removing the fixed --ip=... behavior.
• Updates network creation to avoid pinning a subnet, preventing “pool overlaps” failures when Docker already has a conflicting range allocated.
• Adds a targeted Playwright unit test file to lock in the “no fixed IP” invariant and basic flag/message behavior.

File summaries

File	Description
packages/matrix/tests/synapse-container-networking.spec.ts	Adds tests covering Synapse docker flag construction (no --ip, port publish, volume mount, root behavior) and basic conflict-message content.
packages/matrix/support/synapse/index.ts	Removes fixed-IP behavior, centralizes docker param construction, adds abandoned-container cleanup, and improves bind-conflict error messaging.
packages/matrix/support/docker.ts	Makes dockerCreateNetwork avoid specifying a subnet and documents why subnet pinning is risky.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[backspace]

[Claude Code 🤖] This pass went after convergence with the three earlier threads, and whether anything still reaches Synapse by address or reads the removed SynapseConfig.host. It also checked the two claims the new code rests on: that docker ps --filter publish= selects by the host-side port (moby's includeContainerInList compares PublicPort; confirmed against a local Docker 29 daemon), and that on all three start paths the process holding the owner-pid label outlives the container's use (the matrix suite's global setup, the software-factory serve:support child, the same-process harness API).

No blocking issues. Approving. Two non-blocking notes inline.

The Codex thread's reply describes a sweep keyed on the name prefix plus publish=<port>; head replaces that with the owner-pid sweep, which also covers the case that thread raised, so a one-line follow-up there would keep the thread from describing a mechanism the code no longer has.

1. processIsAlive assumes owner and sweeper share a pid namespace — pin the assumption in a comment or add a host label (see the comment on processIsAlive in packages/matrix/support/synapse/index.ts).
2. Test title wording (see the comment in packages/matrix/tests/synapse-container-networking.spec.ts).

Adjacent, out of scope: in environment mode synapseStart names the suite's container boxel-synapse-<slug>, so the sweep's sf-test-synapse- filter never reaches it. Same-slug leftovers are stopped by name in the stopExisting block; leftovers from other slugs accumulate on dynamic ports. Pre-existing, and unaffected by this change.

[backspace]

[Claude Code 🤖] process.kill(pid, 0) answers for the sweeper's own pid namespace, so the "every unreadable answer spares the container" rule has one gap: an owner that started the container from a different namespace against the same daemon — a harness inside a devcontainer on the host's Docker socket, swept by mise run start-synapse on the host, or the reverse. Its pid number is meaningless there and usually unused, which reads as "owner exited" and removes a live run.

If that setup is out of scope, a sentence here saying the sweep assumes owner and sweeper share a pid namespace pins the assumption. If it is in scope, a second label carrying os.hostname() lets a mismatched host count as another unreadable answer:

'--label',
`${SYNAPSE_OWNER_HOST_LABEL}=${os.hostname()}`,

with abandonedContainerIds sparing any line whose host is not its own.

Non-blocking; a decision rather than a fix.

[habdelra]

[Claude Code 🤖] Closed rather than documented, in d3cc9b2 — it is the same rule I stated, applied to a case I had not considered, so leaving it as a written assumption would have left the one outcome this whole change exists to prevent.

Containers now carry boxel.synapse-owner-host alongside the pid, the query filters and formats both, and abandonedContainerIds takes the sweeper's own host and spares any line that does not match. A foreign host is simply one more answer the sweep cannot read, joining an unparseable owner and a reused pid.

Verified against real containers, including one labelled for another host:

[live run spared]        true
[debris swept]           true
[foreign host spared]    true
[unlabelled spared]      true

The last line is the transitional case: a container from before the labels exists with neither, so the query's label= filters never list it and it is left alone.

[backspace]

[Claude Code 🤖] Nit, non-blocking: "this harness own" is missing its possessive. Rephrasing sidesteps the apostrophe-in-single-quotes formatting question:

Suggested change

	test('the sweep asks only about this harness own containers', () => {
	test('the sweep asks only about containers this harness started', () => {

[habdelra]

[Claude Code 🤖] Taken as suggested in d3cc9b2. The possessive went missing when I stripped an apostrophe out of a single-quoted test name; your rephrasing reads better than what I was trying to write anyway.