Often getting failed to update reverse proxy to serve new deployment.

[emanuelefaja]

At least every couple of days I am getting this:

Sending HTTP traffic to next-574-web
Failed to update reverse proxy to serve deployment 574
Deployment failed
Restoring previous deployment
Waiting for port 3000 on next-573-web
Failed to update reverse proxy to serve deployment 573
Stopping service next-574-web

The solution is just to reboot caddy server, and then it works fine after I redeploy, but it just means that I'm not able to just commit and walk away from my computer. I have to babysit the deployment every time.

[gregsadetsky]

Thanks for the report! Could you help us reproduce the error - are you able to make a minimally reproducible setup ie a Dockerfile, a disco.json file, and a minimal set of server side code to see this?

If you could also share what is the stack used, etc. thanks!

[emanuelefaja]

Stack: Vue 3 + Vite SSG app, Node 22 Alpine, served on port 3000. Single web service, no workers. disco.json is minimal:

{
  "version": "1.0",
  "services": {
    "web": {
      "port": 3000
    }
  }
}

What's happening: The deployment succeeds — image builds, container starts, port 3000 is healthy. The failure is specifically in the Caddy admin API PATCH call. From the disco daemon logs:

File "/disco/app/disco/utils/caddy.py", line 171, in query
    return session.patch(url, json=req_body, headers=HEADERS, timeout=10)
requests.exceptions.ReadTimeout: CaddyConnectionPool(host='disco-caddy', port=None): Read timed out. (read timeout=10)

Caddy's admin API doesn't respond within the hardcoded 10-second timeout. The same timeout then hits on the rollback attempt, leaving Caddy's on-disk config pointing at the failed deployment container while the rollback container is actually running — resulting in 502s.

Pattern: Happens every few days. Restarting Caddy (docker restart disco-caddy) fixes it immediately and the next deploy succeeds. This suggests Caddy's admin endpoint degrades over time (possibly related to caddyserver/caddy#4942).

Possible fix: The 10s timeout in caddy.py seems tight — Caddy config reloads can take longer due to OCSP stapling and cert provisioning (caddyserver/caddy#3503). Bumping to 30-60s would likely prevent the failure. A configurable timeout would be even better.

[gregsadetsky]

Thanks, that's super helpful

Just to help debug, how much ram does the machine have? How many other disco projects are running on the machine?

Just to get an approximate idea of whether there are some resource constraints.

Have you run "free" or "top" just our of curiosity to see memory/cpu usage?

If you restart the whole server, does the issue go away for a bit?

Just also trying to see if it's related to the total size of the caddy config or anything else.

[emanuelefaja]

• 100GB+ free RAM and 96CPUs so not an issue on that, we monitor on Grafana and there have been no spikes
• Around 20 disco projects, with the api project itself having 4 container.
• Also checked Caddy config and its not huge

Part of me wonders if its to do with Caddy proxying tens of thousands of requests and thus it takes more than 10s to respond on the API endpoint, but then again the requests themselves are not getting bottlenecked etc so that does not quite add up.

I set up a cron to do a restart of Caddy every so often anyway, but I'll dig further and report back if I find anything else.

[emanuelefaja]

Update: root-caused and fixed on our side. Sharing the investigation in case it helps.

The failures in our setup came down to a stuck ACME retry loop, which exposed a thin race in the deploy flow. Not sure if this matches everyone hitting the issue, but the pattern feels generalizable.

Root cause (two layers)

The trigger — one of our domains (www.<apex>) could not obtain a cert. Our apex is behind Cloudflare. Disco had an apex-www-redirect-<id> route at the origin that still kept www.<apex> in its host list, so Caddy kept trying to issue a cert for it. But LE's HTTP-01 challenges to www.<apex> were 301'd by Cloudflare to https://<apex>/… (a different hostname), so the challenge could never succeed. After enough production failures Caddy fell back to LE staging, which then rate-limited the account (HTTP 429). Retries kept firing every 5–20 min, indefinitely.

How that becomes deploy failures — every ACME retry spikes Caddy's admin-API latency. In disco/utils/caddy.py:165 serve_service() does:

response = session.patch(url, json=req_body, headers=HEADERS, timeout=10)

timeout=10s, no retries. If Caddy happens to be busy with ACME when disco PATCHes the reverse-proxy handler, the call times out → serve_new_deployment raises → "Failed to update reverse proxy to serve deployment N" → rollback.

The rollback flow is where it gets brittle. serve_new_deployment(recovery=True) PATCHes back to the previous container, but in deploymentflow.py around L904-908 errors are silently swallowed (if not recovery: raise). stop_prev_services then tears down the new container regardless. If any of those PATCHes actually succeeded on Caddy's side (they often do — just slowly), autosave.json ends up pointing at the now-stopped new container → 502s until someone restarts Caddy.

That's why docker restart disco-caddy "fixes" it most of the time: Caddy reloads autosave.json, which usually still has the last-good upstream — unless the failed PATCH made it to disk (worst case — the on-disk config ends up pinned to the stopped container).

Fix for our instance

1. disco domains:remove www.<apex> --project <project> — removed the domain.
2. This left an orphaned apex-www-redirect-<id> route behind, because projectdomains.py:remove_domain (L110-117) takes a branch that auto-creates an origin www→apex redirect when you remove www.x while x still exists. That redirect's host list still contained www.<apex> so Caddy kept retrying ACME for it.
3. Called disco's own caddy.remove_apex_www_redirects(domain_id) manually to delete the orphan. ACME retries stopped immediately. Admin API latency dropped. Deploys have been stable since.

Suggestions for hardening (happy to PR any of these)

1. Retry + longer timeout on Caddy admin calls. timeout=10 → timeout=60, wrap in 3x retry w/ backoff. PATCH is idempotent.
2. Verify after PATCH. GET the handler back, assert the upstream matches, fail loudly otherwise.
3. Don't swallow Caddy errors in recovery. At minimum, confirm Caddy is actually pointing at the old container before running stop_prev_services. Right now the recovery path can cheerfully stop the only running container.
4. Audit the remove_domain auto-redirect logic. When www.x is removed while the apex stays, origin-level www→apex auto-creation breaks if DNS for www.x points at a CDN — the cert challenge can never resolve. Either skip it when DNS doesn't resolve to the server, or gate behind a flag. domains:remove also doesn't give an opt-out right now.

Individually (1) is probably the highest-leverage change — it eliminates the race that turns transient admin-API slowness into a deploy failure, regardless of what's causing the slowness.

Happy to provide deeper logs/diagnostics if useful.

[gregsadetsky]

thanks for this debugging - just trying to sort out a few things such as: if / when this might affect other users, how can we test / reproduce it so that we know that we fixed it.

my first question is regarding the cloudflare setup: you say "But LE's HTTP-01 challenges to www. were 301'd by Cloudflare to https:///… (a different hostname), so the challenge could never succeed"

just to understand, if the challenge could never succeed, how does the site end up working?

I understand that the post-failure timing interplay might make caddy point to a container that has been torn down - I'm trying to wrap my head around the patch calls both failing (timing out) but still being applied on the caddy side. at what point in the teardown process could we be aware that the patch timed out but succeeded - one concern is to not fail fast enough when trying to deploy and leave a new deployment hanging

last thought - increasing timeouts only go so far - in a sense, if there's a deeper root cause (as you're describing here), I think we'd rather focus there. I don't want to make perfect the enemy of good (maybe an increased timeout does solve most of the issues), but that would definitely feel like we just pushed the problem forward rather than fixing it

thanks again

[emanuelefaja]

The reason the site works is because Cloudflare is already doing the redirect from www.blue.cc to our https://blue.cc

So all requests are always coming in via apex not via www.

Yeah, my suggestion for one of the PRs was not just to increase timeouts but to make it more robust in terms of rolling back. Basically, to retry back off longer timeouts and then verify as well, and then don't swallow the errors, right? Basically, making that crucial deployment part a little bit more robust, which may benefit other edge cases as well apart from this one.

[gregsadetsky]

Makes sense!

Do you have a sense of how to test this - and the fix - in a consistent way? Have you been able to make a reproducible case?

[emanuelefaja]

I was able to confirm that this was happening for sure:

The trigger — one of our domains (www.) could not obtain a cert. Our apex is behind Cloudflare. Disco had an apex-www-redirect- route at the origin that still kept www. in its host list, so Caddy kept trying to issue a cert for it. But LE's HTTP-01 challenges to www. were 301'd by Cloudflare to https:///… (a different hostname), so the challenge could never succeed. After enough production failures Caddy fell back to LE staging, which then rate-limited the account (HTTP 429). Retries kept firing every 5–20 min, indefinitely.

Right now we have not had any further failures since we made the fix, but will keep an eye out in the next few days, but this was happening several times per day.

I think the cause of the Caddy slowdown is not as important as making disco handle it gracefully, as there could be various other possible causes of Caddy admin API not being responsive in the future, hence my suggestion about possible PRs.

[gregsadetsky]

yeah for sure - I just meant if you had repro code / something we could run to 100% provoke the issue, and then could run again once we put in a fix to make sure we actually fixed it. if you don't, totally ok - we'll try to recreate it.

[emanuelefaja]

With regards to testing this:

1. Have a domain with a www -> apex domain redirect on Cloudflare or similar
2. Setup the domain on disco with apex (will also setup www).
3. Check if you're getting the ACME / Cert issue continuously.

Now for this to then actually become a problem, I suspect you also need a high amount of traffic as well perhaps

[emanuelefaja]

Re-encountered this today after the original ACME loop fix. Different trigger, same failure mode — and this time it's reliably reproducible without any ACME edge case.

Timeline (from Caddy event log, 06:48 UTC):

06:48:01  admin.api  PATCH /id/disco-project-handler-dev-api   ← a dev-api deploy
06:48:01  admin      admin endpoint started                     ← reload begins
06:48:01  http.auto_https / http.log / http                     ← full reload, 35 TLS domains
06:48:11  admin      stopping current admin endpoint            ← 10s later — reload still finishing
06:48:11  admin.api  PATCH /…dev-api/upstreams/0/dial           ← disco rollback PATCH
06:48:28  admin.api  PATCH /id/disco-project-handler-api        ← my deploy 153 PATCH
06:48:38  admin.api  PATCH /…api/upstreams/0/dial               ← rollback (timed out at 10s)
06:50:37  admin.api  PATCH /…api                                ← deploy 154 PATCH
06:50:47  admin.api  PATCH /…api/upstreams/0/dial               ← rollback (timed out again)
06:53:40  shutting down apps                                    ← manual `docker restart disco-caddy`
06:54:29  admin.api  PATCH /…api ← deploy 155 succeeded

Root cause: every PATCH to /id/disco-project-handler-* triggers a full Caddy config reload. On this server (20 projects, 35 TLS-managed domains) the reload takes 8–12s — sitting right at caddy.py:171's hardcoded timeout=10. A dev-api deploy fired ~30s before the production api deploy; they serialized on Caddy, and the second project's PATCH timed out while Caddy was still finishing the first reload's TLS automation init. Failed deploy → rollback → rollback PATCH also timed out → on-disk autosave.json ended up referencing the just-stopped new container → 502s until manual Caddy restart.

Confirmed the ACME fix is still intact:

• 0 ACME / certificate / challenge / rate-limit events in the past 6h of Caddy logs
• tls.automation.policies has no www-redirect entries
• Idle admin API responds in ~80ms

So the ACME loop wasn't the only path into this failure — any concurrent deploys on a busy server with a non-trivial Caddy config can hit the same wall. The reload duration is inherent to how many domains/policies Caddy is managing; it's not a bug we can chase upstream.

Re-raising PR suggestion #1 from the original thread: bump caddy.py:171 timeout=10 → 60, ideally with 3× retry + backoff (PATCH is idempotent). Reproducer is now trivial: trigger two disco deploy calls against different projects within ~30s on a server with 15+ projects.

Happy to PR this if useful.

[gregsadetsky]

thanks again for confirming, and sorry that this keeps happening

I'll follow your instructions to reproduce it (deploy multiple projects and watch the time it takes caddy to update) and then look into the possible mitigation strategies you proposed (timeouts, retries, etc.)

[gregsadetsky]

Hey @emanuelefaja - happy to see you solved this in your Disco-inspired software

I'll re-open the issue here since if it still affects Disco, we'd still want to fix it. if you could please not close issues here (as long as the issue is not closed in Disco), I'd appreciate it!

cheers