proxy: forward pipelined TLS-terminated requests in the order the client sent them

[iamhuman-cheolheelee]

On a TLS-terminated connection, a request pipelined behind a slow one goes upstream first. Since #561 both share one upstream socket, so under Node, which the CLI runs on, a large response to the overtaking request hangs both. The Node build from just before #561 completes the same exchange.

Root cause

Since #561 each client connection gets a keep-alive agent with maxSockets: 1 (src/sandbox/tls-terminate-proxy.ts:340-343). The agent sends requests in the order https.request is called (:638). The request handler starts forwardUpstream for each request as soon as it is parsed (:237), and each one first awaits filterRequest (:454), SigV4 body buffering (:536) and the vetted address (:626). The request that finishes those first reaches the agent first, so one pipelined behind a slow request overtakes it. The comment at :318-322 says the one socket keeps the client's order, but it only keeps the order in which requests reach the agent.

Under Node, two more things go wrong:

• Hang. The inner server holds a pipelined response back until the one ahead of it has finished. Once the overtaking request's response passes the 16 KiB high-water mark, upRes.pipe(res) pauses and that request never frees the agent's only socket, so the earlier request never gets it and both hang.
• SigV4 reorder. Node emits the next pipelined request before the previous request's body has ended. A request behind one being buffered for SigV4 re-signing therefore overtakes it even with a fast filter; Bun keeps the order here.

Fix

Chain the hand-offs:

• forwardUpstreamGuarded returns its promise, and the request handler passes each request the previous request's promise.
• forwardUpstream awaits that promise just before https.request. Filtering, body buffering and vetting still run concurrently; only the hand-off to the agent is ordered.
• The turn passes when forwardUpstream settles, i.e. once the request is handed to the agent or dropped. The guard's .catch turns a throw into a settle, so no path leaves the next request waiting.
• A request whose client went away while it waited is dropped, as after the dial today.
• The comment at :318-322 now says where the order comes from.

Rejected alternative: holding the turn until the response has finished. The agent's single socket already serialises the responses, and every deny, error and stale-socket path would have to release the turn.

Edge cases covered

• First request denied: the 403 is written and the next request goes upstream.
• Client goes away while the first request is being filtered: the requests behind it are dropped, not sent, and the proxy keeps serving new connections.
• Stale kept-alive socket under the first request: the client connection is closed as in perf(proxy): keep the TLS-terminate upstream connection alive per client connection #561, and nothing is left waiting.
• Request behind one buffered for SigV4 re-signing (Node).
• Large response to a request behind a slow one (Node): both requests used to hang; now both complete.

Testing

• New test/sandbox/tls-terminate-pipelining.test.ts, 6 tests. The two Node cases run the built dist/ proxy in a node child, because neither shows under Bun. CI builds before npm test.
• On main, 4 fail and 2 pass (the denied and stale-socket cases also pass on main). The failures:
  • the two requests arrive in reverse order;
  • the request behind the aborted one is sent upstream;
  • the SigV4 pair arrives in reverse order;
  • the large-response case gets 0 of 2 replies within 3 s.
• With the fix, all 6 pass. The before and after results are the same on macOS arm64 (Node 26.8.1 and 22.23.3) and in a Debian 12 arm64 container (Node 24.21.0 and 20.20.2), with bun 1.4.2.
• CI commands run locally on macOS, all passing: npx eslint ., npm run typecheck, npm run build:java-agent, npm run build, npm test (1204 pass, 689 skip, 0 fail), node test/utils/which-node-test.mjs.

Notes

• network: drain a denied upload before tearing the socket down #558, fix: keep Content-Length for body substitution until length actually changes #568 and fix(proxy): gate request-body inspection on framing headers, not method #421 also edit this file, but on different lines; this branch merges cleanly with network: drain a denied upload before tearing the socket down #558 and fix: keep Content-Length for body substitution until length actually changes #568. fix: keep Content-Length for body substitution until length actually changes #568's new body-buffering await comes before the new ordering point, so it is ordered as well.
• AI-assisted: I used Claude Code while investigating; I reviewed, ran and verified every change myself.

cc @dylan-conway

🤖 Generated with Claude Code