Skip to content

Pool connect retries a refused connection but gives up instantly on one dropped mid-handshake #4396

Description

@lancespeelmon

I have found these related issues/pull requests

Searched for connect-phase retry and pool backoff issues; found nothing covering this. #3315 is ECONNRESET on an established connection under load, which is a different path. Companion to #4395, which is why an EOF at the handshake currently arrives as Protocol rather than Io - the two interact, see below.

Description

PoolInner::connect (sqlx-core/src/pool/inner.rs) classifies connect failures as:

// an IO error while connecting is assumed to be the system starting up
Ok(Err(Error::Io(e))) if e.kind() == std::io::ErrorKind::ConnectionRefused => (),
// We got a transient database error, retry.
Ok(Err(Error::Database(error))) if error.is_transient_in_connect_phase() => (),
// Any other error while connection should immediately terminate
Ok(Err(e)) => return Err(e),

Exactly two shapes are retried. Every other transport failure - EOF mid-handshake, ConnectionReset, a socket timeout - returns on the first attempt with acquire_timeout barely touched. So two situations that are identical from the caller's point of view get opposite treatment:

server behaviour error pool behaviour
nothing listening PoolTimedOut retried for the whole acquire_timeout
accepts, then closes Protocol(..) returns after ~860 µs

The second is what a proxy, load balancer or port forwarder does when it sheds a connection under load - a textbook transient failure, and precisely the one the loop refuses. With a 30 s acquire_timeout, roughly one connection in twelve thousand died in transit for us and failed its request outright instead of being redialled.

Reproduction steps

use std::time::{Duration, Instant};
use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;

async fn eof_server() -> u16 {
    let l = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let port = l.local_addr().unwrap().port();
    tokio::spawn(async move {
        loop {
            let (mut s, _) = l.accept().await.unwrap();
            let mut b = [0u8; 8];
            let _ = s.read_exact(&mut b).await;
            drop(s);
        }
    });
    port
}

async fn refused_port() -> u16 {
    let l = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let port = l.local_addr().unwrap().port();
    drop(l);
    port
}

async fn go(port: u16) -> (Duration, String) {
    let t = Instant::now();
    let e = sqlx::postgres::PgPoolOptions::new()
        .max_connections(1)
        .acquire_timeout(Duration::from_secs(5))
        .connect(&format!("postgres://u:p@127.0.0.1:{port}/d"))
        .await
        .map(|_| "Ok".into())
        .unwrap_or_else(|e| format!("{e:?}"));
    (t.elapsed(), e)
}

#[tokio::main]
async fn main() {
    let (d, e) = go(eof_server().await).await;
    println!("accept-then-close  -> {e} after {d:?}");
    let (d, e) = go(refused_port().await).await;
    println!("connection refused -> {e} after {d:?}");
}

Output:

accept-then-close  -> Protocol("unexpected response from SSLRequest: 0x00 ...") after 860.292µs
connection refused -> PoolTimedOut after 5.002314833s

Suggested direction

Retry the transport kinds that describe a connection which did not survive being set up - ConnectionReset, ConnectionAborted, UnexpectedEof, BrokenPipe, TimedOut alongside today's ConnectionRefused. Deliberately not every io::Error: a bad host or failed lookup will not fix itself, and retrying to the deadline would report PoolTimedOut in place of the error that explains the failure.

Note this needs #4395 too - an EOF at the SSLRequest handshake is currently Protocol, not Io, so it would not match a widened Io arm on its own.

One question for maintainers before I open a PR: this changes an observable timing, so under the Hyrum's Law note in the PR template it is plausibly a breaking behaviour change and would want 0.10.0. If you would rather have it in 0.9.x, it can go behind a PoolOptions opt-in defaulting to today's behaviour instead. Happy either way - I have both.

SQLx version

0.9.0 (also 0.8.6; the code is identical on main at 1d15be8)

Enabled SQLx features

runtime-tokio, tls-rustls-ring, postgres

Database server and version

Postgres 18 - though no server is involved, the connection never reaches one

Operating system

macOS 15 (not OS-specific)

Rust version

rustc 1.94.1 (e408947bf 2026-03-25)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions