Retry Temporal test / dev server startup

[philippzagar]

Motivation

TemporalTestServer starts the dev server (and the time-skipping test server) through Temporal Core. The first start on a machine downloads the Temporal CLI. Core then spawns the CLI and gives it a hardcoded 5 seconds to accept connections before killing it and failing (https://github.com/temporalio/sdk-rust/blob/9cb90f5a92c75167fe690a39c1ffd91f71a0fdfd/crates/sdk-core/src/ephemeral_server/mod.rs#L320). On a loaded host, for example CI running several suites that each start their own server, that window sometimes passes before the server is up:

BridgeError(message: "Connection failed: Failed connecting to test server after 5 seconds, last error: Some(TonicTransportError(tonic::transport::Error(Transport, ConnectError(ConnectError(\"tcp connect error\", 127.0.0.1:35923, Os { code: 111, kind: ConnectionRefused, message: \"Connection refused\" })))))", details: nil)

The timeout can't be configured from Swift, as it is hardcoded in the Core SDK, see the following code + TODO in the Rust Core:

// Try to connect every 100ms for 5s
// TODO(cretz): Some other way, e.g. via stdout, to know whether the server is up?
for _ in 0..50 {
    sleep(Duration::from_millis(100)).await;
    ...
}

The other SDKs built on Core (Python, TS, .NET, Ruby) have the same 5 s limit and don't retry. Go and Java start the server themselves and wait up to 60 s.

Modifications

• Retry a failed server start up to 3 times, waiting 250ms between attempts. Core kills the process of a failed start, so each retry spawns a fresh server on a newly selected port. Only the start is retried, never the test body.

Result

Test suites no longer fail when a server misses Core's 5-second window. Starts that fail permanently, like a failed CLI download, now surface after the retries are used up rather than right away. This is a deliberate choice, as we don't want to parse and interpret the String-based error message that we get from Temporal Core. However, all of these permanent failures fail quickly and the retries are therefore retried quickly.

Test Plan

Existing server-dependent tests exercise the real start path.

[FranzBusch]

Generally looks good but the CI isn't happy about this

[philippzagar]

Generally looks good but the CI isn't happy about this

Seems like some dependency declaration stuff that only now surfaces (for whatever reason), let me fix that

[philippzagar]

Generally looks good but the CI isn't happy about this

Seems like some dependency declaration stuff that only now surfaces (for whatever reason), let me fix that

This seems to be the issue, it's from upstream swift-distributed-tracing: apple/swift-distributed-tracing#240

This breaks our CI, especially as we build with warnings-as-errors. Will fix in another PR

[philippzagar]

Fixed here: #199