fix(stargate): expire reservations after backend RTT

[efegokdemir]

TL;DR

Keep each optimistic routing reservation until one clamped RTT of its selected backend has elapsed instead of clearing it on the next registration update. Adds configurable TTL bounds, reservation metrics, and coverage for expiry, updates, release, and backend removal.

Additional Details

An update can briefly double-count a request if Pylon reports it before its reservation expires. Roll out change-driven Pylon stats before deploying this Stargate version.

Three-arm simulator validation

Completed against the discrete-event simulator at NVIDIA/nvcf#2223 head 9e9a86e. Each arm ran three seeds at 80, 160, 240, and 320 offered RPS, with a 60-second warmup and 180-second measurement per scenario. The model uses the five-region topology, one modeled H100-batched worker per backend, and fixed 6,661-token inputs.

Offered RPS	Baseline goodput	Coalesced stats, reservations cleared	Coalesced stats, TTL enabled
80	47.25	43.58	44.03
160	43.93	40.49	40.07
240	43.74	39.45	39.52
320	42.95	39.28	39.08

Values are mean goodput RPS across the three seeds. Mean TTFT p99 (baseline / coalesced-clear / coalesced-TTL) was 13.45 / 14.14 / 13.98 s at 80 RPS, 14.14 / 14.50 / 14.77 s at 160, 14.17 / 14.71 / 14.72 s at 240, and 14.60 / 14.57 / 14.63 s at 320. Timeout counts were 24/1/0, 29/0/0, 33/0/1, and 75/0/0, respectively.

In this model, TTL versus the coalesced-clear arm has mixed, small goodput/latency differences and no clear throughput improvement. The lower goodput versus baseline appears in both coalesced arms, so it is not attributed to TTL alone. These are simulator results, not GPU benchmark measurements; the TTL change is intended to preserve in-flight reservations across stats updates.

The PR remains a draft while maintainers assess this correctness/throughput trade-off.

For the Reviewer

Please focus on reservation expiry and pruning in the routing snapshot lifecycle.

For QA

• cargo test -p stargate --lib (399 passed)
• cargo test -p stargate --bin stargate -- --skip occupied_metrics_port_fails_before_runtime_construction (52 passed)
• cargo test -p stargate --test stargate_integration (149 passed)
• cargo clippy -p stargate --lib --tests -- -D warnings
• cargo clippy --locked -p stargate --all-targets -- -D warnings
• cargo fmt --all -- --check
• git diff --check
• Full cargo test -p stargate has one failure in the unchanged occupied_metrics_port_fails_before_runtime_construction test; it also fails when run alone in this environment.

Issues

Relates to #2284

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[FamousDirector]

Reviewed the reservation lifecycle (reserve, expiry, release, backend removal, cluster generation replacement), the RTT source (per-backend health-check RTT from latest_ready_rtt_or_probe), config validation, metrics, and docs. Expiry logic and pruning look correct; the CAS in deactivate keeps release and expiry from double-decrementing. Two inline findings below.

Minor, outside the inline comments:

• reservations.rs:117 still says a successful attempt "remains pending until its heartbeat". It now remains pending until its TTL.
• runtime-stats-interface.md says "before enabling this Stargate behavior", but there is no enable switch; the behavior is unconditional once this Stargate ships. Consider "before deploying this Stargate version".

Local results at 4cb4ac8: cargo test -p stargate --lib 396 passed, --test stargate_integration 149 passed, --bins 52 passed with 1 failure in the unchanged occupied_metrics_port_fails_before_runtime_construction (matches what the PR description reports).

[efegokdemir]

I have left this in draft because the contribution guide requires a DCO sign-off on every commit. The three commits added on 9 October (cecaf8c, a7a5dbe, 4e5bf58) are authored by jcameron and co-authored by Claude, but do not contain Signed-off-by lines. I cannot sign off those commits on their authors’ behalf. Could you add the required sign-offs or advise the acceptable next step?