fix(stargate-bench): measure validated stream output

[barrygreengus]

TL;DR

Benchmark success and throughput must reflect completed inference output. Empty or truncated HTTP 200 responses previously counted as successful, requested token limits inflated generated output, and native mock streams omitted usage.

This change validates SSE completion and records observed completion tokens separately from requested limits. Missing external and legacy usage stays unavailable; a reported zero remains available.

Additional Details

• Decode standard SSE framing with sse-core, bound buffered fields, reject malformed JSON and application errors, and finish at [DONE].
• Measure first output from content, reasoning or refusal text. Empty role chunks and keepalives do not establish output timing.
• Honor stream_options.include_usage in mock-dynamo. A five-token context with a two-token prompt reports three generated tokens even when the request permits 100.
• Score throughput and backend/cluster output shares from observed counts. A null standard count permits the cumulative fallback. Saturating storage prevents overflow; unrepresentable totals make throughput and shares unavailable.
• Expose Rust benchmark tests and a native mock-to-driver executable integration through Bazel.

Customer Release Notes

Stargate benchmarks reject incomplete success responses and report observed output usage. Native mock streams provide actual token counts when requested; missing external usage remains unavailable.

Plan Summary

Not applicable.

Usage

Request results add observed_output_tokens. Existing output_tokens retains the requested workload limit. The driver requests streamed usage automatically; direct mock requests may enable stream_options.include_usage.

Dependencies

Added sse-core 0.2.3 with default features disabled and only std enabled. Its enabled dependencies were already present; no other package versions change. Independent audit verified the published Apache-2.0 license, allowlist compatibility, maintained source, copyright attribution, archive checksum and Cargo/Bazel metadata. Benchmark NOTICE and the regenerated root NOTICE index match. Independent license and attribution verification passed. Exact 842e500 license and generated-document CI passed.

For the Reviewer

Review stream validation and usage selection in driver.rs, count availability and aggregation in score.rs, and native mock framing in mock-dynamo/src/openai.rs.

Independent main-source probes reproduced empty/truncated stream success, empty role timing, missing refusal timing and missing usage. The current-main native mock also omits usage when requested. Both original review findings were reproduced: null completion usage masks a cumulative fallback, and two u64::MAX observations panic. Regressions also cover unavailable overflow metrics across separate backends.

This PR targets main after #1818, #1820 and #1823 merged. All ten benchmark patches were independently verified unchanged after history reconciliation. Current main 31f98a9 was merged normally; the final diff contains only this 15-file benchmark layer. Both actionable review threads are resolved.

For QA

Native validation on 51ea4f8 passes 164 benchmark and 36 mock tests, with zero failures or ignored tests, plus executable integration and all-target Clippy with warnings denied. Two registration metric regressions also pass. Current 842e500 has identical Stargate source and dependency metadata; these results apply to its unchanged code. Integration and scoped formatting pass again. Integration requests 100 tokens and records three observed tokens. Workspace-wide formatting reports existing differences in three unchanged Stargate files. Whitespace and fanout pass.

Pinned TruffleHog 3.95.2 scanned the complete history from the earlier common fork: 382 chunks, 239452 bytes, zero results and exit 0. Local Bazel setup was blocked before tests; Linux CI provides Bazel validation. Prior 6e4b22c/mainc55bea3f CI passed all 27 entries and all 11 required/relevant jobs. Current 842e500/main31f98a9e passes all 27 CI entries; all 11 required and relevant jobs explicitly succeed, including root/Stargate Bazel, Rust CodeQL, secrets, license/NOTICE and generated dependency docs. The latest main update changes only the separate rate limiter; Stargate remains unchanged.

No GPU or live-cluster campaign was run. Additional performance QA is not needed for these correctness changes. The benchmark artifact and standard mock usage fields require no NVCF CLI change.

Issues

Relates to #1817

Related Pull Requests

#1818, #1820 and #1823 merged. This final layer targets main.

References

• sse-core published metadata
• sse-core source license

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/nvcf/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 591337f7-f2a7-4bbf-9e90-d3c6e4845784

📥 Commits

Reviewing files that changed from the base of the PR and between 067fe60 and c0ba447.

⛔ Files ignored due to path filters (2)

• MODULE.bazel.lock is excluded by !**/*.lock, !**/MODULE.bazel.lock
• src/libraries/rust/stargate/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (1)

• dependencies.md

🚧 Files skipped from review as they are similar to previous changes (1)

• dependencies.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The mock chat service can emit streamed usage metadata. The benchmark driver validates SSE responses and records observed output tokens. Scoring uses observed tokens for output metrics and suppresses those metrics when usage is missing or totals overflow.

Changes

Stargate benchmark accuracy

Layer / File(s)	Summary
Mock streaming usage contract src/libraries/rust/stargate/crates/mock-dynamo/src/openai.rs, src/libraries/rust/stargate/crates/mock-dynamo/src/tests.rs	Chat requests accept stream_options.include_usage. Enabled streams emit a final usage-only chunk before [DONE]. Tests cover absent, false, and true settings.
Validated SSE request execution src/libraries/rust/stargate/Cargo.toml, src/libraries/rust/stargate/crates/stargate-bench/Cargo.toml, src/libraries/rust/stargate/crates/stargate-bench/src/driver.rs, src/libraries/rust/stargate/crates/stargate-bench/BUILD.bazel, src/libraries/rust/stargate/crates/stargate-bench/benchmark_usage_test.py, dependencies.md, NOTICE, src/libraries/rust/stargate/crates/stargate-bench/NOTICE	The driver requests usage, validates text/event-stream responses and SSE events, records observed usage and first-output timing, and requires [DONE]. Bazel targets and an integration test exercise the benchmark. Dependency and license records include sse-core.
Observed output metric aggregation src/libraries/rust/stargate/crates/stargate-bench/src/score.rs, src/libraries/rust/stargate/crates/stargate-bench/src/report.rs, src/libraries/rust/stargate/crates/stargate-bench/src/k8s_run.rs	Successful output totals, throughput, and shares use observed output tokens. Output metrics are unavailable when successful results lack usage or totals overflow.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant benchmark_usage_test
  participant stargate-bench
  participant mock-dynamo
  participant score
  benchmark_usage_test->>mock-dynamo: Start server and wait for readiness
  benchmark_usage_test->>stargate-bench: Run benchmark manifest
  stargate-bench->>mock-dynamo: Request streamed completion with usage
  mock-dynamo-->>stargate-bench: Return SSE content, usage, and [DONE]
  stargate-bench->>score: Provide observed output tokens
  score-->>benchmark_usage_test: Produce benchmark metrics

Loading

Merge Risk: ⚪ Minimal · up to c0ba4

The dependency inventory matches the benchmark's declared dependency. No actionable merge-blocking risk was identified in the selected change; normal validation still applies.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 41.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 43 functions across 8 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title follows Conventional Commits format with the required scoped customer-impact type fix. It accurately describes the primary change: measuring validated streamed output in stargate-bench.

Full details: Docstring Coverage

Explanation

Docstring coverage is 41.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 43 functions across 8 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 5 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-09-11 22:27:38 UTC | Commit: 54b3e6e}

[along-2017]

LGTM

[balajinvda]

🎉 This PR is included in src/libraries/rust/stargate/v0.19.8 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[balajinvda]

This PR is included in version 1.29.2.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 1.69.1.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 1.25.1.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 1.14.2.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 1.10.1.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 0.5.1.

The release is available on GitHub release.

[balajinvda]

This PR is included in version 0.9.1.

The release is available on GitHub release.