feat(llm-routing): add demo metrics and Grafana dashboard

[priyaselvaganesan]

TL;DR

Makes routing-stack health and per-model request metrics visible through a local collector, VictoriaMetrics and a provisioned Grafana dashboard.

Additional Details

New configurations enable monitoring during stack installation. Existing routing stacks can use attach-monitoring, which discovers routing resources independently of model deployments, GPU placement and recipe source pins. Monitoring attachments retain serving configuration and reject deployment commands.

Traffic verification discovers served model IDs or uses --model or monitoring.model. It sends authenticated streaming and nonstreaming chat requests and checks nine per-model metric increases. Model labels retain their actual values.

The dashboard defaults to All models, discovered from gateway/router labels. Its 17 core panels are runtime-independent. Three llama.cpp panels require an explicit compatible exporter target. Other exporters expose raw metrics and scrape health. Traffic checks require chat completions, streaming and token usage.

For the Reviewer

Review monitoring.py, monitoring_setup.py, the chart and dashboard. Default requests total 300 millicores and 896 MiB, with a separate 5 GiB metrics PVC.

Follow the installation, verification and dashboard commands.

Collector Contrib 0.160.0 and VictoriaMetrics 1.153.0 use Apache-2.0. Grafana OSS 13.2.3 uses AGPL-3.0, outside the repository allowlist, and requires downstream license review. PyYAML 6.0.3 is an MIT-licensed test dependency. NOTICE records these artifacts.

This PR targets the branch from #3 so its diff contains only observability work. Retarget it after that deployment recipe merges.

For QA

Passed:

• python3 -m unittest discover -s tests -v: 263 tests, including alternate-model fixtures, multiple discovered models, label escaping, generic attachment and deployment guards.
• Helm lint/render for core and optional-runtime dashboards. Both collector configurations passed validation with the pinned binary.
• Live routing-only attachment and monitoring upgrade.

Final live traffic/metric-delta and All-model dashboard checks require restored VPN access. The cluster currently serves one model. Multiple-model behavior is covered by fixtures.

Issues

NO-REF

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[priyaselvaganesan]

Superseded by NVIDIA#2325, targeting feat/inference-in-a-box. Please continue review there. The source branch is retained.