Skip to content

feat(llm-routing): add a two-Spark GLM deployment recipe - #3

Closed
priyaselvaganesan wants to merge 20 commits into
mainfrom
feat/spark-llm-deployment
Closed

feat(llm-routing): add a two-Spark GLM deployment recipe#3
priyaselvaganesan wants to merge 20 commits into
mainfrom
feat/spark-llm-deployment

Conversation

@priyaselvaganesan

@priyaselvaganesan priyaselvaganesan commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

TL;DR

Makes GLM-5.3 deployment across two DGX Sparks and gateway/router development repeatable using the shared stack charts.

Additional Details

  • Builds ARM64 images from the current checkout, prepares Helm dependencies automatically, and downloads verified model weights.
  • Discovers existing installations and manages temporary API keys for chat, streaming and authentication checks.
  • Supports gateway/router image updates and rollback while GLM stays loaded.
  • Adds per-endpoint Pylon canary timing and chart-validation fixes.
  • Reports concise installation results, preserves detailed tool output in private logs, and directs failed steps to the relevant checks.
  • Allows init after uninstall to reuse saved placement, image references, credentials and retained PVCs. It archives stale progress after checking release, workload and retained-resource ownership.

llama.cpp attribution is included in NOTICE. Model and container artifacts retain their upstream license terms.

For the Reviewer

Start with the deployment README and spark/spark.py. Uses the merged stack and gateway changes.

For QA

  • Live replay at ba025b9c6: fresh installation/download, two-GPU inference, TLS/authentication, discovery, streaming, recovery, gateway update/rollback and teammate onboarding passed. GLM used a 2048-token context and one request slot.
  • Latest checks: 213 recipe/client tests, 7 backend tests and 8 offline chart configurations passed.
  • Read-only reinitialization checks passed against the retained live resources. The tested patch was applied to the demo checkout.
  • Running init and repeating the installation sequence remains operator validation.
  • Replay required an isolated Docker client configuration and Helm reconciliation after connection interruptions.
  • CI remains blocked by missing release tags and operator CI image access.

Issues

NO-REF

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@priyaselvaganesan
feat(llm-routing): add a two-Spark GLM deployment recipe
9b9c0af 
Use the merged gateway, request router and Pylon Operator charts with
source-matched image updates, model discovery checks and rollback.

Pin llama.cpp f872b591121761ac7b2af18283bd99bdc092a63a (MIT) and the
external GLM-5.3 UD-IQ2_M checkpoint at revision
346b3591c7f28d1a23716f97a065ecf12ec14771 under its upstream model terms.
Record external artifact terms in the recipe and generated root NOTICE.

Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
feat(pylon-operator): expose endpoint canary timing
46a41c4 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
refactor(llm-routing): consume native stack source fixes
b55c7b4 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
priyaselvaganesan
priyaselvaganesan commented Oct 2, 2026
View reviewed changes
Comment thread deploy/helm/llm-gateway-stack/scripts/check-render.sh

@priyaselvaganesan priyaselvaganesan Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed YAML extraction and annotation matching errors that caused chart validation to fail during deployment preparation

@priyaselvaganesan
docs(llm-routing): clarify deployment verification
b0699c4 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
feat(llm-routing): simplify Spark setup and maintenance
ba025b9 
Discover existing installations from kubeconfig and generate configuration for
new K3s installs. Manage temporary caller keys for gateway verification and chat
so teammates can test the running stack without sharing an API key.

Import application image archives through Kubernetes, support Helm 4, and
resume a completed model load after validating its live identity. Simplify the
installation and maintenance guides around these commands.

Validation: 166 recipe tests, backend tests, eight Helm configurations, and live
existing-install update/rollback plus fresh two-Spark installation, model
download, inference, streaming, authentication, worker recovery, and teammate
onboarding. Image builds used an isolated Docker client configuration.
@priyaselvaganesan
fix(llm-routing): reuse saved installation when attaching
2fd899a 
A new checkout shares the context-based work directory with an earlier
installation. Allow attach-existing to validate and reuse that completed
installer state instead of requiring another work directory.

Check live source, release ownership, nodes, runtime and image repositories.
Preserve saved checkpoints, credentials and installer capabilities, while
accepting image tags changed through the update workflow.

Validation: 176 recipe tests, eight Helm configurations, and two successful
live attachments with saved configuration, state and CA files unchanged.
@priyaselvaganesan
fix(llm-routing): build from the current checkout
c920a5f
@priyaselvaganesan
fix(llm-routing): prepare chart dependencies automatically
1ec0e9f
@priyaselvaganesan priyaselvaganesan self-assigned this Oct 5, 2026
@priyaselvaganesan priyaselvaganesan mentioned this pull request Oct 5, 2026
Closed
4 tasks done
@priyaselvaganesan
fix(llm-routing): reinitialize an uninstalled demo with retained storage
b326f6d 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
fix(llm-routing): report concise installation results
962d451 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): document uninstall and retained-data reinstall
acac300 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): separate optional setup from the demo path
6ae14e6 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): keep initial setup focused on fresh installs
e708a97 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): clarify automatic checks and troubleshooting
00c31bf 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): group runtime image mirrors with optional setup
ac6a43f 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): place retry guidance with qualification
5e78186 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): place optional configuration after maintenance
b8ce1f9 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): fix uninstall context and optional releases
372798d 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
docs(llm-routing): move gateway diagnostics to troubleshooting
eee817e 
Signed-off-by: priyaselvaganesan <pselvaganesa@nvidia.com>
@priyaselvaganesan
feat(llm-routing): provision a separate demo ui api key
efd383f
@priyaselvaganesan priyaselvaganesan mentioned this pull request Oct 6, 2026
Merged
4 tasks done
@priyaselvaganesan

priyaselvaganesan commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Replaced by NVIDIA#2322, targeting feat/inference-in-a-box. Review discussion continues there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@estroz estroz Awaiting requested review from estroz
@along-2017 along-2017 Awaiting requested review from along-2017

Assignees

@priyaselvaganesan priyaselvaganesan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@priyaselvaganesan