Describe the bug
The model backend chart (deploy/helm/llm-routing/spark/charts/gguf-backend; #2331 moves it under recipes/) gives the preflight and download Jobs fixed names: <release>-preflight-<target> and <release>-download. After one of these Jobs fails, rerunning the phase applies the same Job name.
- If the Job spec is unchanged, Helm keeps the failed Job and
--wait-for-jobs fails again immediately.
- If the spec changed, for example after a runtime image change, the upgrade fails because a Job's pod template is immutable.
Either way, the Job has to be deleted by hand before the phase can run again. The qualify and chain Jobs avoid this with attempt counters and --retry. #2331 fixes the same problem for the build Job by hashing its inputs into the name.
Steps or code to reproduce bug
- Make
preflight or download fail, for example with a node that lacks free memory or a temporary download failure.
- Fix the cause and rerun the phase.
Expected behavior
A rerun after a failure creates a new Job, for example by recording an attempt number per phase as qualify does, or by deleting the previous failed Job owned by the release before applying.
Additional context
Introduced in #2322. Found during review of #2331.
Describe the bug
The model backend chart (
deploy/helm/llm-routing/spark/charts/gguf-backend; #2331 moves it underrecipes/) gives the preflight and download Jobs fixed names:<release>-preflight-<target>and<release>-download. After one of these Jobs fails, rerunning the phase applies the same Job name.--wait-for-jobsfails again immediately.Either way, the Job has to be deleted by hand before the phase can run again. The qualify and chain Jobs avoid this with attempt counters and
--retry. #2331 fixes the same problem for the build Job by hashing its inputs into the name.Steps or code to reproduce bug
preflightordownloadfail, for example with a node that lacks free memory or a temporary download failure.Expected behavior
A rerun after a failure creates a new Job, for example by recording an attempt number per phase as qualify does, or by deleting the previous failed Job owned by the release before applying.
Additional context
Introduced in #2322. Found during review of #2331.