feat(chart): expose the worker task watchdog as worker.taskMaxRuntime - #57
Merged
Conversation
Opening an analysis enumerates every governed project from GitLab, and that enumeration is a single worker task. The worker watchdog kills the process if any one task exceeds JOBS_WORKER_TASK_MAX_RUNTIME, which defaults to 3m in the application. The chart had no way to set it, so every deployment inherited the default. On an estate of a few thousand projects that enumeration runs well past 3m: on our test instance 3708 projects took 80 GitLab calls averaging 4.7s, about 6m20 in total. The worker is killed mid-list, the task is recovered and retried, and the analysis is accepted but never leaves zero tasks. It is a restart loop with no error surfaced to the user. The value renders only on the worker deployment, since only the process started with --worker reads it, and is omitted entirely when set to null so an operator can fall back to the application default. Default 20m gives headroom over the measured time rather than sitting just above it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WkEoFQP5bxWXS5qgkYxmu8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
worker.taskMaxRuntime, so the worker task watchdog can be configured from the chart. It previously could not be, and every deployment silently inherited the application default of3m.Why
Opening an analysis enumerates every governed project from GitLab, and that enumeration is a single worker task. The watchdog kills the worker process if any one task exceeds
JOBS_WORKER_TASK_MAX_RUNTIME.On an estate of a few thousand projects that enumeration takes far longer than 3 minutes. Measured on our test instance: 3,708 projects, 80 GitLab GraphQL calls averaging 4.7s, about 6m20 total. The result is a loop:
WATCHDOG: Task exceeded max runtime, process exitsThe analysis is accepted, reports 0 tasks, and never progresses. Nothing surfaces to the user as an error.
What it renders
Only on the worker deployment, since only the process started with
--workerreads it:Setting
worker.taskMaxRuntime=nullomits the variable entirely, so an operator can deliberately fall back to the application default.Default
20m, chosen to give headroom over the measured 6m20 rather than sitting just above it. GitLab response time is the dominant factor and varies by instance, so a tight value would be fragile.Verified
helm templaterenders on the worker only, and--set worker.taskMaxRuntime=45moverrides correctly--set worker.taskMaxRuntime=nullyields 0 occurrenceshelm lintpassesTwo things to check before merging
Chart version. Bumped
1.5.0to1.5.1, since v1.5.0 is already published as a GitHub release and the chart content is changing. I also movedappVersionin lockstep because every previous release in this repo keeps the two identical, though strictly the application version has not changed. Happy to split them or drop the bump if your release automation prefers to own it.PMU run their own fork of this chart, so they need the same value added on their side. This PR fixes it for everyone downstream of ours.