Job-level partial rerun: lightweight copies for passed jobs#1095
Open
skipi wants to merge 27 commits into
Open
Job-level partial rerun: lightweight copies for passed jobs#1095skipi wants to merge 27 commits into
skipi wants to merge 27 commits into
Conversation
skipi
force-pushed
the
mk/job-rerun/proto-job-lineage
branch
from
July 8, 2026 11:25
35c7ef0 to
17e7fa9
Compare
skipi
force-pushed
the
mk/job-rerun/job-level-partial-rerun
branch
from
July 8, 2026 11:25
4f3f40c to
44d41bc
Compare
skipi
force-pushed
the
mk/job-rerun/proto-job-lineage
branch
from
July 9, 2026 08:48
17e7fa9 to
48d6913
Compare
skipi
force-pushed
the
mk/job-rerun/job-level-partial-rerun
branch
from
July 9, 2026 08:49
e974bfd to
4c7c5ca
Compare
skipi
force-pushed
the
mk/job-rerun/proto-job-lineage
branch
from
July 13, 2026 06:24
48d6913 to
c86ac5b
Compare
skipi
force-pushed
the
mk/job-rerun/job-level-partial-rerun
branch
from
July 13, 2026 06:24
177f06e to
b2da1bf
Compare
skipi
force-pushed
the
mk/job-rerun/proto-job-lineage
branch
from
July 14, 2026 07:59
c86ac5b to
fffc5c1
Compare
skipi
force-pushed
the
mk/job-rerun/job-level-partial-rerun
branch
from
July 14, 2026 07:59
b2da1bf to
f7360e6
Compare
skipi
force-pushed
the
mk/job-rerun/proto-job-lineage
branch
from
July 21, 2026 07:51
fffc5c1 to
87d96eb
Compare
…rerun The shared jobs/builds schema is owned by this app's migrations. Add the nullable jobs.original_job_id lineage column with a partial index, and a partial unique index on builds.build_request_id backing task schedule idempotency. Mirrors the zebra test-env migrations at identical names so the schemas stay aligned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… builds - nullable indexed original_job_id uuid on jobs (lineage substrate for lightweight copies) - partial concurrent index on jobs.original_job_id (copies are the minority) - partial unique index on builds.build_request_id (idempotency backstop for concurrent schedule races)
- lineage (original_job_id), born finished+passed, preserved timestamps - cloned expires_at incl. accepted near-expiry sweep edge - audited clone whitelist incl. request, NULLed runtime fields - one-hop flattening, backward-compat nil, tenant-scoping
- schema field original_job_id (:binary_id) + @optional_fields entry - create_copy/2 mints a born finished+passed copy row without routing through create/1: clones identity/spec/display/payload columns (incl. request) and the original's real timestamps + expires_at, forces build_id/state/result, nulls runtime-execution-only fields, and flattens lineage one hop (original_job_id = original.original_job_id || original.id)
…e branch Cover the run-vs-copy branch in create_task: exact membership validation against original_task_id, typed invalid_argument outcomes, degrade-to-run paths for missing/non-copyable sources, onprem-metrics skip for copies, request_token idempotency under the concurrent race, all-copy immediate finish, no lifecycle events for copies, and one-hop lineage flattening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Teach create_task to honor the original_job_id marker with exact membership validation against the request's original_task_id: resolve the anchor once (must exist and share the workflow), load its job set once, and validate every marker as a member. Members that are finished+passed with a finished_at become lightweight copies via Job.create_copy inside the same transaction; cross-membership forgery and tenant mismatch fail loud as typed invalid_argument; missing or non-copyable sources degrade to running the job. Also: skip onprem_metrics for copies, add unique_constraint on build_request_id so a concurrent-race duplicate surfaces as a changeset error and re-reads the winning task, and finish all-copy tasks immediately via TaskFinisher.lock_and_process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…izer Cover serialize_job exposing original_job_id for a copy row, the top-level serialize/1 loading the column, and non-copy jobs remaining backward compatible (marker absent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add original_job_id to the job column selection and the serialize_job keyword list. Copies expose the terminal original id; non-copy jobs serialize unchanged because the nil marker is stripped before building the proto message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add roll-up cases proving the finisher handles lightweight copies with zero changes to task_finisher.ex: mixed copy+real passing, passed copy with a failed real job, and an all-copy task that finishes via lock_and_process without a finished_at crash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A marker or original_task_id that is present but not a valid UUID previously reached Repo.get on a binary_id column and raised Ecto.Query.CastError, surfacing as gRPC INTERNAL. Validate both with Ecto.UUID.cast before any lookup so malformed input returns the same typed invalid_argument as the other membership-validation failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…exists A failed CREATE UNIQUE INDEX CONCURRENTLY (e.g. pre-existing duplicate build_request_id values) leaves an INVALID index relation behind. With if_not_exists the migration re-run would see that relation name, skip creation entirely, and report success while the index enforces nothing - silently disabling the schedule idempotency that unique_constraint on build_request_id relies on. Drop if_not_exists so the re-run fails loudly, prompting a DROP INDEX of the invalid relation and a clean rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A task made only of lightweight copies keeps the original jobs' finished_at values, which predate the task itself. That skewed two pre-existing consumers: the task.finishing.duration metric got a multi-day outlier sample, and the TaskFinished event carried a timestamp from before the task existed. Clamp last_job_finished_at to the task's own created_at in both places so a task never reports finishing before it was created. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k scheduling - partition of a rebuilt block's jobs by original describe results - fail-closed predicate, index matching, degrade paths, metrics - job_level_partial_rerun feature flag helper
…st-forming time - flag-gated (job_level_partial_rerun, keyed on org id, fail-closed) - copy iff original job finished and passed; everything else re-runs - partition persisted into block request_args as job_copy_partition with the original block id and per-index original job ids - any failure (provider, describe, count/index mismatch) degrades to whole-block reschedule with a metric, never an STM error
…edule request - original_task_id resolved from the block's own task mapping - per-job original_job_id stamped by string-keyed index - degrade to whole-block request when the anchor is unresolvable
…k schedule request - resolve the original block id to its task id via the block service's own task mapping and set ScheduleRequest.original_task_id - stamp per-job original_job_id from the persisted partition, matched by string-keyed job index - drop all markers and the anchor when the original task is unresolvable (metric emitted), keeping whole-block rerun semantics - no partition present produces today's request unchanged
…sistence end to end - request_args with the partition round-trips through the DB, is rebuilt into ppl_args the same way task scheduling does, and the built request carries the resolved original_task_id plus per-index original_job_id markers - inverse case without a partition yields no original ids
…utation Only the feature check was exception-guarded; a raise anywhere downstream (corrupt persisted description, metrics call, DB exception) propagated out of scheduling_handler and put the block into the STM error path. Wrap the whole partition computation in the same rescue idiom so any unexpected error degrades to rescheduling the whole block with a mismatch metric and a warning, keeping the invariant that a partition failure never blocks scheduling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a copy partition A Task API invalid_argument rejection is deterministic - the persisted job copy markers cannot change between retries, so a rejected partition-carrying request previously looped in the STM retry path until the recovery limit. On a status-3 rejection of a request that carried a job copy partition, strip the partition and reschedule the whole block once, emitting a dedicated metric; a rejection of a partition-free request still falls through to the regular error path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…artial rerun The job copy partition now also records prior-attempt job ids for the jobs that will re-execute (rerun_jobs, keyed by index like the copy markers). The schedule request formatter appends a SEMAPHORE_ORIGINAL_JOB_ID env var to those jobs so CI scripts can fetch the previous attempt's artifacts and logs or compare test history. Copies are untouched - they never execute. Partitions persisted in the older format (no rerun_jobs key) inject nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New optional 'partial_rerun: jobs | block' property, settable pipeline-wide and per block (block overrides pipeline). 'block' forces a whole-block rebuild even when job-level partial rerun is available - the escape hatch for blocks whose jobs must run together, like dynamically split test suites. 'jobs' and an absent key defer to the feature flag; once the flag is removed the YAML property becomes the sole control with 'jobs' as the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…APHORE_JOB_RERUN Align with the platform naming schema (every job attribute lives under SEMAPHORE_JOB_*) and with the existing SEMAPHORE_PIPELINE_RERUN and SEMAPHORE_WORKFLOW_RERUN booleans: re-run jobs of a partial rerun now get SEMAPHORE_JOB_RERUN=true plus SEMAPHORE_JOB_ORIGINAL_ID with the previous attempt's job id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n lands separately Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The builds table has carried a full unique index on build_request_id (index_build_request_ids_on_builds) since 2018, so a duplicate insert raises on that name and a unique_constraint scoped to a differently named new index never matches - the changeset error path was dead in production and the transaction crashed instead of rolling back to the idempotent re-read. Point the changeset at the existing index, drop the redundant new index migration entirely (one less unique index to maintain on a large table), and mirror the production index name in the test-env migration so tests raise on the same constraint. Also harden the surrounding paths: only a unique-constraint violation on build_request_id routes to the already-scheduled re-read (any other changeset error is a typed invalid_argument), a non-UUID request_token is rejected up front instead of raising in the token pre-read, and a vanished winner during the re-read returns a retryable aborted instead of a match error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cate relies on Drive the real chain a block description travels in production - Task API proto response decoded the way TaskApiClient decodes it, persisted into the tasks description jsonb column, read back and reshaped by Block.describe - and assert the exact job shape (uppercase status and result strings, job_id, index) that the job-copy partition predicate consumes. Serialization drift now fails this test instead of silently degrading job-level rerun to whole-block rebuilds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… mode Absent partial_rerun means job-level once the feature is enabled; partial_rerun: block is the explicit way back to whole-block re-runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
skipi
force-pushed
the
mk/job-rerun/job-level-partial-rerun
branch
from
July 21, 2026 07:52
a471dc0 to
daf5bb8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deepens partial rerun from block granularity to job granularity: when a failed block is re-run, its previously passed jobs are carried over as lightweight copies (no re-execution) and only failed jobs actually run again.
Stacked on #1094 (vendored Task proto lineage fields) — the diff here excludes protos. Merge #1094 first, then retarget this to
main.What's inside
Migrations (github_hooks) — the shared jobs/builds schema is owned by this app's migrations, so DDL ships here and must be applied before the zebra/plumber code rolls out:
jobs.original_job_id(uuid) lineage columnindex_jobs_on_original_job_id_not_nullunique_builds_on_build_request_id_not_nullbacking schedule idempotency (deliberately noif_not_exists: a failed CONCURRENTLY build leaves an INVALID index that a silent rerun would keep)zebra — copy minting + validation:
Job.create_copy/2mints a born-finished+passed copy row: clones identity/spec/display/payload columns and original timestamps, nulls runtime-execution fields, flattens lineage one hop (original_job_idalways points at the terminal original)Schedulehonors per-joboriginal_job_idmarkers with exact membership validation againstoriginal_task_id(tenant + workflow checked); malformed UUIDs and cross-membership forgery return typedinvalid_argument; missing or non-copyable sources degrade to running the jobTaskFinisherclamps all-copy finish timestamps so a task never reports finishing before it was createdDescribeexposesoriginal_job_id; non-copy jobs serialize unchangedplumber — flag-gated partition at request-forming time:
request_argsinto the task schedule requestSEMAPHORE_JOB_RERUN=trueandSEMAPHORE_JOB_ORIGINAL_ID(the previous attempt's job id) so CI scripts can fetch prior artifacts or compare test historypartial_rerun: jobs | blockpipeline YAML property (pipeline-wide with per-block override); enabling the feature flag makesjobsthe default rebuild mode, andpartial_rerun: blockis the explicit way back to whole-block rebuilds — the escape hatch for dynamically split test suitesjob_level_partial_rerunfeature flag (org-scoped, fail-closed)Known limitations — read paths do not follow lineage yet
Copy rows carry
original_job_id, but the consumers that build log/artifact/test-result URLs still key on the copy's own job id:job.id, so a copied job's page shows finished/passed with empty logs/artifacts — the data lives under the original job idtest-results gen-pipeline-reporton a rebuild pipeline finds no per-job reports under the new pipeline id (copies don't re-publish), so the rebuild has no pipeline-level test report and its after_pipeline job fails while the pipeline reports passedserver_farm.jobinternal API (which the front job model consumes) does not exposeoriginal_job_idyet — only the internal Task API Describe doesA follow-up PR adds the indirection (
source_job_id = original_job_id || idon read-only execution artifacts, proto + serializer + front): until it merges, thejob_level_partial_rerunfeature flag must stay off in production, which also keeps the copy rows invisible to users. The user-facing docs in this PR describe the shipped behavior (data remains under the original job) rather than the end-state UI.Compatibility
Deploy order
🤖 Generated with Claude Code