Propagate deadlines in the background, without waiting for locks

[SupraSummus]

Problem

Adding a precondition moved deadlines of it and all its preconditions, recursively, in the same transaction. That locked goals along existing dependencies from dependent to precondition, while a worker finishing a goal locks it and updates its dependents' counters, the other way. Reproduced deadlocks:

• a worker returns RetryMeLater(precondition_goals=[B]), B depends on C, C is being achieved. Postgres aborts one of them: either the handler's work is rolled back, or C's finished handler is rolled back and runs again.
• schedule(precondition_goals=[B]) from outside a worker (e.g. a web request), the same way.

Even without a deadlock, schedule() waited for any goal being pursued down the dependency graph, for as long as its handler ran.

Change

• _add_precondition_goals only marks its new dependencies whose precondition has a later deadline (GoalDependency.deadline_propagated=False).
• New handle_deadline_propagation(), run by busy and threaded workers, moves deadlines over marked dependencies, one level per call. It uses row-level locks with SKIP LOCKED only, so it never waits for a goal being pursued: a dependency on a locked goal stays marked for a later call.
• A precondition whose deadline moved gets Goal.deadline_propagated=False, and the next call marks its late dependencies. Marking them in the same transaction could deadlock with another call holding them; the goal row is already locked, so the flag costs no waiting. An edge-only variant deadlocked 59 times in a 40 s stress run, this one 0.
• Each piece of work is recorded once and found by a partial index. Checked with EXPLAIN on 200k goals: adding a precondition to a goal with 50k preconditions touches one row.
• Migration 0012 adds both flags and their partial indexes; existing rows start as propagated.
• update_goals_deadline is removed.
• TODO: add the deadlock in the counter lock of _add_precondition_goals (still there, separate change) and the duplicated handler lists; drop the pypi.yml chore fixed on main.

Behavior change: deadlines reach preconditions in the background, one level per worker turn, not inside schedule().

Testing

• pytest (94), mypy, flake8 on Django 6.1 and 5.2; makemigrations --check.
• A new test holds a goal locked like a worker does, with lock_timeout set: on main it fails with LockNotAvailable, here it passes.
• Each of 7 deliberate breakages of the propagation (waiting for locks, dropping skipped dependencies, busy loop on a locked goal, ...) fails the tests.
• Stress run, not committed: 40 s on a random DAG with concurrent propagation, simulated workers, schedule() calls and deadline drops. main: 29 deadlocks; this branch: 0, and after settling no unachieved precondition has a later deadline than its dependent.

🤖 Generated with Claude Code

https://claude.ai/code/session_017FZQDBWAKp5JbsKYHbMfox

---

Generated by Claude Code