Skip to content

fix: writer preference so budget-monopolizing copies are not starved#112

Merged
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-clamp-deadlock-and-py2-except
Jul 7, 2026
Merged

fix: writer preference so budget-monopolizing copies are not starved#112
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-clamp-deadlock-and-py2-except

Conversation

@ServerSideHannes

@ServerSideHannes ServerSideHannes commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Problem

Prod logs showed a server-side COPY stuck on endless MEMORY_BACKPRESSURE with requested_mb == limit_mb:

{"active_mb": 8.38, "requested_mb": 192.0, "limit_mb": 192.0, "remaining_sec": 6.7, "event": "MEMORY_BACKPRESSURE"}

A multi-GB manifest COPY deliberately clamps its reservation to the whole budget so it runs exclusively — this is the guard against the 3×4.7GB concurrent-copy OOM that #110 fixed (tests/unit/test_copy_clamp_and_concurrency.py). That part is correct and unchanged.

The bug is in the limiter's admission logic. The gate is active_bytes + to_reserve <= limit, so a whole-budget reservation can only be admitted when active_bytes == 0. The limiter had no fairness: every release() does notify_all(), and a small request re-grabs memory before the waiting copy can (it fits), so active_bytes never drains to 0 under steady backup traffic. The copy backpressures the full BACKPRESSURE_TIMEOUT, 503s, retries, starves again — functionally never admitted while the pod has any other traffic.

Fix

Writer preference, not a smaller clamp (lowering the clamp would re-introduce the multi-copy OOM). Track pending whole-budget ("exclusive") waiters; while one waits, hold back new non-exclusive admissions so active_bytes drains to 0 and the copy gets its exclusive slot — bounded by the lifetime of the already-running requests. Copies still reserve the whole budget and run one-at-a-time.

  • _can_admit(to_reserve, exclusive): an exclusive request ignores the pending gate (competes only on active_bytes == 0); a non-exclusive request also waits while _pending_exclusive > 0.
  • _pending_exclusive incremented/decremented around the wait, with a notify_all() on exit so held-back requests resume whether the copy acquired or gave up.

Tests

All existing copy-concurrency / OOM regression tests are unchanged and pass (copies still exclusive, still one-at-a-time).

Added test_pending_exclusive_copy_not_starved_by_small_requests: a pending exclusive copy must not be starved by a stream of small requests — they queue behind it, and the copy wins the slot the moment the in-flight blocker drains. Verified it fails against the old no-fairness limiter with the exact prod signature (requested_mb == limit_mb backpressure while a small request slips ahead), and passes with writer preference.

@ServerSideHannes ServerSideHannes marked this pull request as draft July 7, 2026 05:53
@ServerSideHannes ServerSideHannes force-pushed the fix/copy-clamp-deadlock-and-py2-except branch from 8ef0aba to 9e9a706 Compare July 7, 2026 06:05
@ServerSideHannes ServerSideHannes changed the title fix: copy governor deadlock (reserve == budget) + Py2 except syntax fix: writer preference so budget-monopolizing copies are not starved Jul 7, 2026
@ServerSideHannes ServerSideHannes marked this pull request as ready for review July 7, 2026 06:05
A server-side COPY clamps its reservation to the whole budget so it runs
exclusively (guards the multi-copy OOM #110 fixed). The admission gate is
active_bytes + to_reserve <= limit, so a whole-budget request can only be
admitted when active_bytes == 0. The limiter had no fairness: every release
woke all waiters and a small request re-grabbed memory before the copy could,
so active_bytes never drained to 0 under steady traffic. The copy backpressured
the whole timeout (prod: MEMORY_BACKPRESSURE requested_mb == limit_mb) and 503'd,
retried, starved again -- effectively never admitted while the pod had traffic.

Add writer preference: track pending whole-budget (exclusive) waiters and hold
back new non-exclusive admissions while one waits, so active_bytes drains and
the copy gets its exclusive slot (bounded by in-flight request lifetimes).
Copies still reserve the whole budget and run one-at-a-time -- the existing
copy-concurrency/OOM regression tests are unchanged and still pass.

Regression test: a pending exclusive copy must not be starved by a stream of
small requests; the small ones queue behind it. Verified it fails against the
old no-fairness limiter with the exact prod signature (requested_mb==limit_mb).
@ServerSideHannes ServerSideHannes force-pushed the fix/copy-clamp-deadlock-and-py2-except branch from 9e9a706 to 07115f6 Compare July 7, 2026 06:09
@ServerSideHannes ServerSideHannes merged commit 9f7e648 into main Jul 7, 2026
10 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/copy-clamp-deadlock-and-py2-except branch July 7, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant