[kalverra]

Intent

Postgres reports 40P01 deadlock detected on llo_mercury_transmit_queue when concurrent Delete (locking by transmission_hash) and Prune (locking by seq_nr) lock overlapping rows in opposite orders. Prune now selects candidates with FOR UPDATE SKIP LOCKED in both its batch loop and final truncate, leaving rows locked by Delete for the next prune cycle.

Changes

Prune skips rows locked by Delete instead of blocking on them:

flowchart LR
  D["Delete (by transmission_hash)"] -->|locks row, must block| Q[(llo_mercury_transmit_queue)]
  P["Prune (by seq_nr / OFFSET)"] --> B["Batch & tail candidate select"]
  B --> SL["FOR UPDATE SKIP LOCKED (new)"]:::new
  SL -->|skips locked rows| Q
  classDef new fill:#d4f8d4,stroke:#2da44e

Loading

Prune batch loop skips locked rows

• Batch delete subquery adds FOR UPDATE SKIP LOCKED; skipped rows are picked up by the next prune cycle.

Final truncate locks only excess rows past the OFFSET

• Window function replaced by an unlocked OFFSET subquery wrapped in FOR UPDATE SKIP LOCKED. FOR UPDATE in the same SELECT as OFFSET would lock all maxSize retained rows (up to 300k at the default cap) and shift the offset window when a retained row is locked, over-deleting a kept row.

Test helpers for multi-session lock contention

• NewIndependentSqlxSessions and NewIndependentSqlxDBs share a private helper that opens pinned connections to one fresh database.
• TestORM_PruneSkipsRowsLockedByDelete locks both a pruned and a retained row and asserts Prune skips them without shifting the OFFSET window; TestORM_ConcurrentDeleteAndPrune hammers both paths and fails on any error.

Notes

• Delete intentionally does not use SKIP LOCKED: its caller already removed those transmissions from the in-memory queue, so skipping would silently lose rows.
• The reaper in chainlink/core/services/llo/cleanup.go (reapStale, reapOrphaned) has the same access pattern and is fixed in a separate PR. Chainlink needs a go.mod bump to this commit.

[Copilot]

🟡 Changes recommended

The trimming query locks retained rows consumed by OFFSET, potentially locking 300,000 rows unnecessarily and delaying concurrent Delete calls; locking should apply only to excess candidates.

1 open issue

• Avoid locking retained queue rows during trimming · New

What changed in this PR

This PR addresses deadlocks between LLO transmit-queue pruning and deletion by making Prune skip locked rows without changing exported APIs.

Changes:

• Adds FOR UPDATE SKIP LOCKED to pruning queries and replaces the final window-function query with OFFSET.
• Adds lock-contention regression tests and shared-database test helpers.
• Records the fix in a patch changeset.

File	Description
mercury/​testutils/​sql.go	Adds shared-database handles and single-connection test sessions.
llo/​transmitter/​dataengine/​orm.go	Updates batch pruning and final trimming to skip locked rows.
llo/​transmitter/​dataengine/​orm_test.go	Adds locked-row and concurrent Delete/Prune regression tests.
.changeset/​llo-transmit-queue-deadlock.md	Documents the deadlock fix for release.

_{Review effort: Balanced}

[kalverra]

PR Triage · Risk 🟡 Medium · Complexity 🟡 Medium

Suggests 1 approval, including one from an owner. Suggested reviewers: @brunotm, @pavel-raykov.

	Level	Why
Risk	Medium	changes touch 2 subsystems (threshold 2)
Complexity	Medium	new code size 1,706 (Medium at 1,600)

Visualizations

Score map

flowchart LR
  R["Risk: Medium"]
  rs["1 signal → floor Medium"]
  s0["subsystems 2"] --> rs
  rs --> R
  C["Complexity: Medium"]
  m0["new code size 1,706 ≥ 1,600"] --> C
  RT["Suggests 1 approval · one from an owner"]
  R --> RT
  classDef low fill:#dafbe1,stroke:#1a7f37,color:#1f2328;
  classDef medium fill:#fff8c5,stroke:#9a6700,color:#1f2328;
  classDef high fill:#ffebe9,stroke:#cf222e,color:#1f2328;
  class R medium;
  class C medium;

Loading

Change shape

pie title Change shape
    "Tests" : 1081
    "Source" : 225
    "Trivial" : 5

Loading

— concentrated in llo/transmitter/dataengine/orm_test.go

Files: 4 changed, 1 skipped

File	Risk	Complexity	Why
llo/transmitter/dataengine/orm.go	Low	Low
llo/transmitter/dataengine/orm_test.go	Low	Low
mercury/testutils/sql.go	Low	Low
1 skipped (documentation)			.changeset/llo-transmit-queue-deadlock.md

Why these reviewers

Reviewer	Share	Last commit
@brunotm	87%	2026-08-07
@pavel-raykov	13%	2026-05-04

Ranked by commits to the changed files. Commits in the 360 days before the base tip count, newer ones more. Requested everyone with at least 20% of the score, up to 4. Added 1 reviewer below the share bar to reach the approval floor. Excluded the PR author, @kalverra. Considered only current organization members.

Metrics and policy

Metric	PR value	Scale	Medium at	High at
logic added (cognitive delta)	14	[██████│░░░░░░░░░░░░│]	15	45
syntax change (AST delta)	1,306	[████████████████░░░│]	1,600	—
new code size (Kolmogorov delta)	1,706	[██████████████████│█]	1,600	—
most complex function (max function cognitive complexity)	11	[███░░░░░░░░░░░░░░░░│]	70	70

Risk signals: change spread ○ · subsystems ● · hotspot ○ · blast radius ○ · author familiarity ○ · floor Medium

pr-triage v0.0.0-20261030164835-20c4213aefbb · policy fc234bb

---

_{What is this? · How is this calculated?}

[brunotm]

we need the partitioning predicate also on the middle query taking the lock. this will allow the qeury planner to estimate the scan instead of a sequential or hash join over the whole table,

Suggested change

	DELETE FROM llo_mercury_transmit_queue AS q
	USING (
	SELECT transmission_hash
	FROM llo_mercury_transmit_queue
	WHERE transmission_hash IN (
	SELECT transmission_hash
	FROM llo_mercury_transmit_queue
	WHERE don_id = $1 AND server_url = $2
	) sub
	WHERE row_num > $3
	)
	DELETE FROM llo_mercury_transmit_queue
	WHERE ctid IN (SELECT ctid FROM to_delete);
	WHERE don_id = $1 AND server_url = $2
	ORDER BY seq_nr DESC, inserted_at DESC
	OFFSET $3
	)
	FOR UPDATE SKIP LOCKED
	) AS to_delete
	WHERE q.transmission_hash = to_delete.transmission_hash;
	DELETE FROM llo_mercury_transmit_queue AS q
	USING (
	SELECT transmission_hash
	FROM llo_mercury_transmit_queue
	WHERE don_id = $1 AND server_url = $2
	AND transmission_hash IN (
	SELECT transmission_hash
	FROM llo_mercury_transmit_queue
	WHERE don_id = $1 AND server_url = $2
	ORDER BY seq_nr DESC, inserted_at DESC
	OFFSET $3
	)
	FOR UPDATE SKIP LOCKED
	) AS to_delete
	WHERE q.transmission_hash = to_delete.transmission_hash;