fix: deferred copy tail double-counted in completed object plaintext size

[ServerSideHannes]

Problem

Overnight (10 Jul, 00:22–01:53 UTC) the Scylla backup run sm_20260709080013UTC failed 398 versioning moves (~1.14 TiB) on the new node (rack-12) with rclone reporting:

corrupted on transfer: sizes differ 6846984417 vs 6848114081

The destination was exactly +1,129,664 bytes on every file regardless of tail size — the size of the deferred hybrid tail for a part pinned at rclone's copy-cutoff (4,999,341,932 bytes) against prod's 50MB/6 internal frame layout.

Root cause

For an object larger than rclone's copy_cutoff (4.656 GiB), rclone issues a 2-part UploadPartCopy. Since #124:

1. Part 1 (hybrid passthrough) defers its sub-5MB tail — but recorded plaintext_size = segments + tail for bytes it did not store.
2. Part 2 (streaming, non-zero-start range) consumes the tail via take_deferred_copy_tail and prepends it as leading_plaintext — correctly counting it, since the tail bytes physically live in part 2.
3. Complete sums client-part plaintext sizes → sidecar total_plaintext_size = object + tail → HEAD Content-Length inflated → rclone's post-copy size check fails and it refuses to finish the move. Data on the wire was correct and in order; only the accounting was wrong.

Smaller objects (single-range copies, e.g. all people SSTables) never split, which is why only the ~6.8 GB companies objects failed.

Fix

• _passthrough_copy_part: part 1's metadata reflects stored bytes only (segments; no deferred tail).
• _flush_deferred_copy_tail_for_complete: when the tail is flushed at complete (no later part consumed it), bump the receiving part's plaintext_size alongside ciphertext_size, keeping per-part accounting physical in that path too.
• _build_s3_parts now logs PART_PLAINTEXT_MISMATCH whenever a client part's plaintext disagrees with the sum of its internal parts — this invariant violation is exactly what would have surfaced the bug immediately in prod.

Tests

• test_two_part_defer_tail_total_plaintext_not_double_counted — reproduces the prod 2-part copy shape; fails on main (part 1 plaintext ≠ stored internals; sidecar total = source + 1,129,664-analog), passes with the fix. Asserts sidecar total_plaintext_size == source size and full GET roundtrip equality (content + order).
• test_defer_tail_flushed_on_complete_keeps_part_accounting — guards the complete-time flush path (part 1 only): per-part plaintext == internal parts, total == range span, GET roundtrip.

Impact on the running backup

The ~1.14 TiB marked Failed on rack-12 is retryable — rclone never deletes the source on a failed move, and the same class of copy is pending retry on the other hosts (e.g. rack-2's 364 GiB of companies). Once this deploys, the retry passes should complete the run.

🤖 Generated with Claude Code