fix: hybrid passthrough defer tail + UploadPartCopy keepalive (2026.7.17)#124
Merged
Merged
Conversation
Add request-scoped context (path, uploadId, partNumber) to MEMORY_*, REQUEST_* and S3_ERROR_RESPONSE logs so prod 4xx/5xx are diagnosable without guessing. Harden the postgres-style UploadPart path: map InvalidPart/EntityTooSmall ClientErrors to 400, reject internal part ranges past S3's 10k ceiling, detect truncated upload bodies, retry Redis WATCH conflicts with backoff (25 attempts), and raise S3 read_timeout to 300s for slow seaweed uploads. Co-authored-by: Cursor <cursoragent@cursor.com>
…ents A 4.7GB Scylla part copy sent zero response bytes until the copy finished, so rclone's 5-minute idle timeout killed every attempt (~400s of work vs a 300s deadline) and the proxy kept the doomed copy running as a zombie while scylla-manager retried forever. - Commit to 200 OK and trickle whitespace while the copy runs (AWS S3 long- copy pattern); emit CopyPartResult or an <Error> document as the body - Cancel in-flight backend work when the client disconnects (no zombies) - Run passthrough segment copies concurrently (bounded, default 8) instead of ~570 sequential backend calls - Overlap the synthetic-ETag MD5 source pass with the segment copies instead of serially re-reading the whole source after them - Fix ruff check failures (F841, B039, import order) so CI lint passes
…ion failures) test_copy_full_encrypted_object: the single-segment passthrough copy returned a synthetic plaintext etag to the client but stored it as the part etag too, so CompleteMultipartUpload forwarded an etag the backend never issued -> InvalidPart. This was masked before ae39244: the proxy turned the backend error into a 500, boto3 retried, and the retry succeeded via state reconstruction (which reads real etags from list_parts). Store the backend CopyPartResult etag in part state and have _build_s3_parts use stored etags instead of the client echo. test_part_number_out_of_range: rejecting UploadPart before reading its body (new internal-part-range validation) left the body bytes on the keep-alive connection, so the client's next request (the abort) hit a raw uvicorn 400. Drain small bodies in the error handler; send Connection: close for bodies too large to slurp. Both reproduced locally against real MinIO and verified fixed.
…manifests When part 1 hybrid passthrough ends with a sub-5MB mid-frame tail and part 2 follows, store the tail in upload state instead of uploading it as a non-final internal S3 part. Part 2 prepends the deferred bytes into streaming; complete flushes any leftover tail. Adds debug logs and regression tests. Co-authored-by: Cursor <cursoragent@cursor.com>
ServerSideHannes
force-pushed
the
fix/hybrid-passthrough-defer-tail
branch
from
94d291b to
c60697e
Compare
Use 8MB frames and 36MB part 1 so the copy hits passthrough (not simple copy below STREAMING_THRESHOLD) and internal segments meet MIN_PART_SIZE. Co-authored-by: Cursor <cursoragent@cursor.com>
Mark 500+ segment Scylla prod-shape unit tests as slow, exclude them from parallel test-unit, and run them with -n0 in a separate CI step. Co-authored-by: Cursor <cursoragent@cursor.com>
Streaming copy now calls take_deferred_copy_tail at part start; test _Mgr/_TrackingMgr stubs were missing the method. Co-authored-by: Cursor <cursoragent@cursor.com>
Move prod-scale unit tests out of the parallel unit job into unit-slow with a longer timeout. Shrink part 2 in the two-part prod-shape test so it still exercises streaming + deferred tail without ~1.3GB of work. Co-authored-by: Cursor <cursoragent@cursor.com>
572-segment part 1 + part 2 streaming on a ~5GB mock source exhausts runner memory. Coverage is split: prod part-1 passthrough (slow) and two-part complete with deferred tail (fast). Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EntityTooSmallonCompleteMultipartUpload). Part 2 prepends deferred bytes into streaming; complete flushes any leftover tail.400 0on PUT) seen onoceanio-dc2-scylladb-backups-v3.HYBRID_TAIL_DEFER_DECISION,UPLOAD_PART_COPY_PASSTHROUGH_ALLOC, deferred-tail fields on passthrough/streaming complete,COMPLETE_MULTIPART_DEFERRED_TAIL_PENDING.Test plan
test_upload_part_copy_passthrough.py(fast two-part defer test + full Scylla prod-shape test)test_upload_part_copy_keepalive.pyCOMPLETE_MULTIPART_SUCCESSon.sm_manifests, no fast POSTEntityTooSmall/InvalidPart400s, no new 30m PUT timeoutsFailedtracking deduped bytes that then fail at completeMade with Cursor