Skip to content

[server] Add recoverable bulk-load transactions - #4771

Open
platinumhamburg wants to merge 2 commits into
apache:mainfrom
platinumhamburg:feature/bulkload-server-transactions
Open

platinumhamburg wants to merge 2 commits into
apache:mainfrom
platinumhamburg:feature/bulkload-server-transactions

Conversation

@platinumhamburg

@platinumhamburg platinumhamburg commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Part of #4222. Implements the server-side transaction protocol tracked by #4224, following #4242.

Bulk loading an empty primary-key table or physical partition must keep the target inaccessible until every imported bucket is installed and its assigned replicas are ready. The Coordinator now owns a recoverable Begin / GetInProgress / Commit / Abort lifecycle through internal RPCs.

Brief change log

  • Persist target ownership and the commit decision; fence ordinary access and stale snapshot or remote-log commits while loading.
  • Validate complete bucket coverage and adopt ordinary KV snapshots before atomically activating the target.
  • Reconcile Coordinator failover, TabletServer restart, leadership and assignment changes; retain terminal results and expire transaction metadata.
  • Track leader activation by notification request and advertise server protocol capabilities.
  • Expose active transaction references for the subsequent orphan-cleanup integration. The public SDK, scoped build authorization and Flink integration follow separately.

Tests

  • Java 11: 82 focused tests passed across RPC API/error compatibility, ZooKeeper metadata, Coordinator event processing and BulkLoad transaction recovery.
  • BulkLoadServerITCase: all 12 E2E cases passed, covering import and continued online writes, abort, target fencing, Coordinator/leader loss, TabletServer restart, partition bucket layouts and terminal-result retries.
  • Maven reactor test runs passed Spotless and Checkstyle checks.

API and Format

Adds internal Coordinator RPCs and version 1 of NotifyLeaderAndIsr. Adds optional bulk-load ownership/state and protocol-capability fields to registration metadata, plus persistent bulk-load transaction records. Begin rejects servers that do not advertise the required capability.

Documentation

The public SDK and engine-facing usage documentation are tracked by the remaining tasks under #4222.

@platinumhamburg
platinumhamburg marked this pull request as ready for review October 21, 2026 03:41
@platinumhamburg
platinumhamburg force-pushed the feature/bulkload-server-transactions branch 3 times, most recently from f4ef902 to 1bd7b65 Compare October 23, 2026 09:36
Coordinate bulk-load transactions with durable commit and abort decisions, fenced replica activation, and snapshot recovery. Preserve normal table lifecycle behavior across deletion, restart, and failover.

Cover transaction recovery, replica fencing, snapshot import, continued online writes, and deletion interleavings with integration and regression tests.
@platinumhamburg
platinumhamburg force-pushed the feature/bulkload-server-transactions branch from 1bd7b65 to 6a86372 Compare October 26, 2026 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant