Skip to content

[FLUSS-1027] Stabilize KvSnapshotITCase snapshot triggering#3731

Draft
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/kv-snapshot-itcase
Draft

[FLUSS-1027] Stabilize KvSnapshotITCase snapshot triggering#3731
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/kv-snapshot-itcase

Conversation

@wolfkill

Copy link
Copy Markdown

What changed

This makes KvSnapshotITCase.testKvSnapshotAndDelete trigger KV snapshots deterministically instead of waiting for the periodic snapshot scheduler to eventually create snapshot IDs 0 and 1.

The test now:

  • disables the short auto-snapshot interval for this test cluster
  • waits until the leader high watermark reaches the log offset that must be included in the next snapshot
  • explicitly triggers and waits for each snapshot
  • asserts that the returned snapshot IDs are 0 and 1 before validating snapshot contents

Root cause

putKv(...).get() only waits for the write request to finish. It does not guarantee that the KV pre-write buffer has already been flushed into the state visible to the snapshot path.

KvTabletSnapshotTarget.initSnapshot() skips snapshot creation when the flushed KV log offset has not advanced beyond the previous snapshot offset. The previous test relied on a 1 second periodic snapshot interval and direct ZooKeeper polling, so the scheduler could fire before the KV data became snapshot-ready, leaving snapshot 0 absent and causing the flaky timeout reported in #1027.

Validation

  • git diff --check
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-server -am -Dtest=KvSnapshotITCase#testKvSnapshotAndDelete -DfailIfNoTests=false -DskipITs=false test
  • repeated the same targeted Maven command a second time

Both targeted runs passed with:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Closes #1027.

@wolfkill

Copy link
Copy Markdown
Author

Additional local validation after opening the draft PR:

  • Ran the full KvSnapshotITCase with local service port access enabled:
    • Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
    • reactor result: BUILD SUCCESS
  • Re-ran the original flaky method KvSnapshotITCase#testKvSnapshotAndDelete three more times after that:
    • all three runs completed with Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
    • all three Maven reactor runs completed with BUILD SUCCESS
  • Confirmed git diff --check has no whitespace errors.
  • Confirmed the local worktree has no uncommitted source changes.

One sandboxed run of the full class failed before reaching test assertions because Curator could not bind a local ZooKeeper test port (java.net.SocketException: Operation not permitted). The same command passed after allowing local test service port binding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test KvSnapshotITCase.testKvSnapshotAndDelete is unstable

1 participant