Do not report transient partial sums from waitMetrics

[saintstack]

DDPipelineSaturation failure in nightlies on 7.4:

waitMetrics accumulates byte-sample deltas and replies as soon as the running sum leaves [min, max]. Rewriting a sampled key reaches the waiter as two deltas — a clear (-bytes, from byteSampleApplyClear) followed by a set (+bytes) — and the waiter can reply between them with a value that never existed at any version, e.g. 0 bytes for a shard that holds data. The code already notes the possibility ("The changes here are possibly partial changes…").

Data distribution acts on that value. In DDPipelineSaturation, whose knobs cap shards at 40 KB (below the size of the system keyspace), it produced a livelock on [\xff, \xff\xff):

1. The tracker splits [\xff, \xff\xff) (71,500 bytes > 40,000) into two ~35,750-byte halves.
2. The split's own data move rewrites a serverKeys entry in the right half; a bounded waitMetrics on that half replies 0.
3. DD sees a half below the merge threshold and merges; the merged estimate (0 + 35,750) is under the cap.
4. The merged shard is re-measured at 71,500 and split again.

This repeated every ~0.15 s for ~2,200 s of simulated time — 12,823 merges, all with EndingSize=35750 — until the trace-line limit aborted the process (TracedTooManyLines). Instrumenting the tracker showed the 0 is transient: the next read of the same location returns 35,750.

Fix

Before replying out of bounds, yield (delay(0)) so the rest of the update is applied, re-read getMetrics(req.keys), discard the deltas queued meanwhile, and reply only if the fresh value is still out of bounds. Otherwise keep waiting.

A wrong_shard_server queued by notifyNotReadable() during the yield is caught and goes through the existing error path (the request fails with wrong_shard_server). Without that, the drain rethrows it and takes the storage server down — an earlier version of this change did exactly that.

New code probe: ShardWaitMetrics transient out-of-bounds sum discarded.

Testing

• New unit tests in fdbserver_core_test:
  • /fdbserver/StorageMetrics/waitMetrics/ignoresTransientPartialUpdate — back-to-back clear and re-set of a sampled key must not trigger a reply; a real clear afterwards still replies 0.
  • /fdbserver/StorageMetrics/waitMetrics/notReadableWhileRevalidating — range becomes unreadable during revalidation: request fails with wrong_shard_server, the waiter does not throw.
  • Both fail with the fix reverted, at the expected asserts.
• Joshua, 100,000 runs of DDPipelineSaturation only: 100,000 passed.
• Joshua, 100,000 runs of the full correctness suite: 100,000 passed.

Known limitation

update() yields every DESIRED_UPDATE_BYTES of mutations. A clear and its re-set split across that yield could still produce a transient. Expected to be rare; not addressed here.

Backports

Should be backported to release-8.0 and release-7.4. Backport PRs will go up only after this lands on main.

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: cd86d44
• Duration 0:21:03
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: cd86d44
• Duration 0:38:13
• Result: ❌ FAILED
• Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: cd86d44
• Duration 0:42:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: cd86d44
• Duration 0:45:23
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: cd86d44
• Duration 0:47:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[saintstack]

Let me close this for now. The failures are in 7.4 Will reopen when branch 7.4 reopens for commits (there are few failure types for this test.... Let me land a fix at a time...)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: cd86d44
• Duration 1:37:21
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: cd86d44
• Duration 2:00:04
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)