Skip to content

prevent Nexuses from writing old RackNetworkConfig configs - #11604

Open
nicolaskagami wants to merge 2 commits into
mainfrom
nsk/atomic-bootstore-gen-bump
Open

nicolaskagami wants to merge 2 commits into
mainfrom
nsk/atomic-bootstore-gen-bump

Conversation

@nicolaskagami

@nicolaskagami nicolaskagami commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes: #10855

Context

Currently, two nexuses can race to write to the bootstore and an older configuration of RackNetworkConfig can be written atop a new one. This can be solved with a minimal tweak of how we use the bootstore generation number. If we guarantee that:

  • Nexus observes the gen number before it observes the networking config input.
  • Only one nexus can write having observed a specific gen number.

Then a newer write must have observed a newer config.

What does this PR change?

  • Read the current bootstore gen number in the same transaction as RackNetworkConfigInput.
  • Makes bumping the bootstore gen number a "compare-and-swap" approach, so only one instance can successfully do it for a given gen number.
  • Only write to the bootstore if we successfully bumped the gen number.

Details

  • We can't know if the nexus that lost the race actually had a newer configuration. Currently it will try again next time, but we could prefer to retry immediately.
  • I've tested this on voxel making the race much more likely by adding a random 0-3s delay between reading and writing. With that setup, main wrote older configs 19/78 times while this branch never did.

@nicolaskagami nicolaskagami self-assigned this Oct 27, 2026
@nicolaskagami nicolaskagami added bug Something that isn't working. multi-rack labels Oct 27, 2026
@nicolaskagami nicolaskagami changed the title prevent Nexuses from writing old bootstore configs prevent Nexuses from writing old RackNetworkConfig configs Oct 27, 2026
@nicolaskagami
nicolaskagami marked this pull request as ready for review October 27, 2026 19:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something that isn't working. multi-rack

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nexuses can duel when updating the RackNetworkConfig in the bootstore

1 participant