Skip to content

hud/server: make apiserver config registration atomic - #6815

Open
prajwalshah19 wants to merge 1 commit into
tilt-dev:masterfrom
prajwalshah19:fix/apiserver-config-lost-update
Open

hud/server: make apiserver config registration atomic#6815
prajwalshah19 wants to merge 1 commit into
tilt-dev:masterfrom
prajwalshah19:fix/apiserver-config-lost-update

Conversation

@prajwalshah19

Copy link
Copy Markdown

Fixes #6814.

addToAPIServerConfig and removeFromAPIServerConfig read the shared config under a read lock, released it, mutated an in-memory copy, and later wrote the full snapshot back under a separate write lock. The write lock serializes writers but doesn't protect the whole transaction, so two Tilt processes can read the same base snapshot and the later writer discards the earlier one's registration or removal.

The result is a CLI command failing with No tilt apiserver found: tilt-<port> against a server that is still running and healthy — the entry was simply dropped from the config by an unrelated Tilt process.

This holds the exclusive lock across the entire read-modify-write in both functions, so concurrent processes serialize on the full transaction. modifyConfig was only used by these two callers and is inlined into them; ValidateAPIServerName now runs before the read, so an invalid name no longer takes the lock or touches the file.

No deadlock risk from the nested clientcmd.ModifyConfig call: internal/filelock's init() sets clientcmd.UseModifyConfigLock = false, so ModifyConfig doesn't take a lock of its own.

Testing

TestConcurrentAPIServerConfigUpdates registers and then deregisters 8 controllers concurrently against one config file. It fails on master — an entry survives after every removal, the lost-update — and passes with the fix. Clean under -race -count=10.

Also run: make shorttest (full suite; the only failures are the pre-existing TestKustomizeFlags/TestKustomizeBin, which fail identically on unmodified master because kustomize isn't on my PATH), go vet, goimports -local github.com/tilt-dev, and golangci-lint (0 issues).

Notes for reviewers

  • Mixed versions still race. The lock is advisory and only helps processes running this code, so an older tilt binary alongside a fixed one can still clobber the config.
  • Slightly longer exclusive hold. The read moves from a shared lock to the exclusive one, so two concurrent startups serialize where they previously read in parallel. The file is small, so this should be immaterial — flagging it as a real behavior change.
  • I verified on macOS only; Windows semantics look equivalent (LockFileEx exclusive per handle) but I haven't run it there.

addToAPIServerConfig and removeFromAPIServerConfig read the shared
config file under a read lock, released it, mutated an in-memory copy,
and wrote the full snapshot back under a separate write lock. Two Tilt
processes could read the same base snapshot and the later writer would
discard the earlier writer's registration or removal, leaving clients
failing with "No tilt apiserver found: <name>" while the server was
still running.

Hold the exclusive file lock across the entire read-modify-write so
concurrent processes serialize on the whole transaction.

Signed-off-by: prajwalshah19 <70662078+prajwalshah19@users.noreply.github.com>
@prajwalshah19
prajwalshah19 force-pushed the fix/apiserver-config-lost-update branch from dc4fa68 to e3db3e3 Compare August 2, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent tilt instances can lose each other's apiserver config entries ("No tilt apiserver found" while running)

1 participant