Skip to content

KSQL-14980 | Address misc items for ksql - #11037

Open
Parag Badani (pbadani) wants to merge 39 commits into
8.0.xfrom
pbadani/fix-bugs
Open

KSQL-14980 | Address misc items for ksql#11037
Parag Badani (pbadani) wants to merge 39 commits into
8.0.xfrom
pbadani/fix-bugs

Conversation

@pbadani

Copy link
Copy Markdown
Member

Description

What behavior do you want to change, why, how does your patch achieve the changes?
Address misc items for ksql

Testing done

Describe the testing strategy. Unit and integration tests are expected for any behavior changes.

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")
  • Do these changes have compatibility implications for rollback? If so, ensure that the ksql command version is bumped.

Parag Badani (pbadani) and others added 14 commits May 4, 2026 13:04
…revent CME

The streams and sourceStreams lists in QueryRegistryImpl were plain
ArrayLists. The Kafka Streams state listener thread calls
unregisterQuery() (which removes from streams via removeAll) while
the CommandRunner thread iterates streams during
createOrReplacePersistentQuery and updateStreamsPropertiesAndRestartRuntime.
This race could produce ConcurrentModificationException, crashing
command processing and leaving the server unable to register queries.

Replacing with CopyOnWriteArrayList eliminates the CME: iterators
operate on a snapshot so concurrent structural modifications are safe.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ntime

restartStreamsRuntime() (triggered by ALTER SYSTEM from the CommandRunner
thread) reassigns the kafkaStreams field while HTTP handler threads
concurrently read it via state(), getKafkaStreams(), and lag methods.
Without volatile the JMM gives no visibility guarantee: HTTP threads
can see the old, closed KafkaStreams reference causing NPEs or stale
metrics after a runtime restart.

Declaring the field volatile ensures the updated reference is visible
to all threads immediately after the write completes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…le health status

The state field is written by the Runner thread (when entering DEGRADED
due to corruption, incompatible commands, or command topic deletion)
and read by HTTP handler threads via checkCommandRunnerStatus() and
getCommandRunnerDegradedReason().

Without volatile the JMM does not guarantee visibility: HTTP threads can
continue reading the cached RUNNING value long after the Runner thread
writes DEGRADED, masking node degradation from health checks and load
balancers.

Declaring the field volatile ensures every read observes the most
recently written value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
terminateCluster() is invoked from within the Runner executor thread
(via fetchAndRunCommands). It previously called this.close() which
called closeEarly() -> executor.awaitTermination(15 000 ms). Since the
calling thread IS a task of that executor, awaitTermination blocked
until timeout (15 s), then terminateCluster called closeEarly() a
second time adding another 15 s. Total: ~30 s delay before
serverState.setTerminated() was reached during TERMINATE CLUSTER.

The fix: set closed=true and call commandStore.wakeup() directly, which
is all that is needed to signal the Runner loop to stop. The loop exits
naturally after terminateCluster returns and commandStore.close() runs
in the Runner.run() finally block. The external close() path is
unaffected and continues to call executor.awaitTermination() as before.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…query removal

unregisterQuery() removes a QueryId from persistentQueries before it
removes the same id from insertQueries. A concurrent call to
getInsertQueries() (e.g. from EngineContext.throwIfInsertQueriesExist
during DROP SOURCE processing) could therefore observe the id still in
insertQueries but receive null from persistentQueries.get(). The
subsequent filterQueries.test(sourceName, null) call NPEs because the
FILTER_QUERIES_WITH_SINK / SOURCE lambdas dereference the query object.

Adding Objects::nonNull filter between the map and the predicate closes
the race window: the id is simply treated as already unregistered and
excluded from the result set, which is the correct semantics.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tart

topolgogiesToAdd in SharedKafkaStreamsRuntimeImpl was a plain ArrayList.
start() appends KafkaFuture<Void> entries while stop() iterates and
clears the list; concurrent calls (or a future restart) could produce
ConcurrentModificationException or cause stop() to await futures that
were issued against an already-closed KafkaStreams instance.

Two changes:
1. Replace new ArrayList<>() with Collections.synchronizedList() so
   concurrent add/iterate operations are safe under lock.
2. Clear topolgogiesToAdd inside restartStreamsRuntime() immediately
   after closing the old KafkaStreams. Futures obtained from the closed
   instance will never complete; clearing prevents subsequent stop()
   calls from blocking on them indefinitely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… SharedKafkaStreamsRuntimeImplTest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nResume not onPause

Copy-paste error caused resume() to notify listeners of a PAUSE event instead of RESUME.
Any listener distinguishing the two (e.g. metrics, external state tracking) would mishandle
RESUME commands for all shared-runtime queries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…deregistration

unregisterQuery() removes a query from persistentQueries before removing it from
createAsQueries. A concurrent getCreateAsQuery() call in that window would call
Optional.of(null) and throw NPE, crashing the DROP SOURCE command handler.
Change to Optional.ofNullable to return empty instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…time

overrideStreamsProperties() (ALTER SYSTEM) runs on the CommandRunner thread while
HTTP handler threads read streamsProperties via getStreamProperties(). Without volatile
the JMM does not guarantee HTTP threads will observe the updated configuration,
so stale properties can be served indefinitely after ALTER SYSTEM.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
….close()

A ScheduledExecutorService (ksql-csu-metrics-reporter) is created in the constructor
but never shut down. Each persistent query in dedicated-runtime mode leaks one idle
thread for its lifetime. Repeated CREATE/DROP cycles accumulate threads until the JVM
exhausts its thread limit. Call executorService.shutdown() in close() to clean up.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…PE in evict()

Two bugs in TimeBoundedQueue:
1. Capacity ignored: new ConcurrentLinkedQueue<>(EvictingQueue.create(cap)) copies the
   (empty) EvictingQueue's elements into an unbounded ConcurrentLinkedQueue, silently
   discarding the capacity. Under sustained error conditions errors accumulated for up
   to 1 hour (the eviction duration) before being removed, risking OOM.
   Fix: use Queues.synchronizedQueue(EvictingQueue.create(cap)) so the EvictingQueue
   itself is used and drops the oldest entry when the queue exceeds capacity.

2. NPE in evict(): the old code did isEmpty() then peek().getTimestamp(). A concurrent
   poll() between those two calls could empty the queue, making peek() return null and
   throwing NPE. The FindBugs warning was suppressed rather than fixed.
   Fix: capture the peek result in a local variable and check for null.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…logy state store detection

"__*__" is invalid Java regex — `*` quantifies the preceding `_`, matching only
underscore-only strings. Named-topology state directories like "__queryId__" were
never matched, so the whole application directory was treated as the cleanup unit.
This caused running bin-packed query state stores to be deleted on restart.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…y has no matching query

getAllTopologies() can return a topology whose name is absent from collocatedQueries
if the two data structures fall out of sync (e.g. a query registered but not yet
started, or removed between getAllTopologies() and the loop body). Without the null
check, query.updateTopology() NPE'd and killed the CommandRunner thread permanently,
leaving the server unable to process any further commands.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pbadani
Parag Badani (pbadani) requested a review from a team as a code owner May 18, 2026 06:54
@confluent-cla-assistant

Copy link
Copy Markdown

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Parag Badani (pbadani) and others added 14 commits May 18, 2026 14:11
…aImpl

The TimeBoundedQueue capacity fix added a com.google.common.collect.Queues
import out of alphabetical order between ImmutableList and ImmutableMap,
which breaks the checkstyle validate goal on this module.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…metadata

BinPackedPersistentQueryMetadataImpl declares its own everStarted, isPaused,
and corruptionCommandTopic fields. These shadow the parent class's
AtomicBoolean / volatile versions in QueryMetadataImpl, so every method on a
bin-packed query resolves to the unsafe subclass copy. start(), pause(),
resume(), and setCorruptionQueryError() run on the CommandRunner thread
while getQueryStatus(), getState(), and hasEverBeenStarted() are called from
HTTP handler threads. Without volatile, HTTP threads can read stale values
and report the wrong query status after a state transition.

The public everStarted field is also read directly by
SharedKafkaStreamsRuntimeImpl.start() (collocatedQueries.get(...).everStarted),
which makes the cross-thread visibility issue observable even without going
through any accessor method.

Declaring all three fields volatile re-establishes the JMM happens-before
guarantee that the parent class already provided.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…MetadataImpl

updateTopology() is invoked from SharedKafkaStreamsRuntimeImpl.restartStreamsRuntime()
on the CommandRunner thread to swap in a fresh NamedTopology after a runtime
restart. HTTP handler threads concurrently read the same field via getTopology()
and getTopologyDescription() (the latter calls topology.describe() to render
the topology in the SHOW TOPOLOGIES response). Without volatile, the JMM gives
no visibility guarantee for the reassignment, so an HTTP thread can observe
the old reference or a partially-published replacement and either return stale
data or fail with a NullPointerException on its internals.

Declaring the field volatile ensures the updated reference is immediately
visible to all threads after the write completes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…art()

start() previously looked up the same query twice:
  if (collocatedQueries.containsKey(queryId)
      && !collocatedQueries.get(queryId).everStarted) {
    ... addNamedTopology(collocatedQueries.get(queryId).getTopology()) ...
  }

A concurrent stop() removing the entry between the containsKey() and get() calls
made get() return null and crashed the CommandRunner thread with NPE on the
".everStarted" dereference. collocatedQueries is a ConcurrentHashMap, so its
individual operations are thread-safe but the pair is not atomic.

Replace the double lookup with a single get() + null-check; the worst-case race
now degrades to a deterministic IllegalArgumentException with the existing
"not registered to runtime" message. As a side benefit, the new code does one
hash lookup instead of three.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…_TIMEOUT_MS

HARouting.executeRounds() called future.get() with no timeout on every per-peer
pull-query fetch. If any peer ksqldb node hung (GC pause, blocked downstream
Kafka, lost network), the router thread blocked indefinitely on get(), pinning
one thread from the router pool per concurrent pull query targeting that peer.
A single sick host could therefore drag down the entire fleet's pull query
throughput.

Read the existing KSQL_QUERY_PULL_FORWARDING_TIMEOUT_MS_CONFIG (default 20s)
and pass it to future.get(timeout, MILLISECONDS). On TimeoutException, cancel
the worker (interrupting the wedged blocking call so the thread is reusable),
record the timeout against the node's exception list, and add the partition
locations to the next round so they retry on the next host in priority order.

A non-positive config value preserves the old "block until done" behavior so
tests that don't stub the config — and any environment that deliberately wants
no timeout — keep working.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ownNow()

closeEarly() ignored awaitTermination's boolean return. If the Runner thread
did not finish within SHUTDOWN_TIMEOUT_MS (15s), the executor was left in an
undefined state — still technically running, but with no signal anywhere that
the orderly shutdown failed. Callers (server shutdown, terminate cluster
flows) would proceed believing the command runner had stopped.

Capture the return value. On false, log a warning and call executor.shutdownNow()
so the worker is interrupted (the Runner uses interrupt-aware blocking calls in
commandStore.getNewCommands and Thread.sleep, so the interrupt propagates).

Also wrap the long state-field declaration on line 94 to satisfy the 100-column
LineLength check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…irst report

report() previously did
  if (gauges.containsKey(name)) gauges.get(name).set(point);
  else { gauges.put(name, new gauge); metrics.addMetric(name, gauges.get(name)); }

When two reporters publish the same metric simultaneously they can both observe
the absent state, both put their own gauge (last write wins, the other gauge is
orphaned and silently drops all subsequent updates), and both call
metrics.addMetric — which the Kafka metrics registry rejects as "metric already
exists", propagating an exception out of the metrics reporter callback.

Between containsKey and the subsequent get on line 55 there is a separate NPE
risk: a concurrent cleanup() can remove the entry, making get() return null and
the .dataPointRef dereference NPE.

Replace the pattern with a fast-path single get() (the common case where the
gauge already exists) plus putIfAbsent on the slow path. addMetric is called
exactly once per metric name, by the thread that actually installs the gauge;
losers update the winning gauge's data point. No more orphaned gauges, no more
double addMetric calls, no more containsKey/get NPE window.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ainst NPE

onDeregister called perQuery.get(id).onDeregister() with no null check, while
every other callback in this class null-checks. A duplicate onDeregister
(shutdown-on-error path followed by normal close), or a deregister for a query
that was never registered (sandbox cleanup), would NPE — masking the original
shutdown cause and leaving the metric entry leaked in the map.

Switch to atomic remove and only invoke onDeregister when the listener was
actually present.
…r.onCreate

onCreate did containsKey(id) followed by put(id, new PerQueryListener(...)) on
a ConcurrentHashMap. Two concurrent onCreate calls for the same queryId (e.g.,
sandbox commit racing with the REST listener registration path) could both
pass the contains-check and both construct + register a PerQueryListener with
Kafka Metrics. The second registration either throws on duplicate MetricName,
or the loser's listener is overwritten in the map while its gauges remain
registered — a permanent metric leak.

Switch to computeIfAbsent so put-if-absent is atomic.
The Vert.x connection-close handler ConnectionQueries.handle() iterated the
live queries set while calling PushQueryHolder.close() on each. close() invokes
the closeHandler (this::removeQuery), which mutates the same HashSet
mid-iteration — throwing ConcurrentModificationException on the second query
and leaving every subsequent push query (and its KafkaStreams resources)
never closed. Each dropped HTTP connection that held more than one push query
leaked resources.

Iterate over a snapshot so close() can safely remove from the set.
…er ordering

Two related issues in DefaultServiceContext:

1. close() invoked adminClient.close(), topicAdminClient.close(), and
   ksqlClient.close() sequentially with no exception handling. A transient
   Admin.close() failure (broker reset, auth re-validation, etc.) skipped the
   subsequent closes — stranding the topic admin / ksql client connections,
   threads, and sockets on shutdown. Wrap each close in a per-client try/catch
   that logs the failure and continues.

2. MemoizedSupplier.get() set initialized = true BEFORE invoking the underlying
   supplier. If the supplier threw on first call, isInitialized() returned
   true even though no resource existed; close() would then call get() again,
   either creating a brand-new resource just to immediately close it, or
   re-throwing the same exception and masking the original error. Set
   initialized only after a successful supplier.get().
… state store

StorageUtilizationMetricsReporter.handleRemovedSstFileSizeMetric decremented
numberStatefulTasks unconditionally on every total-sst-files-size metric
removal. handleNewSstFilesSizeMetric increments it once per *task* (only when
the first store for that task is seen), so a task with N state stores caused
N decrements when one task departed. The static counter quickly drifts
negative and the num_stateful_tasks gauge becomes meaningless for capacity
dashboards and autoscaling decisions.

Move the decrement inside the "task fully gone" branch so increments and
decrements are paired per-task.
…yCleanup

isLeaked() short-circuited on foundInLocalCommands(resource) and returned true
without first verifying the corresponding query was terminated. Combined with
the substring-based matching inside foundInLocalCommands, this meant a stale
local-commands entry pointing at a queryId that has since been restarted —
or a user topic whose name happens to contain a known local-commands appId
substring — could cause findLeakedTopics() / findLeakedStateDirs() to flag
the live query's topics or state directories for deletion.

Run isCorrespondingQueryTerminated() before any other branch so a resource
that any currently live query claims (by queryId substring) is never reported
as leaked, regardless of what the local-commands set says.
ThroughputTotalMetric.throughputTotalMetrics was a plain HashMap. The
reporter's add()/remove() ran inside synchronized addMetric/removeMetric on
the Streams thread, but measure() is invoked by the Kafka Metrics polling
thread without holding that lock and iterates the map via
.values().stream().reduce(...). Concurrent mutation while iterating throws
ConcurrentModificationException — observable as intermittent reporter errors
and metric value corruption.

Switch the backing map to ConcurrentHashMap so iteration is weakly consistent
and writes are safe under concurrent reads.
…ntext leak

DefaultConnectClient owns a pooled CloseableHttpClient (its own connection
manager, evictor thread, and SSLContext) but had no close() method, and
ConnectClient as an interface had no close at all. DefaultServiceContext.close()
therefore left these resources stranded on every ServiceContext shutdown —
accumulating sockets, threads, and SSL state per sandbox / per-request
ServiceContext.

Add a default no-op close() to ConnectClient, override it in
DefaultConnectClient to close the underlying HttpClient, and have
DefaultServiceContext.close() close the connect client (when initialized) on
its own try/catch — mirroring the pattern just added for the admin and ksql
clients.
…untime

dropQuery called runtimesToSourceTopics.get(appId).removeAll(...) without
checking the map for the key. If dropQuery was invoked for a query that was
never registered with this assignor — for example a sandbox copy whose
transaction never committed, or a replayed drop after the runtime had already
been cleaned up — .get(...) returned null and .removeAll NPE'd, crashing the
CommandRunner thread and stalling all further DDL on that ksqlDB node.

Null-check the lookup, log a warning, and still remove the queryId from
idToRuntime so the assignor's own state stays consistent.
The default sleep lambda in retryWithBackoff caught InterruptedException and
only logged at DEBUG. Thread.sleep clears the interrupt flag when it throws,
so this swallowed the cancellation entirely — callers that depend on
Thread.interrupted() (CommandRunner, executor service shutdown, etc.) had no
way to observe a shutdown signal and would continue retrying until the
maxRetries budget was exhausted, prolonging shutdown.

Restore Thread.currentThread().interrupt() so the flag persists for callers.
The local-routing branch of PushRouting.handlePushQuery captures the
PushPhysicalPlanManager's closeable into an AtomicReference inside the first
thenApply(), then runs it from the .exceptionally(...) handler. If
pushPhysicalPlanManager.reset() or pushPhysicalPlanManager.closeable() threw
before closeable.set(...) executed, the AtomicReference held null and
closeable.get().run() NPE'd — masking the real cause that the handler was
trying to wrap into the surfaced KsqlException.

Null-check before invoking the close runnable so the original failure
propagates to the caller.
onQueued() called limitHandler.limitReached() with no null guard, but the
limit handler is installed by the REST resource *after* the query begins
producing rows. For small LIMITs (e.g., LIMIT 1) the Streams thread can hit
the limit and invoke onQueued before the resource ever calls setLimitHandler
— NPE'ing on the Streams thread and terminating the query.

Null-check the handler. setLimitHandler already has a passedLimit() branch
that fires limitReached when it is finally installed against an already-
reached limit, so the signal is not lost.
…pIsClosing

When a scalable-push catchup signalled latest (incrementing catchupJoiners)
and then closed abnormally before ever joining, catchupIsClosing decremented
the counter back to zero but did not notify the latest thread parked in
checkShouldWaitForCatchup's wait(10_000). The latest consumer stalled the
full 10s WAIT_TIME_MS — delaying scalable-push row delivery for every client
on the topic whenever a catchup connection dropped mid-flight.

notifyAll() inside the signalled-but-closing branch so latest wakes promptly.
…cs error

deleteTopics() caught Exception and threw a new KsqlException without the
underlying cause, so cluster-termination diagnostics lost the root failure
(broker auth, network, ACL, etc.) — operators saw "Exception while deleting
topics: ..." with no stack trace pointing at the actual problem.

Pass the caught exception as the cause.
…isher

When the client's Accept header for a print-topic request was not the
delimited content type, handlePrintPublisher sent a 406 and called
response().end() — but then FELL THROUGH to subscribe the PrintSubscriber to
the BlockingPrintPublisher. That started a KafkaConsumer poll loop writing
into an already-ended response: every write was silently dropped, but the
consumer kept polling broker connections until the end handler chain
eventually closed it.

Add the missing return.
…ose throws

CommandTopic.close() ran commandConsumer.close() followed by
commandTopicBackup.close() with no exception handling. A transient Kafka
error in the consumer's close (broker reset on shutdown, partition rebalance
in flight, etc.) skipped the backup close, leaking the backup file handle
and any watcher resources for the lifetime of the JVM.

Wrap each close in its own try/catch so a failure in one does not prevent
the other from running.
…istributingExecutor

Two related cleanups in executeInjected:

1. The rate-limiter check ran *after* createTransactionalProducer() +
   initTransactions(). initTransactions() registers a transactional.id with
   the broker, so a throttled request opened a server-side transactional
   registration AND a Kafka producer client that were never closed (the
   try/finally that closes the producer only covers the second try block,
   which is bypassed by the rate-limiter exception). Move the rate-limit
   check before the producer is created.

2. The first try block (around initTransactions) caught TimeoutException and
   generic Exception and threw KsqlServerException / KsqlStatementException
   without closing the producer. On every init-time failure the local
   producer client and any transactional broker registration leaked.
   closeQuietly() on both catch paths.
The regression test added in 7abada2 (regex fix) creates a directory
/tmp/cat/appId/__test__/ but does not clean it up. /tmp/cat/ is shared across
tests in this class, so when shouldDeleteExtraStateStores runs after the
named-topology test it finds the leftover __test__ directory and (now that
the regex is correct) constructs a QueryCleanupTask with a present queryId.
That code path dereferences KsqlConfig.getString(...) for the service id and
persistent-query prefix — both unstubbed on the mock, returning null and
NPE'ing inside QueryCleanupTask.

- Recreate /tmp/cat each test so leftovers from one test do not bleed into
  the next.
- Stub the two KsqlConfig.getString lookups in @before so the cleanup path
  has the values it needs when it does fire.
@VedarthConfluent

Copy link
Copy Markdown
Member

Code review

Found 1 issue:

  1. Duplicate import java.util.Map in QueryRegistryImplTest.java — the PR inserts a second import java.util.Map; at line 48, but the original file already has one at line 52. The post-merge file has both, which is a Checkstyle violation (the build enforces CustomImportOrder with no duplicates). This will fail CI.

import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.Map;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.OptionalInt;
import java.util.Set;

Fix: remove the newly added import java.util.Map; at line 48 (or consolidate so only one remains), and also add the missing import java.util.List; in the correct sorted position.


Other changes reviewed and look correct (historical context checked via git log and git blame):

  • BinPackedPersistentQueryMetadataImpl.resume() calling listener.onResume instead of onPause — genuine bug fix.
  • StorageUtilizationMetricsReporter decrement-after-guard fix — correct; prevents negative num_stateful_tasks for multi-store tasks.
  • TransientQueryCleanupService.isLeaked reordering — correct; live-query short-circuit must run first to prevent deletion of active topics.
  • DistributingExecutor rate-limit ordering + producer close on initTransactions failure — correct; the old ordering leaked the producer on every throttled request.
  • All volatile additions, CopyOnWriteArrayList, ConcurrentHashMap/putIfAbsent, and interrupt-restore in RetryUtil look correct and are consistent with historical threading fixes in the same files.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@VedarthConfluent

Copy link
Copy Markdown
Member

Code review

Found 3 issues:

  1. SharedKafkaStreamsRuntimeImpl.stop() iterates topolgogiesToAdd (now a Collections.synchronizedList) and then calls .clear() without holding the list's intrinsic lock. start() concurrently appends via topolgogiesToAdd.add(...). Per the Collections.synchronizedList contract, iteration requires external synchronized(list) { ... } or a ConcurrentModificationException may be thrown. restartStreamsRuntime() in the same PR correctly wraps the clear() in synchronized (topolgogiesToAdd) { ... }; stop() should do the same around the for-each + clear.

if (kafkaStreams.state().isRunningOrRebalancing()) {
try {
for (KafkaFuture<Void> toAdd : topolgogiesToAdd) {
toAdd.get();
}
topolgogiesToAdd.clear();
kafkaStreams.removeNamedTopology(queryId.toString(), resetOffsets)
.all()

  1. QueryMetadataImpl.TimeBoundedQueue.toImmutableList() calls ImmutableList.copyOf(queue) on a Queues.synchronizedQueue(...) without external synchronization. The Guava/JDK contract for synchronized collection wrappers requires manual synchronized(queue) { ... } while iterating; ImmutableList.copyOf uses the queue's iterator. queryErrors.add() runs on the Streams uncaught-handler thread while getQueryErrors()toImmutableList() runs from REST handlers, so this can race and throw CME. The comment at line 391 ("all queue operations are thread-safe under concurrent add/evict calls") overstates what synchronizedQueue provides for iteration.

}
public List<QueryError> toImmutableList() {
evict();
return ImmutableList.copyOf(queue);
}

  1. TransientQueryQueue.limitHandler is written by the REST thread in setLimitHandler() and read by the Streams thread in onQueued(), but the field is not volatile. The added null-check (final LimitHandler handler = limitHandler; if (handler != null) ...) prevents the NPE but does not establish a happens-before relationship — the Streams thread may continue to observe a stale null indefinitely under the Java Memory Model. The passedLimit() fallback in setLimitHandler() mitigates the user-visible symptom, but the field should be volatile to be correct (matching private volatile boolean closed = false; on line 43 of the same class, which handles an analogous cross-thread access).

private volatile boolean closed = false;
private final AtomicInteger remaining;
private LimitHandler limitHandler;
private CompletionHandler completionHandler;

// passedLimit() block fires limitReached once the handler is finally
// installed, so dropping it here is safe.
final LimitHandler handler = limitHandler;
if (handler != null) {
handler.limitReached();
}
}

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

Parag Badani (pbadani) added a commit that referenced this pull request May 22, 2026
…tQueryQueue LIMIT-handler null-guard (#11052)

* KSQL-14999 | fix(engine): never delete a live query's resources from TransientQueryCleanup

isLeaked() short-circuited on foundInLocalCommands(resource) and returned true
without first verifying the corresponding query was terminated. Combined with
the substring-based matching inside foundInLocalCommands, this meant a stale
local-commands entry pointing at a queryId that has since been restarted —
or a user topic whose name happens to contain a known local-commands appId
substring — could cause findLeakedTopics() / findLeakedStateDirs() to flag
the live query's topics or state directories for deletion.

Run isCorrespondingQueryTerminated() before any other branch so a resource
that any currently live query claims (by queryId substring) is never reported
as leaked, regardless of what the local-commands set says.

Adds regression test shouldNotConsiderLocalCommandLeakedIfQueryIsStillLive.

Backport of a4a7f81 from PR #11037 (8.0.x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* KSQL-14999 | fix(engine): null-check TransientQueryQueue.onQueued limit handler

onQueued() called limitHandler.limitReached() with no null guard, but the
limit handler is installed by the REST resource *after* the query begins
producing rows. For small LIMITs (e.g., LIMIT 1) the Streams thread can hit
the limit and invoke onQueued before the resource ever calls setLimitHandler
— NPE'ing on the Streams thread and terminating the query.

Null-check the handler. setLimitHandler already has a passedLimit() branch
that fires limitReached when it is finally installed against an already-
reached limit, so the signal is not lost.

Adds regression test shouldNotNpeWhenLimitReachedBeforeLimitHandlerSet.

Backport of 5db4f26 from PR #11037 (8.0.x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* KSQL-14999 | refactor(engine): inline limitHandler null-check in TransientQueryQueue.onQueued

Drop the intermediate local capture introduced in 5db4f26; read the
field directly inside the null-check. The field is only ever assigned
once (null -> handler) by setLimitHandler, so the inlined re-read cannot
observe a regression to null, and the call site stays a single line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* KSQL-14999 | fix(engine): make TransientQueryQueue.limitHandler volatile

Addresses Copilot review on PR #11052: limitHandler is written from the
REST thread (setLimitHandler) and read from the Streams thread (onQueued)
with no happens-before relation. Without volatile, the Streams thread can
read a stale null even after setLimitHandler ran — and in the race where
setLimitHandler observed passedLimit()=false, that stale read would silently
skip limitReached() forever, leaving the query stuck.

Mark the field volatile so the Streams thread always observes the latest
write. The field only ever transitions null -> non-null (setLimitHandler is
called once), so a subsequent inlined re-read in onQueued cannot regress
to null after a non-null check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* KSQL-14999 | refactor(engine): revert volatile + inline, restore local-capture in TransientQueryQueue.onQueued

Reverts both 57daec7 (inline) and ca5f6e8 (volatile). Restore the
local-capture pattern so the null-check and the call act on the same
reference without requiring volatile on the field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants