Repository navigation
Report poller target on each poll and in the worker heartbeat - #3159
Open
veeral-patel wants to merge 2 commits into
Open
veeral-patel wants to merge 2 commits into
veeral-patel wants to merge 2 commits into
Conversation
An autoscaling poller pool now sends its current target with every workflow and activity poll, so the server can spread scaling suggestions across a fleet by share rather than by whoever happens to receive the most tasks. Suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's share of suggestions tracks its share of pollers -- growth is proportional to current size, which preserves imbalance between identical workers instead of correcting it. A reported target gives the server the one number it needs to compare a worker against the fleet mean. AsyncPoller owns the scale report handle, so it installs the target supplier on each poll task when it creates that handle. The poll request is rebuilt per poll because the target moves; pools with no supplier installed -- Nexus, and any non-autoscaling pool -- send nothing, which the server reads as "no report". The api submodule moves to a branch carrying the new proto field. It is pinned one commit ahead of the previously recorded api commit rather than to current api main, which would also drag in Callback.NexusHandler.source_context and fail generatePayloadLimitValidator for reasons unrelated to this change.
veeral-patel
marked this pull request as ready for review
October 20, 2026 20:11
veeral-patel
force-pushed
the
vpatel/poller-load-balancing
branch
from
October 22, 2026 03:28
00456fd to
ada916a
Compare
The poller target is already sent on each poll, but matching consumes it inside the scaling decision and discards it -- nothing persists it and nothing can observe it. Reporting it on the heartbeat too lets the server read it per worker and export it as a metric, making fleet-wide poller distribution visible without instrumenting every SDK. WorkerHeartbeat already carries four WorkerPollerInfo sub-messages, so this is per-task-type for free, including the normal-vs-sticky workflow split. PollScaleReportHandle is a local inside AsyncPoller.start() and is not reachable from the heartbeat builder. PollerTracker already exists to bridge exactly that gap -- its javadoc says so -- so the target supplier goes there, and buildPollerInfo reads it alongside the in-flight count it already reads. AsyncNexusPollTask now overrides setPollerTargetSupplier purely to feed its tracker. It had taken the interface's no-op default because the nexus poll request has no field for a target, but it does have an autoscaler and a heartbeat slot. Fixed-size pools run MultiThreadedPoller, which has no scale report handle, so no supplier is installed and the tracker reports 0 -- which the proto defines as unknown. The api submodule moves to the commit carrying the new field.
veeral-patel
force-pushed
the
vpatel/poller-load-balancing
branch
from
October 22, 2026 03:40
ada916a to
dcb6b69
Compare
Quinn-With-Two-Ns
approved these changes
Oct 28, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An autoscaling poller pool now reports its current target in two places:
PollerScalingInfo.poller_targeton the workflow and activity poll requests.WorkerPollerInfo.poller_target, for all four poller types.Why
Scaling suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's share of suggestions tracks its share of pollers. Growth is proportional to current size, which preserves imbalance between identical workers instead of correcting it — measured at 74/12/14 percent across three identical workers, with a different winner each run.
Reporting the target gives the server the one number it needs to compare a worker against the fleet mean and withhold suggestions that would push it further from its share.
The poll-request copy is consumed inside the matching decision and discarded. Reporting it on the heartbeat too lets a follow-up read it per worker and export it as a metric, making fleet-wide poller distribution visible without instrumenting every SDK.
How
Poll path —
AsyncPollerinstalls a target supplier on each poll task when it creates that task'sPollScaleReportHandle. The poll request is rebuilt per poll, because the target moves. Adefaultno-op onPollTaskAsynckeeps pollers with nothing to report from having to implement it.Heartbeat —
PollScaleReportHandleis a local insideAsyncPoller.start()and is not reachable fromWorker.buildHeartbeatCallback.PollerTrackeralready exists to bridge exactly that gap — its javadoc says it "bridges the gap between poll tasks (created insidestart()) and heartbeat building" — so the supplier goes there, andbuildPollerInforeads it alongside the in-flight count it already reads.Each tracker is already the right one per type:
WorkflowWorkerhandsstickyPollerTrackerto the sticky poll task andpollerTrackerto the normal one, so the normal/sticky split works even though both share onePollerOptions.Design notes
AsyncNexusPollTasknow overridessetPollerTargetSupplierpurely to feed its tracker. It had taken the interface's no-op default because the nexus poll request has no field for a target — but it does have an autoscaler and a heartbeat slot, so the heartbeat reports it.MultiThreadedPoller, which has no scale report handle, so no supplier is installed and the tracker reports 0 — which the proto defines as unknown.Dependency, and the submodule bump
Needs temporalio/api#887.
The
temporal-serviceclient/src/main/protosubmodule moves tovpatel/poller-load-balancing-java-baseon the api repo — the same two proto commits cherry-picked onto the api commit this repo already recorded, so the bump is minimal andWorkerPollerInfois byte-identical to the PR branch.It is pinned there rather than to the api PR branch deliberately: that branch is based on current api
main, and taking it would also pull inCallback.NexusHandler.source_context, whichgeneratePayloadLimitValidatorrejects as an unclassified payload-bearing field — a failure unrelated to this change. The submodule should move to a released api version before merge.Testing
./gradlew :temporal-sdk:compileJavaand:temporal-sdk:compileTestJavapass; spotless applied. Not yet exercised end to end against a server — that has been done via the Go SDK (temporalio/sdk-go#2799), which shares the wire contract.