feat: retry transient Yandex Cloud errors - #701
Merged
Conversation
errorMetadataMiddleware runs inside retryMiddleware once response headers have arrived, wrapping failures as ApiError instead of ClientError, so retryMiddleware's instanceof guard skips them regardless of retryableStatuses. Document this as a known, accepted gap: issue #700's own log line shows the failure arrived before headers (ClientError, not ApiError), so the reported case is unaffected.
…ryableError - Set axios timeout: 30_000 on the token-exchange POST so a half-open connection can't hang forever without the retry budget ever applying. - destroyVm logged 'Create VM' / 'Failed to create instance' copy-pasted from createVm; corrected to the delete equivalents and added the matching idempotency-key comment at the delete call site. - Added a test for grpcRetryOptions' onRetryableError, the only untested branch in retry.ts, asserting the exact info() message emitted with a real nice-grpc ClientError. Rebuilds dist/ to match.
|
Tick the box to add this pull request to the merge queue (same as
|
The schedule opened at 10s, so a hiccup that cleared in a moment still cost ten seconds. Prepend three fast steps and let the climb double the whole way: 1s, 2s, 4s, 8s, 16s, 32s, then a fixed minute. Keeping it a pure doubling preserves the invariant the module rests on -- the SDK computes min(cap, 2^k * base) and never sees CLIMB_MS, so the table and the middleware only agree while the table is geometric. The 5-minute default now buys 9 retries instead of 6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the action survive transient Yandex Cloud failures — notably
RESOURCE_EXHAUSTED: The limit on maximum number of active operations has exceeded, which shows up when many jobs start at once — instead of failing the job on the first rejection.Resolves #700.
Approach
@yandex-cloud/nodejs-sdkalready ships retry middleware with the right retryable statuses; it is inert becauseretrydefaults tofalsefor non-idempotent methods and the call sites passed no options. A newsrc/retry.tsowns the backoff schedule and budget math and feeds the existing middleware. The token-exchange POST is not a gRPC call, so it gets a small hand-rolled wrapper on the same schedule.13333/80000rather than10000/60000because the middleware multiplies by(1 + Math.random()) / 2, whose mean is 0.75 — using the nominal values directly would make real delays 25% short.RESOURCE_EXHAUSTED,UNAVAILABLE,UNKNOWN,INTERNAL.Create/Deletesends oneidempotency-keyUUID generated before the call, so all retries share it. This is what makes retrying non-idempotent methods safe —UNAVAILABLE/UNKNOWNcan mean the request succeeded but its response was lost, and without a shared key that retry would create a second, untracked, billed VM.timeout: 30_000, without which a half-open connection would hang forever and the budget would mean nothing.New input
retry-timeout, ISO 8601 duration, defaultPT5M.PT0Sdisables retries — and that is an exact behavioural restore, sinceretry: falseis precisely what the middleware defaulted to before. Values abovePT30Mare clamped with a warning, never a validation error: an oversized value must not fail the job at startup. Unparseable input resolves to zero (moment silently yields 0 rather than throwing), which disables retries and logs an info line so the typo is visible.The budget is per request, not per job.
dist/is committed and is what GitHub actually executes. It had not been rebuilt since b149198 (2025-08-08), because.github/workflows/check-dist.ymlrunsnpm run build— which istsc, writing tolib/— and then diffsdist/. That check has therefore always been trivially green, and every dependabot bump since August landed without a bundle rebuild.Rebuilding the bundle here consequently pulls in changes unrelated to this feature:
@actions/core1.11.1 → 2.0.2 (major)@actions/github6.0.1 → 7.0.0 (major)@grpc/grpc-js1.13.4 → 1.14.3axios1.11.0 → 1.13.5None of these declare
engines, sousing: 'node20'is not violated. The newdist/xds/,dist/proto/xds/anddist/protoc-gen-validate/asset directories are grpc-js ORCA/channelz proto assets from that drift, not from this feature; they were verified byte-identical to theirnode_modulesoriginals. Worth a smoke test before tagging a release.Fixing
check-dist.ymlto runnpm run packagebelongs in a separate PR that can first confirm ncc output is byte-reproducible on CI's Node 20.Known limitation
Documented in the design doc: the SDK's
errorMetadataMiddlewareruns insideretryMiddleware, and it rethrows asApiError(which extendsError, notClientError) whenever the failed call received response headers.retryMiddlewarebails on anything that is not aClientError, so those failures are not retried regardless ofretryableStatuses. Coverage is partial. The #700 case is unaffected — its log readsClientError:, meaning no headers arrived.Testing
29 new unit tests covering the schedule, the budget→attempts math, the clamp, the options factory, the retry-log formatting, the HTTP wrapper's classification and jitter bounds, and
parseRetryTimeout. Full suite: 62 passed, 1 pre-existing skip.The two call-site wiring tasks ship without new tests: this repo has no gRPC or HTTP mocking infrastructure and the existing suite is pure-function only. The options object's correctness is covered by unit tests, and review verified against the SDK sources that
metadata— and therefore the idempotency key — is forwarded unchanged to every retry attempt.Follow-ups
check-dist.ymlto runnpm run packageso the bundle is actually verified.Idempotency-Keyas gRPC metadata (it is documented as an HTTP header). If it does not,Createshould retry onlyRESOURCE_EXHAUSTED, which is a pre-admission rejection and cannot duplicate.src/main.ts:206/:236— stray trailing apostrophe inside thecore_errortemplate literals.🤖 Generated with Claude Code