Skip to content

feat: retry transient Yandex Cloud errors - #701

Merged
nikolaymatrosov merged 13 commits into
mainfrom
feat/retry-transient-errors
Aug 4, 2026
Merged

feat: retry transient Yandex Cloud errors#701
nikolaymatrosov merged 13 commits into
mainfrom
feat/retry-transient-errors

Conversation

@nikolaymatrosov

Copy link
Copy Markdown
Member

Makes the action survive transient Yandex Cloud failures — notably RESOURCE_EXHAUSTED: The limit on maximum number of active operations has exceeded, which shows up when many jobs start at once — instead of failing the job on the first rejection.

Resolves #700.

Approach

@yandex-cloud/nodejs-sdk already ships retry middleware with the right retryable statuses; it is inert because retry defaults to false for non-idempotent methods and the call sites passed no options. A new src/retry.ts owns the backoff schedule and budget math and feeds the existing middleware. The token-exchange POST is not a gRPC call, so it gets a small hand-rolled wrapper on the same schedule.

  • Backoff: 10s → 20s → 40s → then every minute, with jitter. The jitter matters: the error being retried is caused by many jobs hitting the cloud at once, and unjittered retries would collide all over again. The SDK constants are 13333/80000 rather than 10000/60000 because the middleware multiplies by (1 + Math.random()) / 2, whose mean is 0.75 — using the nominal values directly would make real delays 25% short.
  • Retryable gRPC statuses: RESOURCE_EXHAUSTED, UNAVAILABLE, UNKNOWN, INTERNAL.
  • Idempotency: every Create/Delete sends one idempotency-key UUID generated before the call, so all retries share it. This is what makes retrying non-idempotent methods safe — UNAVAILABLE/UNKNOWN can mean the request succeeded but its response was lost, and without a shared key that retry would create a second, untracked, billed VM.
  • HTTP retries: 429, any 5xx, and response-less network failures. Other 4xx and non-axios errors are not retried. The POST also gained timeout: 30_000, without which a half-open connection would hang forever and the budget would mean nothing.

New input

retry-timeout, ISO 8601 duration, default PT5M. PT0S disables retries — and that is an exact behavioural restore, since retry: false is precisely what the middleware defaulted to before. Values above PT30M are clamped with a warning, never a validation error: an oversized value must not fail the job at startup. Unparseable input resolves to zero (moment silently yields 0 rather than throwing), which disables retries and logs an info line so the typo is visible.

The budget is per request, not per job.

⚠️ This PR also ships a year of un-shipped dependency drift

dist/ is committed and is what GitHub actually executes. It had not been rebuilt since b149198 (2025-08-08), because .github/workflows/check-dist.yml runs npm run build — which is tsc, writing to lib/ — and then diffs dist/. That check has therefore always been trivially green, and every dependabot bump since August landed without a bundle rebuild.

Rebuilding the bundle here consequently pulls in changes unrelated to this feature:

  • @actions/core 1.11.1 → 2.0.2 (major)
  • @actions/github 6.0.1 → 7.0.0 (major)
  • @grpc/grpc-js 1.13.4 → 1.14.3
  • axios 1.11.0 → 1.13.5

None of these declare engines, so using: 'node20' is not violated. The new dist/xds/, dist/proto/xds/ and dist/protoc-gen-validate/ asset directories are grpc-js ORCA/channelz proto assets from that drift, not from this feature; they were verified byte-identical to their node_modules originals. Worth a smoke test before tagging a release.

Fixing check-dist.yml to run npm run package belongs in a separate PR that can first confirm ncc output is byte-reproducible on CI's Node 20.

Known limitation

Documented in the design doc: the SDK's errorMetadataMiddleware runs inside retryMiddleware, and it rethrows as ApiError (which extends Error, not ClientError) whenever the failed call received response headers. retryMiddleware bails on anything that is not a ClientError, so those failures are not retried regardless of retryableStatuses. Coverage is partial. The #700 case is unaffected — its log reads ClientError:, meaning no headers arrived.

Testing

29 new unit tests covering the schedule, the budget→attempts math, the clamp, the options factory, the retry-log formatting, the HTTP wrapper's classification and jitter bounds, and parseRetryTimeout. Full suite: 62 passed, 1 pre-existing skip.

The two call-site wiring tasks ship without new tests: this repo has no gRPC or HTTP mocking infrastructure and the existing suite is pure-function only. The options object's correctness is covered by unit tests, and review verified against the SDK sources that metadata — and therefore the idempotency key — is forwarded unchanged to every retry attempt.

Follow-ups

  • Fix check-dist.yml to run npm run package so the bundle is actually verified.
  • Confirm empirically that Yandex Cloud honours Idempotency-Key as gRPC metadata (it is documented as an HTTP header). If it does not, Create should retry only RESOURCE_EXHAUSTED, which is a pre-admission rejection and cannot duplicate.
  • src/main.ts:206/:236 — stray trailing apostrophe inside the core_error template literals.

🤖 Generated with Claude Code

errorMetadataMiddleware runs inside retryMiddleware once response headers
have arrived, wrapping failures as ApiError instead of ClientError, so
retryMiddleware's instanceof guard skips them regardless of
retryableStatuses. Document this as a known, accepted gap: issue #700's
own log line shows the failure arrived before headers (ClientError, not
ApiError), so the reported case is unaffected.
…ryableError

- Set axios timeout: 30_000 on the token-exchange POST so a half-open
  connection can't hang forever without the retry budget ever applying.
- destroyVm logged 'Create VM' / 'Failed to create instance' copy-pasted
  from createVm; corrected to the delete equivalents and added the
  matching idempotency-key comment at the delete call site.
- Added a test for grpcRetryOptions' onRetryableError, the only untested
  branch in retry.ts, asserting the exact info() message emitted with a
  real nice-grpc ClientError.

Rebuilds dist/ to match.
@mergify

mergify Bot commented Aug 3, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

The schedule opened at 10s, so a hiccup that cleared in a moment still cost
ten seconds. Prepend three fast steps and let the climb double the whole way:
1s, 2s, 4s, 8s, 16s, 32s, then a fixed minute.

Keeping it a pure doubling preserves the invariant the module rests on -- the
SDK computes min(cap, 2^k * base) and never sees CLIMB_MS, so the table and
the middleware only agree while the table is geometric.

The 5-minute default now buys 9 retries instead of 6.
@nikolaymatrosov
nikolaymatrosov merged commit 9adc996 into main Aug 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add retries for transient RESOURCE_EXHAUSTED errors

1 participant