Skip to content

Wait out longer CloudKit outages when stapling - #9230

Merged
myleshorton merged 1 commit into
mainfrom
staple-retry
Oct 28, 2026
Merged

myleshorton merged 1 commit into
mainfrom
staple-retry

Conversation

@myleshorton

@myleshorton myleshorton commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Apple's CloudKit ticket-delivery endpoint has now failed xcrun stapler staple with NSURLErrorDomain -1005 after notarization was already accepted twice in four days — the v10.0.4-beta.3 release run (Oct 25, run 40008674087) and the Oct 28 nightly dispatch (run 40765793487). Both outages outlasted retry-transient.sh's default three attempts, which only wait ~90 seconds in total, so the whole macOS build failed and each release needed a manual gh run rerun --failed.

  • scripts/ci/retry-transient.sh gains a -b BACKOFF flag: the wait before attempt N+1 is N*BACKOFF seconds (default 30, the previously hardcoded value, so the gomobile call site is unchanged).
  • The Makefile's STAPLER_RETRY becomes -n 6 -b 60: six attempts waiting 60/120/180/240/300s between them — 15 minutes of patience for a staple/validate that is idempotent and already past notarization, instead of failing the platform build.
  • Leading-zero values are rejected for both -n and -b (a round-1 gate catch): $((...)) arithmetic reads 010 as octal 8 and chokes on 08, so -b 08 would have broken the retry loop instead of waiting 8 seconds.

Testing

  • bash -n clean; exercised the script directly: success passthrough, retry loop with -n 2 -b 1 (1s elapsed, correct messages), non-matching -m failure exits immediately with the command's code, and 0/08/010/x/empty are rejected for both flags with exit 2.
  • make -n notarize-darwin confirms both staple and validate expand to retry-transient.sh -n 6 -b 60 -- xcrun stapler ....
  • Not covered: a real CloudKit outage can't be reproduced on demand; worst-case added wall-clock on a persistent staple failure is 15 minutes, well inside the macOS job's timeout.

Risk

  • CI-only. Riskiest line: the shared option parsing in retry-transient.sh — the other call site (gomobile bind) passes neither -n nor -b and keeps the exact previous behavior (3 attempts, 30s base).

How to revert

Revert the commit; no state to undo.

Pre-PR review (local Codex gate)

  • Rounds: 2 — final: round=2 range=origin/main...HEAD CODEX GATE: verdict=approve critical=0 important=0 minor=0 preexisting=0 reviewed=5
  • Reviewed at: 73edcbbdb
  • Exit: approve (high reasoning effort; the gate read both call sites, the workflows that invoke the Makefile targets, and the option-parsing edge cases)
  • Step back: not needed
  • Fixed: scripts/ci/retry-transient.sh — -b validation accepted leading-zero values that $((...)) reads as octal (-b 08 would error, -b 010 would wait 8s); both -n and -b now reject leading zeros.
  • Dismissed: none
  • Pre-existing (not introduced here, worth filing): none
  • Proposed AGENTS.md additions: none

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Improvements
    • Automated retries for temporary stapling and validation failures now allow up to six attempts, with a 60-second interval between attempts. Retry intervals can also be configured for other automated tasks. Invalid retry counts and intervals, including values with leading zeros, are now rejected to help prevent unexpected retry behavior.

Apple's ticket-delivery endpoint failed stapling with NSURLErrorDomain
-1005 for longer than retry-transient.sh's default three attempts
(~90s of waiting) on the v10.0.4-beta.3 release run and again on the
2026-10-28 nightly, after notarization had already been accepted both
times. Add a -b flag for the backoff base and give the stapler six
attempts at a 60s base, so it rides out up to 15 minutes of CloudKit
flakiness instead of failing the whole platform build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 21:48
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: df44c50c-cca0-420c-8365-37964c40b789
Commits

Reviewing files that changed from the base of the PR and between 7d32f92 and 73edcbb.

Files selected for processing: 2
  • Makefile
  • scripts/ci/retry-transient.sh

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

Walkthrough

The retry script now supports a configurable backoff and validates retry and backoff values. The Makefile configures the stapling retry command with six attempts and a 60-second backoff.

Changes: CI Retry Configuration

Layer / File(s) Summary
Add configurable retry backoff
scripts/ci/retry-transient.sh
The script adds -b BACKOFF, defaults it to 30 seconds, and waits attempt * backoff seconds before each retry. It rejects invalid values for -n and -b, including numbers with leading zeros.
Configure stapling retries
Makefile
STAPLER_RETRY passes -n 6 -b 60 to the retry script.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: atavism

Merge Risk: Minimal · up to 73edc

The longer stapling retry window introduces no identified merge-blocking risk. Merge after normal checks pass.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: extending retry time for CloudKit-related stapling outages.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.)

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

No active review issues remain for the stapling retry changes.

No open issues 🎉

What changed in this PR

This PR makes macOS release builds more resilient to CloudKit outages after notarization by extending retries for stapling and validation.

Changes:

  • Adds configurable retry backoff with -b, preserving the existing 30-second default.
  • Configures stapler operations for six attempts with 15 minutes of total waiting per operation.
  • Rejects leading-zero values for attempts and backoff to avoid Bash octal arithmetic issues.
File Description
scripts/​ci/​retry-transient.sh Adds configurable backoff, validates numeric options, and updates usage documentation.
Makefile Sets stapler retries to six attempts with a 60-second base backoff.

Review effort: Balanced

@myleshorton
myleshorton merged commit 80ad548 into main Oct 28, 2026
20 checks passed
@myleshorton
myleshorton deleted the staple-retry branch October 28, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants