Skip to content

[FSSDK-13238] ci: retry only failed unit tests and upload results on failure - #661

Merged
muzahidul-opti merged 1 commit into
masterfrom
muzahid/native-test-retry
Oct 7, 2026
Merged

muzahidul-opti merged 1 commit into
masterfrom
muzahid/native-test-retry

Conversation

@muzahidul-opti

Copy link
Copy Markdown
Contributor

Summary

Unit tests now retry only the failing tests, and test results are uploaded when a job fails.

 unittests job
-  for attempt in 1 2 3:            # rebuild + re-run all 620 tests each time
-    run_unit_tests.sh || rm -rf build COVERAGE
+  run_unit_tests.sh                # xcodebuild -retry-tests-on-failure -test-iterations 3
+                                   # → only failed tests re-run, no rebuild
 Archive Test Results
-  if: steps.unit_tests.outcome != 'success'   # implicit success() && … → never ran
-  path: build/Logs                            # never existed
+  if: failure() && steps.unit_tests.outcome == 'failure'
+  path: COVERAGE/Logs/Test                    # single .xcresult with every retry
-timeout-minutes: 90
+timeout-minutes: 45

Why: the iPad job on the #660 merge commit (run 37558972075) failed after 3 full-suite attempts, each turning up different flaky tests. The archive step was skipped, so no results were available to debug them.

Evidence

  • Before: iPad job, 3 × 620 tests, a new set of flakes each attempt → ❌, and Check on failures (Archive Test Results) was skipped
  • After (local, temporary test that fails only on its first run):
    Retrying tests on failure. Running tests repeatedly up to 3 times.
    testTempFlaky (Iteration 1 of 3) failed
    testTempFlaky (Iteration 2 of 3) passed   ← only the failed test re-ran
    ** TEST SUCCEEDED **  exit=0
    
  • CI on this PR: all 4 unit test jobs

Merge Danger

Door: two-way (CI-only, revert restores the loop)
Blast Radius: unit test jobs only. Infra failures (e.g. simulator boot) are no longer auto-retried; use "Re-run failed jobs". Flaky tests are still masked, but the .xcresult now records them.

Issues

  • FSSDK-13238

🤖 Generated with Claude Code

Replace the 3x full-suite bash retry loop with xcodebuild's
-retry-tests-on-failure so only failing tests re-run. Fix the
archive step, which never ran (implicit success() in its if) and
pointed at build/Logs instead of COVERAGE/Logs/Test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@muzahidul-opti
muzahidul-opti merged commit 01d9a14 into master Oct 7, 2026
10 of 12 checks passed
@muzahidul-opti
muzahidul-opti deleted the muzahid/native-test-retry branch October 7, 2026 06:17
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 93.929%. remained the same — muzahid/native-test-retry into master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants