Fail loudly when preparing the emulator for a test fails - #37
Draft
ofalvai wants to merge 1 commit into
Draft
Conversation
The beforeEach hook in testBuilder swallowed every error from prepareEmulatorForTest, so an emulator in a bad state surfaced later as an unrelated assertion failure somewhere else in the scenario. It now logs and rethrows. Rethrowing on its own would have broken every scenario, because iOS "simctl terminate" exits non-zero when the app is not running, and the app is not running before the first test of a scenario. Verified on Xcode 26.6 / iOS 26.5: terminate reports "found nothing to terminate" identically whether the app was never installed or is merely idle, so the error text and the exit code cannot tell a missing app apart from a broken simulator. simctl appinfo, get_app_container and listapps do not help either, as they only report whether the app is installed, and their output is unchanged by the app running or not. So IOSEmulatorManager.endRunningApplication now establishes the running state up front with "simctl spawn booted launchctl list", where a running app appears as UIKitApplication:<bundle id> with a live pid, and skips terminate when the app is not running. Everything else propagates. Android needs no equivalent guard: "am force-stop" exits 0 whether or not the app is installed or running (verified against an emulator), and "pm clear" is already guarded by an existence check. Adding one there would only add an adb round-trip per test. Verified end to end against a live simulator and emulator for the installed-but-idle, never-installed and actually-running cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The beforeEach hook in testBuilder swallowed every error from prepareEmulatorForTest, so an emulator in a bad state surfaced later as an unrelated assertion failure somewhere else in the scenario. It now logs and rethrows.
Rethrowing on its own would have broken every scenario, because iOS "simctl terminate" exits non-zero when the app is not running, and the app is not running before the first test of a scenario. Verified on Xcode 26.6 / iOS 26.5: terminate reports "found nothing to terminate" identically whether the app was never installed or is merely idle, so the error text and the exit code cannot tell a missing app apart from a broken simulator. simctl appinfo, get_app_container and listapps do not help either, as they only report whether the app is installed, and their output is unchanged by the app running or not.
So IOSEmulatorManager.endRunningApplication now establishes the running state up front with "simctl spawn booted launchctl list", where a running app appears as UIKitApplication: with a live pid, and skips terminate when the app is not running. Everything else propagates.
Android needs no equivalent guard: "am force-stop" exits 0 whether or not the app is installed or running (verified against an emulator), and "pm clear" is already guarded by an existence check. Adding one there would only add an adb round-trip per test.
Verified end to end against a live simulator and emulator for the installed-but-idle, never-installed and actually-running cases.