Run tests in parallel with isolated per-test tmp locations for speed - #218
Merged
Conversation
Build each test's git repository in the bats-managed BATS_TEST_TMPDIR instead of sharing the tests/ directory, and reference the transcrypt script via an absolute $TRANSCRYPT path instead of cwd-relative ../transcrypt. Every test is now independent of the others, so the suite can runs in parallel (eg: `bats --jobs 16` runs in 1/3 the time in testing). Isolate tests from developer and system git config by exporting GIT_CONFIG_GLOBAL/GIT_CONFIG_SYSTEM=/dev/null in the test helper, which supersedes the per-setting merge.conflictStyle workaround and protects against other leaking settings such as commit.gpgsign. Also fix a latent race the parallel runs exposed: the crypt-dir tests shared a hardcoded /tmp/crypt path and deleted it at start; each now uses a path under BATS_TEST_TMPDIR. CI installs GNU parallel and runs with --jobs 4.
Contributor
Author
|
I'm happy to also submit on the transcript-v3 branch depending on how you decide to merge/rebase/cherry-pick/whatever. |
Collaborator
|
This is a tremendous improvement @mattmc3, thanks very much! |
Collaborator
|
Thanks for offering to deal with the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #217:
Makes the bats suite isolated and parallel-safe, per the approach
outlined in the issue I just opened:
$BATS_TEST_TMPDIRinstead of theshared
tests/directory; bats owns cleanup, sonuke_git_repo,cleanup_all, and the flag-file guard are removed.$TRANSCRYPTrather thancwd-relative
../transcrypt.GIT_CONFIG_GLOBAL=/dev/nullandGIT_CONFIG_SYSTEM=/dev/null(git 2.32+), superseding themerge.conflictStyleworkaround./tmp/crypt.bats --jobs 4 tests/.No changes to the transcrypt script itself. Serial
bats tests/stillwork, but new runs can specify
bats --jobs 4 testsor however many make sense.Local test timing: 95s serial → 26s with
--jobs 16; 79/79 tests pass.Note that the macOS users and GH runners needed to
brew install parallel