docs(flaky-tests): document the Swift report sources added in CLI 0.15.5 - #333
dfrankland wants to merge 9 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
Pushed b4b583f, folding in the customer-facing instructions from this thread. Three things it covered that the pages did not, plus one correction:
One correction to the thread, worth knowing before these instructions go to a customer again: "we find |
|
Pushed bcb4e56: The Swift Testing and GitHub Actions snippets glob rather than naming both files — Also noted what the dedupe does: a file matched by more than one pattern is uploaded once. That is the question a glob raises and a file list does not, and the answer is not guessable. No new version note — globs ship in the same 0.15.5 these pages already require. That does assume analytics-cli#1201 lands before 0.15.5 goes stable; if it slips to 0.15.6, this needs a revisit. Merge order for the set, all gated on that release: analytics-cli#1201 → analytics-uploader#147 → this. |
`swift test --xunit-output` writes two files and neither is named what was asked for, which the Swift Testing page did not say: swift-testing results go to `<name>-swift-testing.xml` and XCTest ones to `<name>`, the latter only when `--parallel` is passed. A reader following the old page uploaded the wrong file, or silently uploaded half a run. Document `--swift-test-xunit-paths` as the way to upload them. The argument exists because that XML carries no file path for any test case, so each test is attributed to the file a language server says it is declared in — which is what CODEOWNERS attribution and file links need, and what `--junit-paths` does not do for these files. That attribution changes test ids once for anyone moving off `--junit-paths`, since `file` feeds the id, so the page warns rather than leaving a reset history to be discovered. Document `--use-experimental-xcresult-test-locations` as an opt-in beta on the XCTest page. It is hidden in `--help`, but it is the only way to give a *passing* xcresult test a file at all, and an Objective-C repository needs the `TRUNK_XCRESULT_TEST_LOCATIONS_*` budget raised to finish resolving — neither of which is discoverable without docs. Both flags ship in 0.15.5; the pages say so, since `latest` is still 0.15.4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the instructions drafted for a customer trying the flag, which covered ground the pages did not. A reader cannot tell whether the flag worked without being told what to look at, so document the offline trial: `--dry-run` with `--disable-quarantining`, because the quarantine lookup is the one call `--dry-run` does not skip, and `grep -c 'file="' bundle_upload/junit/0` against an all-passing bundle, where the count is 0 without the flag. Both resolution summaries print by default; only the per-test line needs `-v`. Say which is which, and say that an unresolved test leaves a blank file rather than failing the upload, since that is how a mismatched checkout presents. The tuning knobs are flags as well as environment variables, and the flag is the form someone reaches for while trialling it by hand — list both. Same for the `=` that `require_equals` makes mandatory when passing a value. Correct how `sourcekit-lsp` is found: `xcrun` on macOS, `PATH` on Linux, where the Swift toolchain provides it and there is no `xcrun` at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rement swiftlang/swift-package-manager#4752 was already linked from the sentence above, but the warning is the part someone lands on from a search or an anchor, and "the XCTest file is not written at all" reads like a Trunk limitation without the citation. Narrow the older sentence while here. It claimed `--parallel` was needed "for the XML report to output properly", which predates knowing a run writes two files: the swift-testing one appears either way, and it is the XCTest half that goes missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
trunk-io/analytics-cli#1201 gives the argument the expansion `--junit-paths` has always had, so "files, not globs" stops being true in the release these pages already require. One pattern reaches both files a run writes, which is a better example than naming them, so the Swift Testing and GitHub Actions snippets now glob. Note what the dedupe does while here: overlapping patterns cannot upload a test twice, which is the question a glob raises and a file list does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`swift test --xunit-output` writes JUnit XML, so listing it alongside XML as a fourth format contradicts itself. Four sources is what the list means and what the tabs below it show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XCResult and `swift test --xunit-output` are two ways into the same language, so counting them separately makes Swift look like half the list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"XML" alone says nothing about which schema, and the sentence goes on to mention a second XML that is also JUnit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches the sentence above it, in both the upload and test tab strips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream added the input to every upload example on these pages while this branch was open; the two added here were written before that and were the only ones without it. Left out of the `--dry-run --disable-quarantining` trial, which uploads nothing, so a collection to upload into would be noise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4b93927 to
1e1957a
Compare
Documents
--swift-test-xunit-paths(trunk-io/analytics-cli#1181, trunk-io/analytics-uploader#144) and--use-experimental-xcresult-test-locations(trunk-io/analytics-cli#1180), across four pages.Hold this until 0.15.5 is out of beta. Both flags exist only in
0.15.5-beta.1/beta.2today;lateststill resolves to 0.15.4, so the GitHub Actions snippet here fails with an unknown-argument error until the stable release lands. The pages state "requires 0.15.5 or newer" rather than pinning a beta version.Swift Testing page
The page said the report lands at
./junit.xml. It does not, for a Swift Testing suite: oneswift test --xunit-output junit.xml --parallelrun writes two files —junit-swift-testing.xmlfor@Test/@Suiteandjunit.xmlfor XCTest — and without--parallelthe XCTest file is never written even though those tests ran. Anyone following the old page uploaded the wrong file, or half a run without being told.The upload example now uses
--swift-test-xunit-paths, with a section on why it is a separate argument from--junit-paths: that XML carries no file path for any test case, so Trunk resolves each test to its declaring file by queryingsourcekit-lspover the checkout. That is what CODEOWNERS attribution and file-level links need. Requirements (sourcekit-lsponPATH, running inside the checkout) are listed.A
<Warning>covers the history impact, which seemed worse to leave for a customer to discover:filefeeds the test id, so a repository moving off--junit-pathschanges every affected id once, and the same test uploaded as.xcresultand as xunit XML is two distinct tests.XCTest page
Adds
--use-experimental-xcresult-test-locationsas an opt-in beta. It ishide = truein clap, so this is a deliberate call to surface it — flagging that here for review. The case for documenting it: it is the only way a passing xcresult test gets a file at all, and an Objective-C heavy repository will quietly fail to resolve everything untilTRUNK_XCRESULT_TEST_LOCATIONS_BUDGET_SECSis raised, which nobody can find without docs. The Quick /+testInvocationslimitation and the "unset rather than set tofalse" env-var wart are both noted, along with a tuning table carrying the real defaults fromLimits.Also adds a pointer for SwiftPM users, who have no
.xcresultto upload, toward the Swift Testing page.CLI reference and GitHub Actions
Swift tabs in the
uploadandtestformat tab groups, argument-table rows for both flags, and aswift-test-xunit-pathsexample in the ActionsCodeGroup.Testing
Read the flag definitions,
handle_swift_test_xunit,test_locations.rsand theswift-test-xunitfixture onorigin/mainof both repos; every default and behavior claim here traces to that code. Prose only — no nav ordocs.jsonchanges.mintlify broken-linkscould not run locally (unsupported on Node 25), so the four internal links were checked by hand against the file tree and heading anchors.🤖 Generated with Claude Code