Skip to content

docs(flaky-tests): document the Swift report sources added in CLI 0.15.5 - #333

Open
dfrankland wants to merge 9 commits into
mainfrom
docs/swift-test-xunit-and-xcresult-locations
Open

dfrankland wants to merge 9 commits into
mainfrom
docs/swift-test-xunit-and-xcresult-locations

Conversation

@dfrankland

Copy link
Copy Markdown
Member

Documents --swift-test-xunit-paths (trunk-io/analytics-cli#1181, trunk-io/analytics-uploader#144) and --use-experimental-xcresult-test-locations (trunk-io/analytics-cli#1180), across four pages.

Hold this until 0.15.5 is out of beta. Both flags exist only in 0.15.5-beta.1/beta.2 today; latest still resolves to 0.15.4, so the GitHub Actions snippet here fails with an unknown-argument error until the stable release lands. The pages state "requires 0.15.5 or newer" rather than pinning a beta version.

Swift Testing page

The page said the report lands at ./junit.xml. It does not, for a Swift Testing suite: one swift test --xunit-output junit.xml --parallel run writes two files — junit-swift-testing.xml for @Test/@Suite and junit.xml for XCTest — and without --parallel the XCTest file is never written even though those tests ran. Anyone following the old page uploaded the wrong file, or half a run without being told.

The upload example now uses --swift-test-xunit-paths, with a section on why it is a separate argument from --junit-paths: that XML carries no file path for any test case, so Trunk resolves each test to its declaring file by querying sourcekit-lsp over the checkout. That is what CODEOWNERS attribution and file-level links need. Requirements (sourcekit-lsp on PATH, running inside the checkout) are listed.

A <Warning> covers the history impact, which seemed worse to leave for a customer to discover: file feeds the test id, so a repository moving off --junit-paths changes every affected id once, and the same test uploaded as .xcresult and as xunit XML is two distinct tests.

XCTest page

Adds --use-experimental-xcresult-test-locations as an opt-in beta. It is hide = true in clap, so this is a deliberate call to surface it — flagging that here for review. The case for documenting it: it is the only way a passing xcresult test gets a file at all, and an Objective-C heavy repository will quietly fail to resolve everything until TRUNK_XCRESULT_TEST_LOCATIONS_BUDGET_SECS is raised, which nobody can find without docs. The Quick / +testInvocations limitation and the "unset rather than set to false" env-var wart are both noted, along with a tuning table carrying the real defaults from Limits.

Also adds a pointer for SwiftPM users, who have no .xcresult to upload, toward the Swift Testing page.

CLI reference and GitHub Actions

Swift tabs in the upload and test format tab groups, argument-table rows for both flags, and a swift-test-xunit-paths example in the Actions CodeGroup.

Testing

Read the flag definitions, handle_swift_test_xunit, test_locations.rs and the swift-test-xunit fixture on origin/main of both repos; every default and behavior claim here traces to that code. Prose only — no nav or docs.json changes. mintlify broken-links could not run locally (unsupported on Node 25), so the four internal links were checked by hand against the file tree and heading anchors.

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
trunk 🟢 Ready View Preview Sep 16, 2026, 7:21 PM

@dfrankland

Copy link
Copy Markdown
Member Author

Pushed b4b583f, folding in the customer-facing instructions from this thread. Three things it covered that the pages did not, plus one correction:

  • An offline trial of the xcresult flag. --dry-run with --disable-quarantining (the quarantine lookup being the one call --dry-run doesn't skip), then grep -c 'file="' bundle_upload/junit/0 — which is zero without the flag for an all-passing bundle, so the reader sees the difference directly.
  • Tuning knobs are flags too. The table now carries --xcresult-test-locations-* alongside the env vars, since the flag is what you reach for while trialling by hand. Same for the = that require_equals makes mandatory when passing a value.
  • What unresolved tests look like. A test that doesn't resolve is left blank rather than failing the upload, which is how a mismatched checkout presents. Split by log level while documenting it: the xcresult test files: N from a declaration, N with no declaration found summary is info! and prints by default, only the per-test is declared at line is debug! and needs -v.

One correction to the thread, worth knowing before these instructions go to a customer again: "we find sourcekit-lsp through xcrun" is macOS-only. Since trunk-io/analytics-cli#1179, find_program tries xcrun --find on macOS and falls back to PATH, and on Linux it's PATH only — which is what makes --swift-test-xunit-paths work on a Linux runner, where the Swift toolchain ships sourcekit-lsp and there is no xcrun. The Swift Testing page states it that way.

@dfrankland

Copy link
Copy Markdown
Member Author

Pushed bcb4e56: --swift-test-xunit-paths takes globs now, per trunk-io/analytics-cli#1201, so the "files, not globs" wording is gone from all three places it appeared.

The Swift Testing and GitHub Actions snippets glob rather than naming both files — junit*.xml reaches the pair a single run writes, which is a better example than spelling them out, and it stops the XCTest half from being quietly dropped by someone who copies the line and only edits one filename. The cli-reference tabs still list both files explicitly, since that is where the two-file behavior is worth showing.

Also noted what the dedupe does: a file matched by more than one pattern is uploaded once. That is the question a glob raises and a file list does not, and the answer is not guessable.

No new version note — globs ship in the same 0.15.5 these pages already require. That does assume analytics-cli#1201 lands before 0.15.5 goes stable; if it slips to 0.15.6, this needs a revisit.

Merge order for the set, all gated on that release: analytics-cli#1201 → analytics-uploader#147 → this.

dfrankland and others added 9 commits September 16, 2026 19:19
`swift test --xunit-output` writes two files and neither is named what was
asked for, which the Swift Testing page did not say: swift-testing results go
to `<name>-swift-testing.xml` and XCTest ones to `<name>`, the latter only
when `--parallel` is passed. A reader following the old page uploaded the
wrong file, or silently uploaded half a run.

Document `--swift-test-xunit-paths` as the way to upload them. The argument
exists because that XML carries no file path for any test case, so each test
is attributed to the file a language server says it is declared in — which is
what CODEOWNERS attribution and file links need, and what `--junit-paths`
does not do for these files. That attribution changes test ids once for
anyone moving off `--junit-paths`, since `file` feeds the id, so the page
warns rather than leaving a reset history to be discovered.

Document `--use-experimental-xcresult-test-locations` as an opt-in beta on
the XCTest page. It is hidden in `--help`, but it is the only way to give a
*passing* xcresult test a file at all, and an Objective-C repository needs
the `TRUNK_XCRESULT_TEST_LOCATIONS_*` budget raised to finish resolving —
neither of which is discoverable without docs.

Both flags ship in 0.15.5; the pages say so, since `latest` is still 0.15.4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the instructions drafted for a customer trying the flag, which
covered ground the pages did not.

A reader cannot tell whether the flag worked without being told what to
look at, so document the offline trial: `--dry-run` with
`--disable-quarantining`, because the quarantine lookup is the one call
`--dry-run` does not skip, and `grep -c 'file="' bundle_upload/junit/0`
against an all-passing bundle, where the count is 0 without the flag.

Both resolution summaries print by default; only the per-test line needs
`-v`. Say which is which, and say that an unresolved test leaves a blank
file rather than failing the upload, since that is how a mismatched
checkout presents.

The tuning knobs are flags as well as environment variables, and the flag
is the form someone reaches for while trialling it by hand — list both.
Same for the `=` that `require_equals` makes mandatory when passing a
value.

Correct how `sourcekit-lsp` is found: `xcrun` on macOS, `PATH` on Linux,
where the Swift toolchain provides it and there is no `xcrun` at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rement

swiftlang/swift-package-manager#4752 was already linked from the sentence
above, but the warning is the part someone lands on from a search or an
anchor, and "the XCTest file is not written at all" reads like a Trunk
limitation without the citation.

Narrow the older sentence while here. It claimed `--parallel` was needed
"for the XML report to output properly", which predates knowing a run
writes two files: the swift-testing one appears either way, and it is the
XCTest half that goes missing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
trunk-io/analytics-cli#1201 gives the argument the expansion `--junit-paths`
has always had, so "files, not globs" stops being true in the release these
pages already require.

One pattern reaches both files a run writes, which is a better example than
naming them, so the Swift Testing and GitHub Actions snippets now glob. Note
what the dedupe does while here: overlapping patterns cannot upload a test
twice, which is the question a glob raises and a file list does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`swift test --xunit-output` writes JUnit XML, so listing it alongside XML as
a fourth format contradicts itself. Four sources is what the list means and
what the tabs below it show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XCResult and `swift test --xunit-output` are two ways into the same
language, so counting them separately makes Swift look like half the list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"XML" alone says nothing about which schema, and the sentence goes on to
mention a second XML that is also JUnit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches the sentence above it, in both the upload and test tab strips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream added the input to every upload example on these pages while this
branch was open; the two added here were written before that and were the
only ones without it.

Left out of the `--dry-run --disable-quarantining` trial, which uploads
nothing, so a collection to upload into would be noise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant