Repository navigation
tests/functional: show sanitizer reports when a test fails - #81
Closed
joshheinrichs-shopify wants to merge 3 commits into
Closed
joshheinrichs-shopify wants to merge 3 commits into
joshheinrichs-shopify wants to merge 3 commits into
Conversation
The functional tests run with `log_path` in ASAN_OPTIONS so that sanitizer output doesn't break tests that compare stderr. The flip side is that when a sanitizer aborts a process, the test log shows nothing but an "Aborted" from bash: the report is in a file in the build directory that CI never sees. Dump any such reports from `onError`.
The sanitizer runtimes print raw addresses unless llvm-symbolizer is on PATH, which makes a report like the heap-use-after-free that aborts `nix serve` in the binary-cache test nearly useless. Put llvm-symbolizer in the environment of every instrumented component.
joshheinrichs-shopify
force-pushed
the
ci-dump-sanitizer-logs
branch
from
October 10, 2026 02:56
54cfe19 to
5d2b1f7
Compare
`GCTrackedStackAllocator` stored a pointer to the owning
`SinkToSource`/`SourceToSink`'s `stackCookie` and dereferenced it when
the coroutine's stack was deallocated, on the assumption that the owner
outlives the coroutine. With nested coroutines that assumption fails:
a decompressor's `sourceToSink` coroutine feeds a `sinkToSource` sink,
so the outer coroutine yields from the inner coroutine's stack. When the
outer `SinkToSource` is destroyed while suspended, boost unwinds the
stack it yielded from -- the inner one -- and that unwinding destroys
the outer stack's frames, among them the `unique_ptr` owning the inner
`SourceToSink`. The inner coroutine's own stack is only deallocated
afterwards, when its fiber exits, and its allocator then reads and
writes the cookie through a pointer into the freed `SourceToSink`:
ERROR: AddressSanitizer: heap-use-after-free
#0 GCTrackedStackAllocator::deallocate(...) serialise.cc:410
...
#12 sinkToSource(...)::SinkToSource::~SinkToSource()
#16 CmdServe::handleRequest(...)::State::~State
#19 MHD_destroy_response
`nix serve` hits this whenever a NAR download is cut short, which is
exactly what tests/functional/binary-cache.sh does to test resuming, so
the sanitizer legs of CI failed there with a silent abort. Without ASan
it is a use-after-free that happens to go unnoticed.
Keep the cookie in a cell shared between the owner and the allocator
(whose copy in the fiber record lives exactly as long as the stack), so
either can go first.
Contributor
Author
|
Superseded: the fix landed on main as 499125b by itself; the two CI-only commits here were debugging aids. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
main's sanitizer and coverage legs fail inbinary-cachewithnix servebeingAbortedduring the truncated-NAR resume test — silently, because ASan writes its report tolog_pathand nothing shows it. This makesonErrordump those reports, so the sanitizer leg of this PR should tell us what the actual finding is.Context
One-line harness change, no effect on passing tests. The underlying
nix serveabort is new since the v3.23.1 sync and only reproduces under clang+ASan on Linux (macOS and the remote builder are unavailable to me right now), hence getting the report via CI.