Skip to content

fix(ci_visibility): fix coverage crash and data loss with copied/restored execution contexts - #20615

Draft
gnufede wants to merge 8 commits into
mainfrom
dd/fix/pytest-coverage-context-stack-20260929
Draft

gnufede wants to merge 8 commits into
mainfrom
dd/fix/pytest-coverage-context-stack-20260929

Conversation

@gnufede

@gnufede gnufede commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Since #19956, pytest tests that execute user code through asgiref's
async_to_sync / sync_to_async (Django async test support, sync_to_async
ORM calls) with CI Visibility coverage enabled abort the whole session:

INTERNALERROR>   File ".../ddtrace/internal/coverage/code.py", line 417, in __exit__
INTERNALERROR>     covered_lines_stack.pop() → IndexError: pop from empty list

A large suite dies at its first async test (Django: ~282 of 2035); both
line-level and file-level coverage modes are affected. #19956 stores
per-context coverage data in ContextVars holding plain list stacks, and
plain lists break the __enter__/__exit__ push/pop pairing in two
independent ways
when contexts are copied or restored — this branch fixes
both, one per commit.

Failure mode 1 — stacks shared by reference into copied contexts

copy_context() shares ContextVar objects by reference, so a collector
exited in a different context (threading wrappers, async bridges) pops the
wrong entry — or an empty list (IndexError).

Fix (88884502bb "guard coverage stack exits"): copy-on-write stack
per context on __enter__; __exit__ pops only if the stack top is the
collector's own entry; get_covered_lines() reads the collector's own
entry, so results are correct regardless of where it exits.

Failure mode 2 — value-based context restoration masks/forces stack swaps

asgiref's _restore_context restores context values by comparing them with
!=, and plain lists compare by value: the initial restore into the
asyncio task is masked (two empty stacks compare equal), so the test body's
coverage lands in the loop-thread wrapper's entry; the final restore then
swaps the wrapper's stack into the pytest context, whose pop double-pops the
emptied stack → IndexError → session aborted. Even when guarded away, the
data stays stranded: async tests silently report empty coverage and can
never be safely skipped under ITR — guarding mode 1 alone does not fix this.

Fix (a89ce3907f "compare coverage stacks by identity for context
restores"):
stacks are now _ContextStack objects comparing by
identity, so restores are never masked (coverage is attributed to the
right test), become no-ops when the context already holds that exact stack
(pops stay balanced), and genuine propagation across copies still works.

Regression tests

Four new tests, two per mode, in tests/coverage/test_coverage.py and
tests/coverage/test_coverage_asgiref_context_propagation.py. The key one
hand-rolls the value-based restore contract across threads with no new
dependencies
; the other drives the real asgiref flow (skipif-guarded).
Both fail on main with the exact production IndexError, and the asgiref
test additionally fails with empty coverage on the guarded-exits-only
fix, discriminating mode 2. The entire pre-existing coverage suite (121
tests) passes on unfixed main — nothing covered context propagation
across threads.


PR by Bits - View session in Datadog
Comment @DataDog to request changes

@datadog-official

Copy link
Copy Markdown
Contributor

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/coverage/code.py                                       @DataDog/ci-app-libraries
releasenotes/notes/fix-ci-visibility-copied-context-coverage-f8c173bd.yaml  @DataDog/apm-python
tests/coverage/test_coverage.py                                         @DataDog/ci-app-libraries
tests/coverage/test_coverage_asgiref_context_propagation.py             @DataDog/ci-app-libraries

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 201 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 201 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=132)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=130)
ddtrace.internal.opentelemetry.context -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=130)
ddtrace.llmobs._integrations.vllm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=130)
ddtrace.llmobs._utils -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=130)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@pr-commenter

pr-commenter Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-10-06 20:12:16

Comparing candidate commit d69c01c in PR branch dd/fix/pytest-coverage-context-stack-20260929 with baseline commit 8385955 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@datadog-official

datadog-official Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Tests

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d69c01c | Docs | View more details | Give us feedback!

@gnufede
gnufede force-pushed the dd/fix/pytest-coverage-context-stack-20260929 branch from cf30d48 to 8bf5751 Compare October 6, 2026 17:00
@gnufede gnufede changed the title fix(ci_visibility): prevent pytest coverage crash fix(ci_visibility): fix coverage crash and data loss with copied/restored execution contexts Oct 6, 2026
Comment thread releasenotes/notes/fix-ci-visibility-copied-context-coverage-f8c173bd.yaml Outdated
datadog-bits and others added 5 commits October 6, 2026 17:36
Co-authored-by: gnufede <412857+gnufede@users.noreply.github.com>
…estores

The guarded stack exits from the previous commit prevent the crash caused
by stack ownership moving between execution contexts, but a second
failure mode remains: value-based context restoration (e.g. asgiref's
``_restore_context``, used by Django's async test support via
``async_to_sync``) compares ContextVar values with ``!=``. Plain list
stacks compare by value, so two distinct stacks holding equal entries
compare equal and the restore is masked: the asyncio task keeps the
loop-thread wrapper's stack, the test body's coverage data is recorded
into the wrapper's entry, and the per-test collector reads its own (now
empty) entry. The final restore, running before the executor thread join,
then replaces the caller's stack with the wrapper's, stranding the data:
async tests silently report no coverage (and can never be selected for
skipping under ITR).

Make the per-context stacks compare by identity so value-based restores
propagate the correct stack: distinct stacks are never equal (restores
are no longer masked, so the task starts with the caller's stack and
coverage is attributed to the right test), and a context already holding
the exact stack object ignores further restores of it (no forced swap,
push/pop pairing stays balanced).

Verified against Django's test suite (pytest + CI Visibility coverage,
asgiref ``async_to_sync``/``sync_to_async``): both line-level and
file-level coverage now complete the full run (2035 tests) with no
internal errors, and async tests report non-empty coverage again.

Co-authored-by: Claude <noreply@anthropic.com>
The stacks hold different element types (defaultdict[str, CoverageLines]
for lines, set[str] for files), so parameterize the generic base with
Any to satisfy mypy's type-arg check without duplicating the class.
@gnufede
gnufede force-pushed the dd/fix/pytest-coverage-context-stack-20260929 branch from 8ac6dc7 to 7f2ce0a Compare October 6, 2026 17:38
@gnufede
gnufede marked this pull request as ready for review October 6, 2026 18:03
@gnufede
gnufede requested review from a team as code owners October 6, 2026 18:03
@gnufede
gnufede requested review from sabrenner and removed request for a team October 6, 2026 18:03
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T18:33:21.996581Z c084f7c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f2ce0a543

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/internal/coverage/code.py

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Async tasks created during module import retain a completed import collector, silently losing their later execution coverage from the active test. The regression affects both line- and file-level coverage.

Open Bits AI session

🤖 Bits Code Review · Commit 7f2ce0a · @DataDog review to ask questions

Comment thread ddtrace/internal/coverage/code.py
On Python 3.14+, sys.monitoring callbacks run in a snapshot context and
fall back to the thread-local coverage state when they cannot observe
ContextVar changes. When a collector entered in a copied context is
exited from a context whose stack top is a different collector, the
early return left _tls_coverage pointing at the completed collector, so
subsequent instrumented code in that thread was recorded into the stale
entry instead of the active collector.

Re-sync the fallback to the current stack top (or clear it) before
returning, mirroring what the pop path already does.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c084f7c30a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/internal/coverage/code.py Outdated
@gnufede
gnufede marked this pull request as draft October 6, 2026 19:28
… entries

A context copied while a collector was active (eg. a task scheduled by a
module imported inside a test) keeps a reference to a stack that still
contains that collector's entry after it completes. Coverage recorded in
such a context was written into the completed collector's entry — one
that after_import() had already consumed — instead of the enclosing
active collector, so the data was silently lost.

Mark each collector's entries as closed when it exits, and make the
context resolvers (and the Python 3.14+ TLS fallback) walk down the stack
to the nearest entry that is still open, attributing new coverage to the
closest active collector instead of an orphaned entry.
The coverage module is private (nothing under docs/ renders it), so
docstrings and comments should read as plain text per the repository
convention in AGENTS.md: drop the rST double-backtick literals and write
the names as-is.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants