refactor: abstract llmobs from anthropic contrib using the subscriber pattern by emmettbutler · Pull Request #20760 · DataDog/dd-trace-py

emmettbutler · 2026-10-02T17:05:06Z

This change reduces the coupling between contrib.anthropic and llmobs. Change summary:

Contrib side: the Anthropic integration builds an LlmRequestEvent without an llmobs_integration. That field is now optional. The contrib sets the anthropic.request.model tag itself, through the event's tags.
Tracing side: LlmTracingSubscriber sends out three new events, defined as LlmEvents:
- SPAN_STARTING, before the span is created. It's the only point where the span type can still be set to LLM.
- SPAN_STARTED, right after the span is created.
- SPAN_FINISHING, just before the span is finished.
LLMObs side: three subscribers in ddtrace/llmobs/_contrib/anthropic/ listen to those events. They ignore events from other integrations, and they call AnthropicIntegration, which they create on first use.
Registration: the subscribers stay registered even when LLMObs is disabled, so the APM shadow tags still appear. listen_integrations() in ddtrace/llmobs/_product.py hooks the anthropic.patch and anthropic.unpatch events. It runs from the product's post_preload (ddtrace-run) and from LLMObs.enable().
Other integrations: llama_index, the only other user of LlmRequestEvent, still passes llmobs_integration= and goes through the old path unchanged.

datadog-prod-us1-4 · 2026-10-02T17:06:13Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/apm-reliability/dd-trace-py | build macos amd64: [3.9 3.10 3.11] — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 3bcbf5b | Docs | View more details | Give us feedback!}

cit-pr-commenter-54b7da · 2026-10-02T17:06:44Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.claude/skills/apm-integrations/references/implementation-guide.md      @DataDog/python-guild
.claude/skills/llmobs-integrations/SKILL.md                             @DataDog/python-guild
.claude/skills/llmobs-integrations/references/failure-modes.md          @DataDog/python-guild
.claude/skills/llmobs-integrations/references/implementation-guide.md   @DataDog/python-guild
ddtrace/_trace/subscribers/llm.py                                       @DataDog/apm-sdk-capabilities-python @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/contrib/_events/llm.py                                          @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/contrib/internal/anthropic/_streaming.py                        @DataDog/ml-observability
ddtrace/contrib/internal/anthropic/patch.py                             @DataDog/ml-observability
ddtrace/llmobs/_contrib/__init__.py                                     @DataDog/ml-observability
ddtrace/llmobs/_contrib/anthropic/__init__.py                           @DataDog/ml-observability
ddtrace/llmobs/_contrib/anthropic/subscribers.py                        @DataDog/ml-observability
ddtrace/llmobs/_integrations/anthropic.py                               @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_product.py                                              @DataDog/ml-observability
tests/contrib/anthropic/conftest.py                                     @DataDog/ml-observability

cit-pr-commenter-54b7da · 2026-10-02T17:06:58Z

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

cit-pr-commenter-54b7da · 2026-10-02T17:07:43Z

Dependency direction analysis

📈 Existing violations got worse

5 pre-existing violation(s) increased in severity (e.g. their target became more depended-on, or got pulled into an import cycle), though the edge itself isn't new:

ddtrace.contrib.internal.botocore.services.bedrock -×-> ddtrace.llmobs._constants  (contrib -> product:llmobs, score=40, +1 vs base)
ddtrace.contrib.internal.litellm.patch -×-> ddtrace.llmobs._constants  (contrib -> product:llmobs, score=40, +1 vs base)
ddtrace.contrib.internal.openai._realtime -×-> ddtrace.llmobs._constants  (contrib -> product:llmobs, score=40, +1 vs base)
ddtrace.contrib.internal.openai._endpoint_hooks -×-> ddtrace.llmobs._constants  (contrib -> product:llmobs, score=40, +1 vs base)
ddtrace.contrib.internal.litellm.utils -×-> ddtrace.llmobs._constants  (contrib -> product:llmobs, score=40, +1 vs base)

⚠️ Existing dependency direction violations

There are 172 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 172 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=124)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=122)
ddtrace.llmobs._integrations.pydantic_ai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=122)
ddtrace.llmobs._integrations.langchain -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=122)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=122)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

✅ Dependency direction violations removed

2 violation(s) have been removed by this PR.

ddtrace.contrib.internal.anthropic._streaming -×-> ddtrace.llmobs._utils  (contrib -> product:llmobs, score=42)
ddtrace.contrib.internal.anthropic.patch -×-> ddtrace.llmobs._integrations  (contrib -> product:llmobs, score=15)

chatgpt-codex-connector

💡 Codex Review

dd-trace-py/ddtrace/contrib/internal/stream_handler.py

Line 266 in 542b999

# AIDEV-NOTE: next() without exhausting never hits StopIteration, so

Remove the newly added AIDEV anchor comments

This move adds deprecated AIDEV-NOTE: labels to a new file (and repeats one in TracedAsyncStream). The repository's CI checker inspects added diff lines rather than matching moved content, so these labels are reported as violations and block the commit; replace them with plain inline comments.

AGENTS.md reference: AGENTS.md:L151-L154

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pr-commenter · 2026-10-02T17:29:46Z

Benchmarks

Benchmark execution time: 2026-10-09 17:39:09

Comparing candidate commit 3bcbf5b in PR branch emmett.butler/llmobs-contrib with baseline commit db006ef in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 4 performance regressions! Performance is the same for 370 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-tracecontext_headers

🟥 execution_time [+2.576µs; +2.803µs] or [+8.391%; +9.132%]

scenario:msgpackencoderscenario-simple_one_span

🟥 execution_time [+506.817ns; +557.303ns] or [+13.168%; +14.479%]

scenario:otelspan-start

🟥 execution_time [+2.598ms; +3.443ms] or [+10.010%; +13.263%]

scenario:recursivecomputation-shallow

🟥 execution_time [+58.481µs; +61.585µs] or [+8.418%; +8.865%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-733.778ns; +740.810ns] or [-7.044%; +7.112%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-38.786ns; +39.411ns] or [-5.829%; +5.922%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-1961.523ns; +1864.369ns] or [-9.830%; +9.343%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1980.878ns; +1723.623ns] or [-10.311%; +8.971%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-367.408ns; +391.345ns] or [-8.642%; +9.205%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-216.494ns; +240.514ns] or [-8.089%; +8.986%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [-87.777ns; +98.858ns] or [-6.496%; +7.316%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-4631.770ns; +4596.524ns] or [-9.553%; +9.480%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-931.620ns; +950.687ns] or [-9.093%; +9.279%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.751µs; +2.861µs] or [+20.655%; +21.481%]

scenario:span-start

🟥 execution_time [+1.588ms; +1.875ms] or [+14.521%; +17.141%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+162.009ns; +199.401ns] or [+8.208%; +10.103%]

scenario:tracer-small

🟥 execution_time [+44.383µs; +45.256µs] or [+18.227%; +18.585%]

Known flaky benchmarks without significant changes:

scenario:errortrackingflasksqli-baseline
scenario:flasksimple-iast-get
scenario:sethttpmeta-all-enabled
scenario:telemetryaddmetric-record-100-metrics

base_stream_handler.py has no LLM Observability dependencies, but it lived under ddtrace/llmobs/_integrations, so every contrib that traces streamed responses (and AI Guard) imported from the LLMObs product. Move it to ddtrace/contrib/internal/stream_handler.py and update all importers. The unit tests are renamed to tests/llmobs/test_stream_handler.py, and the new path is added to the llmobs suitespec component so changes to it keep triggering the LLM integration suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

emmettbutler · 2026-10-06T20:49:51Z

Pulled out some unrelated test fixes to #20845

chatgpt-codex-connector · 2026-10-09T12:59:15Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-09T17:20:14.263493Z	`3bcbf5b`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-prod-us1-4

LlamaIndex requests bypass their legacy initialization, leaving model/provider metadata unset. Manual Anthropic instrumentation also loses shadow tags and token metrics when LLMObs is disabled.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 28a077b}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28a077b8a6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

fix in-flight requests after unpatch()

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd863312fa

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

emmettbutler requested review from a team as code owners October 2, 2026 17:05

emmettbutler requested review from rachelyangdog, wantsui and wconti27 and removed request for a team October 2, 2026 17:05

emmettbutler marked this pull request as draft October 2, 2026 17:05

emmettbutler requested review from avara1986 and brettlangdon and removed request for a team October 2, 2026 17:05

emmettbutler mentioned this pull request Oct 2, 2026

refactor: eliminate LLMObsIntegration knowledge from the anthropic contrib module #20496

Closed

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread ddtrace/contrib/_events/llm.py

emmettbutler and others added 2 commits October 2, 2026 10:46

abstract llmobs from anthropic contrib using the subscriber pattern

72cb2ae

emmettbutler force-pushed the emmett.butler/llmobs-contrib branch from 542b999 to 72cb2ae Compare October 2, 2026 18:15

emmettbutler changed the base branch from main to emmett.butler/move-stream-handler October 2, 2026 18:16

emmettbutler added 2 commits October 2, 2026 11:18

rm cassettes

b2f95ee

merge whoopsie

d991e24

emmettbutler changed the title ~~abstract llmobs from anthropic contrib using the subscriber pattern~~ refactor: abstract llmobs from anthropic contrib using the subscriber pattern Oct 2, 2026

emmettbutler added the changelog/no-changelog A changelog entry is not required for this PR. label Oct 2, 2026

import cycle

b288318

chrikrah mentioned this pull request Oct 4, 2026

[BUG]: Anthropic streamed spans report 1h cache writes as ephemeral_5m_input_tokens #20782

Open

wconti27 reviewed Oct 5, 2026

View reviewed changes

Comment thread ddtrace/llmobs/_contrib/anthropic/subscribers.py

emmettbutler requested a review from wconti27 October 5, 2026 15:01

fix test isolation problem exposed by refactor

8416503

quinna-h mentioned this pull request Oct 7, 2026

refactor(llmobs): abstract llmobs from llama_index integration code #20860

Open

Base automatically changed from emmett.butler/move-stream-handler to main October 9, 2026 12:45

emmettbutler added 2 commits October 9, 2026 05:46

Merge branch 'main' into emmett.butler/llmobs-contrib

6a1dcdc

Merge branch 'main' into emmett.butler/llmobs-contrib

28a077b

emmettbutler marked this pull request as ready for review October 9, 2026 12:59

datadog-prod-us1-4 Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread ddtrace/_trace/subscribers/llm.py

Comment thread ddtrace/llmobs/_contrib/__init__.py

emmettbutler marked this pull request as draft October 9, 2026 13:07

chatgpt-codex-connector Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread ddtrace/_trace/subscribers/llm.py Outdated

Comment thread ddtrace/llmobs/_contrib/__init__.py

emmettbutler and others added 4 commits October 9, 2026 06:11

typing

1397a45

update rules about direct patch() calls

5398902

fix in-flight requests after unpatch()

Merge branch 'main' into emmett.butler/llmobs-contrib

2842d96

Merge branch 'main' into emmett.butler/llmobs-contrib

26e1d6d

emmettbutler marked this pull request as ready for review October 9, 2026 15:00

Merge branch 'main' into emmett.butler/llmobs-contrib

bd86331

chatgpt-codex-connector Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/anthropic/_streaming.py

emmettbutler and others added 2 commits October 9, 2026 08:07

wrong merge

8a556b6

Merge branch 'main' into emmett.butler/llmobs-contrib

3bcbf5b

Conversation

emmettbutler commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Oct 2, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

📈 Existing violations got worse

⚠️ Existing dependency direction violations

✅ Dependency direction violations removed

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pr-commenter Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationextract-tracecontext_headers

scenario:msgpackencoderscenario-simple_one_span

scenario:otelspan-start

scenario:recursivecomputation-shallow

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

Known flaky benchmarks

scenario:httppropagationinject-ids_only

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Uh oh!

Uh oh!

emmettbutler commented Oct 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-prod-us1-4 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Reviewers

Assignees

emmettbutler commented Oct 2, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Oct 2, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Oct 2, 2026 •

edited

Loading

pr-commenter Bot commented Oct 2, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 9, 2026 •

edited

Loading