Skip to content

Bug fix: Prevent exemplar drop when filtered attributes exceed 128-rune limit - #7883

Open
NesterovYehor wants to merge 12 commits into
open-telemetry:mainfrom
NesterovYehor:fix/prometheus-exporter-exemplar-limit
Open

NesterovYehor wants to merge 12 commits into
open-telemetry:mainfrom
NesterovYehor:fix/prometheus-exporter-exemplar-limit

Conversation

@NesterovYehor

@NesterovYehor NesterovYehor commented Feb 8, 2026 •

Copy link
Copy Markdown
Contributor

I’ve added a test case for #6718 to show what happens when filtered attributes are too long.

Currently, the test fails because the Prometheus client_golang has a 128-rune limit for exemplars. When we move filtered attributes into the exemplar and they exceed this limit, the whole thing gets dropped, including the trace_id and span_id. It even causes a parsing error in the output:

--- FAIL: TestExemplars (0.00s)
    --- FAIL: TestExemplars/exemplar_overflow_does_not_drop_exemplar (0.00s)
        exporter_test.go:1405:
            Error Trace:    /Users/yehornesterov/dev/Go/opentelemetry-go/exporters/prometheus/exporter_test.go:1405
            Error:          Expected value not to be nil.
            Test:           TestExemplars/exemplar_overflow_does_not_drop_exemplar
FAIL
FAIL    go.opentelemetry.io/otel/exporters/prometheus    0.373s

To fix this, I can add a check that ensures we always fit the trace_id and span_id first. Then, we only add the user's filtered attributes if there is enough space left in the 128-rune budget. If an attribute doesn't fit, we just skip it so we don't lose the Trace ID.

@dashpole

Copy link
Copy Markdown
Contributor

Thanks for adding the test. You will need to add the fix as well for us to merge the PR.

@NesterovYehor

Copy link
Copy Markdown
Contributor Author

okey, I started by adding the test to clearly capture the failure case and expected behavior. I’ll add the fix next so we can get this merged.

@dmathieu
dmathieu marked this pull request as draft February 11, 2026 08:12
@codecov

codecov Bot commented Feb 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.5%. Comparing base (7213d23) to head (0593174).
⚠️ Report is 37 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #7883     +/-   ##
=======================================
- Coverage   89.5%   89.5%   -0.1%     
=======================================
  Files        351     351             
  Lines      23191   23202     +11     
=======================================
+ Hits       20771   20780      +9     
- Misses      2420    2422      +2     
Files with missing lines Coverage Δ
exporters/prometheus/exporter.go 92.1% <100.0%> (-1.2%) ⬇️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@NesterovYehor NesterovYehor changed the title test: add reproduction for Prometheus exemplar 128-rune limit breach Bug fix: Prevent exemplar drop when filtered attributes exceed 128-rune limit Feb 12, 2026
@NesterovYehor
NesterovYehor marked this pull request as ready for review February 12, 2026 12:34
Comment thread CHANGELOG.md Resolved
@NesterovYehor
NesterovYehor marked this pull request as draft February 12, 2026 14:13
@NesterovYehor
NesterovYehor force-pushed the fix/prometheus-exporter-exemplar-limit branch from 92005e2 to cfec6ba Compare February 12, 2026 14:31
@NesterovYehor
NesterovYehor marked this pull request as ready for review February 12, 2026 14:43
Comment thread exporters/prometheus/exporter.go Resolved
Comment thread CHANGELOG.md Resolved
@dashpole

Copy link
Copy Markdown
Contributor

My only concern with this is that we (Prometheus) are planning to relax the exemplar limit in OM 2.0: prometheus/docs#2835. I'm not sure if we would be able to remove this code without causing problems for people still using OM 1.0.

@bwplotka do you think we should contribute this fix to the prometheus client instead so that it can have different exemplar behavior for OM 1.0 vs 2.0?

@NesterovYehor
NesterovYehor force-pushed the fix/prometheus-exporter-exemplar-limit branch from ab3eec2 to b2e1eef Compare February 14, 2026 12:07
@NesterovYehor

Copy link
Copy Markdown
Contributor Author

Do you want me to make any further updates, or is the direction to close it?

@dashpole

Copy link
Copy Markdown
Contributor

Reasons to have this logic in OTel-go:

  • We can intelligently decide how to truncate the exemplar (by keeping trace id and span id, and dropping the rest).

Reasons to have this logic in Prometheus client-golang:

  • They can truncate or not truncate based on the scrape protocol (truncate for OM 1.0, and not truncate for Prometheus proto or OM 2.0).

If client_golang is willing to treat trace_id or span_id as "special" keys, it would make sense to have truncation logic there. Otherwise, it is probably OK to have it here, and eventually make it configurable. It would stink a bit that it will probably have to be limited by default, even after OM 2.0. I'm opening an issue...

@dashpole

Copy link
Copy Markdown
Contributor

Just wait a bit if that is OK.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Waiting on the author · refreshed 2026-11-04 20:55 UTC

Resolve merge conflicts.

Respond to 3 review items (e.g. link a commit, explain why not, ask a follow-up):

  • Inline threads: 1, 2
  • Top-level threads: 3
Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
  • Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Comment thread exporters/prometheus/exporter.go Resolved
Comment thread exporters/prometheus/exporter.go Resolved
Comment thread exporters/prometheus/exporter_test.go Resolved
@NesterovYehor

Copy link
Copy Markdown
Contributor Author

@dashpole Just to clarify, based on our discussion in #1953, I understood we'd handle this on the Prometheus side instead. Should I still proceed with the requested changes here?

@opentelemetry-pr-dashboard

This comment has been minimized.

@dashpole

Copy link
Copy Markdown
Contributor

prometheus/client_golang#2177 raised prometheus.ExemplarMaxRunes to 2048 and truncates to 128 runes for OpenMetrics 1.0 scrapes, but client_golang still drops the entire exemplar if attributes exceed prometheus.ExemplarMaxRunes. If we use prometheus.ExemplarMaxRunes here, this fixes the issue today on v1.23.2 and still protects trace_id and span_id at the 2048-rune limit once we upgrade client_golang.

@NesterovYehor
NesterovYehor requested a review from ps-mir as a code owner October 27, 2026 09:31
@NesterovYehor
NesterovYehor marked this pull request as draft October 27, 2026 09:31
@NesterovYehor
NesterovYehor marked this pull request as ready for review October 27, 2026 14:25
}
traceID := hex.EncodeToString(exemplar.TraceID)
spanID := hex.EncodeToString(exemplar.SpanID)
if exemplarLabelRunes(labels, traceID, spanID) > prometheus.ExemplarMaxRunes {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that attributesToLabels enforces remainingSpace per attribute, this exemplarLabelRunes check (and exemplarLabelRunes below, added in #9031) is unreachable and global.Warn is no longer called when attributes are dropped. Should we remove exemplarLabelRunes and call global.Warn when attributesToLabels skips an attribute that exceeds remainingSpace?

Comment thread CHANGELOG.md

### Fixed

- Fix dropping entire exemplar when filtered attributes exceed the 128-rune limit.(#7883)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #9031 already added a changelog entry for this on line 72, could we update that entry (or combine them) to mention go.opentelemetry.io/otel/exporters/prometheus and note that filtered attributes fitting within the prometheus.ExemplarMaxRunes budget are retained?

@opentelemetry-pr-dashboard

Copy link
Copy Markdown

Hi @NesterovYehor — just a friendly reminder that this pull request is waiting on you. The dashboard status comment has the open items and is kept current.

  • Replying is enough to hand it off — answer, explain why no change is needed, or ask a follow-up. The dashboard routes it onward once nothing on the list is waiting on you.
  • To hand it back for any other reason, including the dashboard getting this wrong, comment /dashboard route:reviewers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants