parse: add protobuf parser and native histograms by nicolastakashi · Pull Request #25 · prometheus/scrape.rs

nicolastakashi · 2026-11-04T17:06:54Z

Closes #20

Adds ProtobufParser, a Layer 0 Parser for FmtProtoDelim ported from protobufparse.go at v0.315.1, with start timestamps, exemplars, and native histograms as Entry::Histogram. It needs the dto feature. The Go harness now marshals the createTestProtoBuf inputs and writes the NewProtobufParser output to testdata/protobufparse/.

The Parser trait now returns metadata, series, and histogram bytes borrowed from the parser, because the protobuf parser builds them. This breaks #19, which needs metric_type and unit in ActiveParser to return &[u8] and Entry::Histogram added to the Entry::Comment arm. We should agree on the merge order.

Differences from Go:

No enableTypeAndUnitLabels option. https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/protobufparse.go#L86-L89
The classic re-emit and the summary stop after _sum happen even if the caller never calls histogram() or series(), because both take &self. https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/protobufparse.go#L127-L131 https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/protobufparse.go#L203-L205
The span check runs on every native histogram, because compaction would index out of bounds without it. https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/protobufparse.go#L596-L614
Each family is decoded whole, so a malformed metric fails before earlier metrics in its family are returned. https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/protobufparse.go#L437-L450
A label with no name, or any string that is not UTF-8, is an error. prost checks every string and cannot tell an absent name from an empty one. https://github.com/prometheus/prometheus/blob/v0.315.1/prompb/io/prometheus/client/decoder.go#L245-L247 https://github.com/prometheus/prometheus/blob/v0.315.1/prompb/io/prometheus/client/decoder.go#L280

Tested: the AGENTS.md checks, the OTAP compat suite, and go generate && go test with no testdata/ diff. Negative controls on compaction, the classic re-emit, the span check, and the label sort each fail a test.

Performance against upstream BenchmarkParsePromProto on expfmt/protobuf: Rust takes 51.0µs and 1359 allocs/op, and Go takes 30.5µs and 497. Most of the gap is copying label strings, tracked in #26.

Not tested: fuzzing and real exporter input.

Add ProtobufParser, a Layer 0 Parser for FmtProtoDelim that ports model/textparse/protobufparse.go from prometheus v0.315.1, with start timestamps, exemplars, and native histograms as Entry::Histogram. The Go harness now marshals the createTestProtoBuf inputs into testdata/textparse/TestProtobufParse.pb and writes Go's entries for three parser option sets to testdata/protobufparse/. SPEC.md Phase 3 drops FmtProtoText, FmtProtoCompact, and NHCB conversion. Closes #20 Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

Keep comment() returning input slices, add DecodeError::InvalidProtobuf instead of unreachable!, name the histogram in span errors like Go, reuse the series bytes buffer, and update the SPEC trait block. Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

Format the quantile or le label once per series, share the span checks and bucket copy between float and int histograms, use one helper for the histogram lookup and prost::decode_length_delimiter, and share the AST lookup between the Go harness extractors. Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

update_series sorts one reused index buffer, and labels() reads it instead of rebuilding and sorting the pairs again. Measured with BenchmarkParsePromProto inputs: about 10% less time and 15% fewer allocations per parse. Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

The format module still negotiates them. Only the parser leaves them out, as the Phase 3 note says. Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

nicolastakashi · 2026-11-04T20:38:14Z

@copilot review

Copilot · 2026-11-04T20:41:16Z

@copilot review

Reviewed the last six commits, including protobuf parser state transitions, histogram handling, API integration, and Go fixture generation. No actionable correctness findings. All three protobuf-focused tests passed, and the latest CI run is green. No changes made.

bwplotka

Thanks! A few points, otherwise LGTM

Main proposal I would have is to always stick to NHCBs instead of classic histograms. This will be flaky and expensive for legacy protocols but seemless on OM2 and proto. WDYT?

cc @krajorama

bwplotka · 2026-11-04T21:41:37Z

+                        self.switch_to_classic();
+                        continue;
+                    }
+                    if !self.next_metric() {


When self.redo_classic is false, advancing to the next metric here does not reset field_pos or exemplar_pos like Entry::Series does, so draining exemplars on one native histogram will skip exemplars on the next native histogram in the same family.

Suggested change

if !self.next_metric() {

self.field_pos = -2;

self.exemplar_pos = 0;

if !self.next_metric() {

bwplotka · 2026-11-04T21:41:38Z

+
+    #[test]
+    fn rejects_malformed_input_without_panicking() {
+        let valid = delimited(&[histogram_family("h", vec![1, 0])]);


Since positive_delta is delta-encoded, vec![1, 0] gives bucket counts [1, 1], and none of the fixtures in TestProtobufParse.pb have zero-count buckets or adjacent offset == 0 spans either. Worth adding a small test with empty buckets like vec![1, -1, 2] and offset == 0 spans so the bucket-removal and span-splitting branches in compact_buckets are exercised.

bwplotka · 2026-11-05T02:53:44Z

 - Implement Layer 0, Layer 1, and Layer 2 support for OpenMetrics 2.0 matching [`openmetrics2parse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetrics2parse.go) (native histograms in text format, `st@` start timestamps, composite values) and add `specs/expfmt/openmetrics_2_0.abnf`.
- Implement delimited and text protobuf decoding (`FmtProtoDelim`, `FmtProtoText`, `FmtProtoCompact`), including native sparse histograms and NHCB conversion matching [`protobufparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/protobufparse.go) and [`nhcbparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/nhcbparse.go).
+- Implement delimited protobuf decoding (`FmtProtoDelim`), including native sparse histograms, start timestamps, and exemplars, matching [`protobufparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/protobufparse.go).
+- Out of scope: `FmtProtoText` and `FmtProtoCompact`, because Prometheus only scrapes `encoding=delimited` ([config.go#L584](https://github.com/prometheus/prometheus/blob/v0.315.1/config/config.go#L584)), and NHCB conversion, because it is opt-in in Prometheus ([interface.go#L209-L228](https://github.com/prometheus/prometheus/blob/v0.315.1/model/textparse/interface.go#L209-L228)).


Hmmmm - can we ALWAYS convert to NHCB?

hm - this is hard on legacy protocols, but maybe we should try?

nicolastakashi added 4 commits November 4, 2026 17:06

nicolastakashi mentioned this pull request Nov 4, 2026

Phase 4: borrow label strings in the protobuf parser #26

Open

nicolastakashi added 2 commits November 4, 2026 20:05

parse: say only OpenMetrics caches start timestamps

55c16a7

Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

spec: keep text and compact-text in the protobuf content types

dfdb39e

The format module still negotiates them. Only the parser leaves them out, as the Phase 3 note says. Signed-off-by: Nicolas Takashi <nicolas.takashi@dash0.com>

nicolastakashi marked this pull request as ready for review November 4, 2026 20:38

nicolastakashi requested a review from bwplotka November 4, 2026 20:38

Copilot started work on behalf of nicolastakashi November 4, 2026 20:38

View session

Copilot finished work on behalf of nicolastakashi November 4, 2026 20:41

bwplotka reviewed Nov 5, 2026

View reviewed changes

Navigation Menu

Repository navigation

parse: add protobuf parser and native histograms - #25

parse: add protobuf parser and native histograms#25
nicolastakashi wants to merge 6 commits into
mainfrom
nicolastakashi/phase3-protobuf-parser

nicolastakashi commented Nov 4, 2026 •

edited

Loading

Uh oh!

nicolastakashi commented Nov 4, 2026

Uh oh!

Copilot AI commented Nov 4, 2026

Uh oh!

bwplotka left a comment

Uh oh!

bwplotka Nov 4, 2026

Uh oh!

bwplotka Nov 4, 2026

Uh oh!

bwplotka Nov 5, 2026

Uh oh!

bwplotka Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nicolastakashi commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicolastakashi commented Nov 4, 2026

Uh oh!

Copilot AI commented Nov 4, 2026

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

bwplotka Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

bwplotka Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

bwplotka Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

bwplotka Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nicolastakashi commented Nov 4, 2026 •

edited

Loading