Repository navigation
Conversation
Add otel-arrow-dfe-pdata-views 0.54 under the arrow feature and track contiguous data-point, attribute, and exemplar index ranges in MetricTable, AttrTable, and ExemplarTable so view iterators can slice rows in O(1) without scanning parent_id columns. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…oder Add DecodedMetricsView and associated zero-copy view types implementing the otel-arrow-dfe-pdata-views hierarchy over ArrowDecoder's staged metric, data-point, exemplar, and interned attribute buffers. Expose ArrowDecoder::decode_view, ArrowDecoder::view, and DecodeStream::next_view so OTAP encoders can traverse staged scrapes without materializing intermediate Arrow RecordBatches. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
marked this pull request as ready for review
November 5, 2026 05:13
…ricsView Verify in tests/otap-compat that passing DecodedMetricsView (from ArrowDecoder::decode_view and DecodeStream::next_view) to otel_arrow_dfe_pdata::encode::encode_metrics_otap_batch produces transport-optimizable OtapArrowRecords::Metrics batches and equivalent OTLP output across all metric types and upstream fixtures. Migrate benches/decode.rs to benchfmt.rs (benchfmt::Suite and CountingAlloc) and add tests/otap-compat/benches/otap_encode.rs comparing direct ArrowDecoder::decode column builders, ArrowDecoder::decode_view, and encode_metrics_otap_batch(&view) in Go benchfmt format for benchstat. Fixes #28 Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
force-pushed
the
bwplotka/phase4-pdata-views
branch
from
November 5, 2026 05:44
db4a814 to
022bf8f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on prometheus/scrape.rs#19.
Applies the
benchfmt.rsand rawbenchstatreporting rules from prometheus/scrape.rs#32.Fixes prometheus/scrape.rs#28.
Implements the zero-copy
otel-arrow-dfe-pdata-viewstrait hierarchy,MetricsViewand its child views, overArrowDecoder's staged columnar buffers and string interners so downstream OTAP consumers can passDecodedMetricsViewdirectly intootel_arrow_dfe_pdata::encode::encode_metrics_otap_batchor inspect decoded scrapes without materializingRecordBatches.Commits in order:
945b9b2addsotel-arrow-dfe-pdata-views = "0.54"under thearrowfeature, MSRV 1.85 with zero mandatory transitive dependencies, and records contiguous data-point, attribute, and exemplar index ranges inMetricTable,AttrTable, andExemplarTable.aa9a1b9addsDecodedMetricsViewand its child view and iterator types insrc/arrow/views.rsplusArrowDecoder::decode_view,ArrowDecoder::view, andDecodeStream::next_view.022bf8fverifies OTLP equivalence betweenArrowDecoder::decodeandencode_metrics_otap_batch(&view)across all metric types, translation policies, upstream fixtures, and streaming chunks, migratesbenches/decode.rstobenchfmt.rs, and addstests/otap-compat/benches/otap_encode.rs.benches/decode.rsbenchstatcomparison betweenbefore.txtat54d4ff7andafter.txtat022bf8f:Heap bytes and allocation counts on all existing
decodeanddecode_streampaths are unchanged, while the small+2.26%to+3.71%shifts on two rows reflect recording contiguous(start, end)row ranges onMetricTable,AttrTable, andExemplarTableplus shared-VM variance (± 12%to± 23%).tests/otap-compat/benches/otap_encode.rsbenchstat -col /modecomparison on022bf8f:Direct
ArrowDecoder::decoderemains ~1.5x to 2.3x faster and allocates ~10x to 40x fewer bytes thanencode_metrics_otap_batch(&view)becauseArrowDecoder::decodebuilds dictionary-encoded Arrow columns directly from its arena interners without re-hashing string keys and values insideotel-arrow-dfe-pdata's genericAdaptiveArrayBuilders, whiledecode_viewavoidsRecordBatchmaterialization altogether andDecodedMetricsViewprovides drop-in compatibility with anyMetricsViewconsumer.