From fb55ff35911f825049b74626de516449a1d434ab Mon Sep 17 00:00:00 2001 From: Bartek Plotka Date: Tue, 3 Nov 2026 20:36:51 +0000 Subject: [PATCH] docs: rename project to scrape.rs and move expfmt spec to specs/expfmt/SPEC.md Signed-off-by: Bartek Plotka --- README.md | 57 +++++++++++++++++++++++-- SPEC.md => specs/expfmt/SPEC.md | 10 ++--- specs/{ => expfmt}/openmetrics_1_0.abnf | 0 3 files changed, 59 insertions(+), 8 deletions(-) rename SPEC.md => specs/expfmt/SPEC.md (97%) rename specs/{ => expfmt}/openmetrics_1_0.abnf (100%) diff --git a/README.md b/README.md index 9815d6c..021bce1 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,58 @@ -# expfmt-rs +# scrape.rs > **Status: Experimental (Work in Progress)** -Prometheus and OpenMetrics exposition format decoders in Rust. +Official Rust libraries for Prometheus and OpenMetrics metric collection pipelines: wire exposition format decoding (`expfmt`), target and metric relabeling, standard service discoveries, and scrape management. -See [SPEC.md](./SPEC.md) for the architecture, supported formats, and implementation plan. +## Overview + +`scrape.rs` provides the building blocks required to discover targets, relabel labels, scrape HTTP endpoints, and decode Prometheus and OpenMetrics payloads in Rust: + +```text +Service discoveries + -> Target relabeling + -> Scrape manager / HTTP fetch + -> Streaming expfmt parser + -> Metric relabeling / drop hook + -> Columnar Apache Arrow (OTAP) batches or Protobuf DTOs +``` + +The project serves two main use cases: + +1. Columnar scraping for OpenTelemetry Arrow. [`open-telemetry/otel-arrow#3277`](https://github.com/open-telemetry/otel-arrow/issues/3277) defines an OTAP-native `prometheus_scrape` receiver in Rust. `scrape.rs` streams scrape bytes through pre-allocation relabeling and metric family assembly directly into Apache Arrow (`OtapArrowRecords`) tables on thread-per-core `!Send` runtimes with bounded memory. +2. Standalone Rust Prometheus tooling. Validation harnesses such as OpenTelemetry Weaver live-checking `/metrics` endpoints against semantic conventions, CLIs, and custom collectors can use individual components (such as `expfmt` DTO decoding or `relabel`) without pulling in the full scrape runtime. + +## Components and scope + +| Component | Status | Specification | Scope | +| :--- | :--- | :--- | :--- | +| `expfmt` | In progress | [specs/expfmt/SPEC.md](./specs/expfmt/SPEC.md) | `Content-Type` and `Accept` negotiation, zero-copy streaming parsers for Prometheus Text 0.0.4, OpenMetrics 1.0, OpenMetrics 2.0, and Prometheus Protobuf, primary Apache Arrow / OTAP decoder (`ArrowDecoder`), and [`MetricFamily`](https://github.com/prometheus/prometheus/blob/main/prompb/io/prometheus/client/metrics.proto) protobuf DTO decoder. | +| `relabel` | Planned | TBD | Prometheus-compatible target and metric relabeling (`replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`, `lowercase`, `uppercase`, `keepequal`, `dropequal`), including pre-allocation filtering on borrowed parser label buffers. | +| `discovery` | Planned | TBD | Standard Prometheus service discovery mechanisms (static targets, file-based SD, HTTP SD, DNS SD, Kubernetes SD) emitting target groups and `__meta_*` labels. | +| `scrape` | Planned | TBD | Bounded HTTP scrape scheduling, target pool lifecycle, compression and content negotiation, stale marker handling, target state reporting, and self-telemetry. | + +## Development and testing + +Run the unit and fixture test suite: + +```sh +cargo test --all-features +``` + +Run the [`otel-arrow-dfe-pdata`](https://github.com/open-telemetry/otel-arrow/tree/main/rust/otap-dataflow/crates/pdata) compatibility test suite: + +```sh +cargo test --manifest-path tests/otap-compat/Cargo.toml +``` + +Run the Go differential fixture extractor and tests (requires Go): + +```sh +cd tests/go-harness && go test ./... +``` + +Run the Criterion decoding benchmarks: + +```sh +cargo bench --bench decode +``` diff --git a/SPEC.md b/specs/expfmt/SPEC.md similarity index 97% rename from SPEC.md rename to specs/expfmt/SPEC.md index 8eff985..e9d8c86 100644 --- a/SPEC.md +++ b/specs/expfmt/SPEC.md @@ -1,6 +1,6 @@ -# `expfmt-rs` specification and implementation plan +# `expfmt` specification and implementation plan -`expfmt-rs` provides official Rust decoders for Prometheus and OpenMetrics wire exposition formats. +The `expfmt` crate in `scrape.rs` provides official Rust decoders for Prometheus and OpenMetrics wire exposition formats. ## Background and goals @@ -25,7 +25,7 @@ The `format` module mirrors [`prometheus/common/expfmt/expfmt.go`](https://githu ### ABNF-driven OpenMetrics parsing OpenMetrics 1.0 defines a normative [ABNF grammar](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf) that takes precedence over the prose specification. -- We will store machine-readable ABNF grammar files in the repository under `specs/openmetrics_1_0.abnf` and `specs/openmetrics_2_0.abnf`. +- We will store machine-readable ABNF grammar files in the repository under `specs/expfmt/openmetrics_1_0.abnf` and `specs/expfmt/openmetrics_2_0.abnf`. - We will either generate the OpenMetrics parser state tables directly from the ABNF or validate a hand-written zero-copy lexer against an ABNF reference parser in tests and fuzzers. ## Decoding architecture and APIs @@ -171,14 +171,14 @@ To verify that `ArrowDecoder` meets the [`open-telemetry/otel-arrow#3277`](https - Add the `go generate` fixture sync tool in `tests/go-harness` and generate initial `testdata/` fixtures from [`prometheus/prometheus/model/textparse`](https://github.com/prometheus/prometheus/tree/main/model/textparse) and [`prometheus/common/expfmt`](https://github.com/prometheus/common/tree/main/expfmt), verified in CI with `git diff --exit-code`. ### Phase 2: Prometheus Text 0.0.4 and OpenMetrics 1.0 across Layer 0, Layer 1, and Layer 2 -- Add `specs/openmetrics_1_0.abnf` from the [OpenMetrics 1.0 specification](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf). +- Add `specs/expfmt/openmetrics_1_0.abnf` from the [OpenMetrics 1.0 specification](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf). - Implement Layer 0 `Parser` for Prometheus Text 0.0.4 and OpenMetrics 1.0, including UTF-8 quoting and unescaping, exemplars, `_created` timestamps, `# UNIT`, and `# EOF` validation. - Implement Layer 1 `arrow::ArrowDecoder` targeting [`open-telemetry/otel-arrow#3277`](https://github.com/open-telemetry/otel-arrow/issues/3277), assembling classic histograms, summaries, counters, gauges, untyped, info, and stateset metrics into OTAP Arrow `RecordBatch` tables. - Implement Layer 2 `dto::decode_to_metric_families` and `dto::arrow_to_metric_families` on top of `ArrowDecoder`. - Wire up the Go `prometheus/common/expfmt` encoder harness and [`otel-arrow-dfe-pdata`](https://github.com/open-telemetry/otel-arrow/blob/main/rust/otap-dataflow/crates/pdata/Cargo.toml) validation in CI for round-trip differential and OTAP schema testing across Prometheus Text and OpenMetrics 1.0. ### Phase 3: OpenMetrics 2.0 and Prometheus protobuf decoding -- Implement Layer 0, Layer 1, and Layer 2 support for OpenMetrics 2.0 matching [`openmetrics2parse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetrics2parse.go) (native histograms in text format, `st@` start timestamps, composite values) and add `specs/openmetrics_2_0.abnf`. +- Implement Layer 0, Layer 1, and Layer 2 support for OpenMetrics 2.0 matching [`openmetrics2parse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetrics2parse.go) (native histograms in text format, `st@` start timestamps, composite values) and add `specs/expfmt/openmetrics_2_0.abnf`. - Implement delimited and text protobuf decoding (`FmtProtoDelim`, `FmtProtoText`, `FmtProtoCompact`), including native sparse histograms and NHCB conversion matching [`protobufparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/protobufparse.go) and [`nhcbparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/nhcbparse.go). - Extend the Go `expfmt` encoder harness to cover `FmtOpenMetrics_2_0_0` and `FmtProtoDelim`. diff --git a/specs/openmetrics_1_0.abnf b/specs/expfmt/openmetrics_1_0.abnf similarity index 100% rename from specs/openmetrics_1_0.abnf rename to specs/expfmt/openmetrics_1_0.abnf