docs: rename project to scrape.rs and move expfmt spec to specs/expfmt/SPEC.md

README.md

@@ -1,7 +1,58 @@
-# expfmt-rs
+# scrape.rs
 
 > **Status: Experimental (Work in Progress)**
 
-Prometheus and OpenMetrics exposition format decoders in Rust.
+Official Rust libraries for Prometheus and OpenMetrics metric collection pipelines: wire exposition format decoding (`expfmt`), target and metric relabeling, standard service discoveries, and scrape management.
 
-See [SPEC.md](./SPEC.md) for the architecture, supported formats, and implementation plan.
+## Overview
+
+`scrape.rs` provides the building blocks required to discover targets, relabel labels, scrape HTTP endpoints, and decode Prometheus and OpenMetrics payloads in Rust:
+
+```text
+Service discoveries
+  -> Target relabeling
+  -> Scrape manager / HTTP fetch
+  -> Streaming expfmt parser
+  -> Metric relabeling / drop hook
+  -> Columnar Apache Arrow (OTAP) batches or Protobuf DTOs
+```
+
+The project serves two main use cases:
+
+1. Columnar scraping for OpenTelemetry Arrow. [`open-telemetry/otel-arrow#3277`](https://github.com/open-telemetry/otel-arrow/issues/3277) defines an OTAP-native `prometheus_scrape` receiver in Rust. `scrape.rs` streams scrape bytes through pre-allocation relabeling and metric family assembly directly into Apache Arrow (`OtapArrowRecords`) tables on thread-per-core `!Send` runtimes with bounded memory.
+2. Standalone Rust Prometheus tooling. Validation harnesses such as OpenTelemetry Weaver live-checking `/metrics` endpoints against semantic conventions, CLIs, and custom collectors can use individual components (such as `expfmt` DTO decoding or `relabel`) without pulling in the full scrape runtime.
+
+## Components and scope
+
+| Component | Status | Specification | Scope |
+| :--- | :--- | :--- | :--- |
+| `expfmt` | In progress | [specs/expfmt/SPEC.md](./specs/expfmt/SPEC.md) | `Content-Type` and `Accept` negotiation, zero-copy streaming parsers for Prometheus Text 0.0.4, OpenMetrics 1.0, OpenMetrics 2.0, and Prometheus Protobuf, primary Apache Arrow / OTAP decoder (`ArrowDecoder`), and [`MetricFamily`](https://github.com/prometheus/prometheus/blob/main/prompb/io/prometheus/client/metrics.proto) protobuf DTO decoder. |
+| `relabel` | Planned | TBD | Prometheus-compatible target and metric relabeling (`replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`, `lowercase`, `uppercase`, `keepequal`, `dropequal`), including pre-allocation filtering on borrowed parser label buffers. |
+| `discovery` | Planned | TBD | Standard Prometheus service discovery mechanisms (static targets, file-based SD, HTTP SD, DNS SD, Kubernetes SD) emitting target groups and `__meta_*` labels. |
+| `scrape` | Planned | TBD | Bounded HTTP scrape scheduling, target pool lifecycle, compression and content negotiation, stale marker handling, target state reporting, and self-telemetry. |
+
+## Development and testing
+
+Run the unit and fixture test suite:
+
+```sh
+cargo test --all-features
+```
+
+Run the [`otel-arrow-dfe-pdata`](https://github.com/open-telemetry/otel-arrow/tree/main/rust/otap-dataflow/crates/pdata) compatibility test suite:
+
+```sh
+cargo test --manifest-path tests/otap-compat/Cargo.toml
+```
+
+Run the Go differential fixture extractor and tests (requires Go):
+
+```sh
+cd tests/go-harness && go test ./...
+```
+
+Run the Criterion decoding benchmarks:
+
+```sh
+cargo bench --bench decode
+```

SPEC.md → specs/expfmt/SPEC.md

@@ -1,6 +1,6 @@
-# `expfmt-rs` specification and implementation plan
+# `expfmt` specification and implementation plan
 
-`expfmt-rs` provides official Rust decoders for Prometheus and OpenMetrics wire exposition formats.
+The `expfmt` crate in `scrape.rs` provides official Rust decoders for Prometheus and OpenMetrics wire exposition formats.
 
 ## Background and goals
 
@@ -25,7 +25,7 @@ The `format` module mirrors [`prometheus/common/expfmt/expfmt.go`](https://githu
 ### ABNF-driven OpenMetrics parsing
 
 OpenMetrics 1.0 defines a normative [ABNF grammar](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf) that takes precedence over the prose specification.
-- We will store machine-readable ABNF grammar files in the repository under `specs/openmetrics_1_0.abnf` and `specs/openmetrics_2_0.abnf`.
+- We will store machine-readable ABNF grammar files in the repository under `specs/expfmt/openmetrics_1_0.abnf` and `specs/expfmt/openmetrics_2_0.abnf`.
 - We will either generate the OpenMetrics parser state tables directly from the ABNF or validate a hand-written zero-copy lexer against an ABNF reference parser in tests and fuzzers.
 
 ## Decoding architecture and APIs
@@ -171,14 +171,14 @@ To verify that `ArrowDecoder` meets the [`open-telemetry/otel-arrow#3277`](https
 - Add the `go generate` fixture sync tool in `tests/go-harness` and generate initial `testdata/` fixtures from [`prometheus/prometheus/model/textparse`](https://github.com/prometheus/prometheus/tree/main/model/textparse) and [`prometheus/common/expfmt`](https://github.com/prometheus/common/tree/main/expfmt), verified in CI with `git diff --exit-code`.
 
 ### Phase 2: Prometheus Text 0.0.4 and OpenMetrics 1.0 across Layer 0, Layer 1, and Layer 2
-- Add `specs/openmetrics_1_0.abnf` from the [OpenMetrics 1.0 specification](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf).
+- Add `specs/expfmt/openmetrics_1_0.abnf` from the [OpenMetrics 1.0 specification](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf).
 - Implement Layer 0 `Parser` for Prometheus Text 0.0.4 and OpenMetrics 1.0, including UTF-8 quoting and unescaping, exemplars, `_created` timestamps, `# UNIT`, and `# EOF` validation.
 - Implement Layer 1 `arrow::ArrowDecoder` targeting [`open-telemetry/otel-arrow#3277`](https://github.com/open-telemetry/otel-arrow/issues/3277), assembling classic histograms, summaries, counters, gauges, untyped, info, and stateset metrics into OTAP Arrow `RecordBatch` tables.
 - Implement Layer 2 `dto::decode_to_metric_families` and `dto::arrow_to_metric_families` on top of `ArrowDecoder`.
 - Wire up the Go `prometheus/common/expfmt` encoder harness and [`otel-arrow-dfe-pdata`](https://github.com/open-telemetry/otel-arrow/blob/main/rust/otap-dataflow/crates/pdata/Cargo.toml) validation in CI for round-trip differential and OTAP schema testing across Prometheus Text and OpenMetrics 1.0.
 
 ### Phase 3: OpenMetrics 2.0 and Prometheus protobuf decoding
-- Implement Layer 0, Layer 1, and Layer 2 support for OpenMetrics 2.0 matching [`openmetrics2parse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetrics2parse.go) (native histograms in text format, `st@` start timestamps, composite values) and add `specs/openmetrics_2_0.abnf`.
+- Implement Layer 0, Layer 1, and Layer 2 support for OpenMetrics 2.0 matching [`openmetrics2parse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/openmetrics2parse.go) (native histograms in text format, `st@` start timestamps, composite values) and add `specs/expfmt/openmetrics_2_0.abnf`.
 - Implement delimited and text protobuf decoding (`FmtProtoDelim`, `FmtProtoText`, `FmtProtoCompact`), including native sparse histograms and NHCB conversion matching [`protobufparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/protobufparse.go) and [`nhcbparse.go`](https://github.com/prometheus/prometheus/blob/main/model/textparse/nhcbparse.go).
 - Extend the Go `expfmt` encoder harness to cover `FmtOpenMetrics_2_0_0` and `FmtProtoDelim`.