feat(p2p/sensor): ClickHouse backend + full/first-seen event model#961
Merged
Conversation
Add a ClickHouse backend for the p2p sensor as an alternative to GCP Datastore, selectable with --database=clickhouse --clickhouse-dsn. - Implements the Database interface with client-side batched, append-only writes (no read-before-write) via a generic drop-on-full rowBatcher, so the sensor hot path never blocks on the database. - Recovers and stores the block signer at ingest (util.Ecrecover), matching the sensor's block-signer validation and the data-analysis tools. - Adds an integration test (skipped unless POLYCLI_TEST_CLICKHOUSE_DSN is set) verifying writes and signer round-trip against a real server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist the remaining header fields (uncle hash, roots, logs bloom, extra data, mix digest, nonce) so downstream consumers (panoptichain, data-analysis) can reconstruct the header and re-run bor/clique ecrecover. Cheap on ClickHouse: the Datastore cost was index bloat, not these values. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # go.sum
…lpers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Handle the error returns from rows.Close and conn.Close to satisfy errcheck, and regenerate the p2p sensor docs for the ClickHouse flags. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
- Add Close() to the Database interface and implement graceful shutdown: ClickHouse now drains the batcher flushers (WaitGroup + internal cancel) before closing the connection, so buffered rows aren't lost on SIGINT; Datastore waits for in-flight async writes then closes the client; JSON and nodb are no-ops. The sensor defers db.Close(). - Retry failed batch inserts (immediate, fresh connection per attempt) before dropping, and wrap flush/NodeList errors with %w. - Set LZ4 compression + pool defaults on the ClickHouse connection when the DSN doesn't specify them. - Add blocks.is_parent so parent-vs-live analysis is queryable. - Drop transactions.first_seen (redundant with ingested_at); earliest first-seen is derived at query time from transaction_events. - Wire first-tx-event in both ClickHouse and Datastore backends (ShouldWriteFirstTransactionEvent was previously a silent no-op). The ClickHouse schema in sensor-network-tools#102 is reconciled to match (is_parent added, transactions keyed/partitioned/TTL'd on ingested_at). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first-tx-event write was wired into both backends, but handleNewPooledTransactionHashes still gated tx fetching on ShouldWriteTransactions/ShouldWriteTransactionEvents, so the --write-txs=false --write-tx-events=false --write-first-tx-event=true mode never fetched pooled-hash-announced txs. Add ShouldWriteFirstTransactionEvent to that gate. Also apply behavior-preserving simplifications: share the identical event-append closure between block_events/transaction_events, make newBlock a free function (no receiver use), hoist peer.URLv4() out of per-item loops, unify the WriteBlockHashFirstSeen guard, and use range-over-int in Datastore.Close. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Condense the multi-sentence explanatory comments added for the ClickHouse backend to concise notes, keeping the load-bearing rationale (drop-on-full batching, detached shutdown flush, retry, is_parent semantics, the WriteBlockBody gating divergence). Comments only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously write-{block,tx}-events and write-first-{block,tx}-event
produced identical output: the fetch-dedup cache gated the event write,
so both collapsed to one event per first-seen hash. Decouple event
recording from the fetch/broadcast dedup so the flags mean what they say:
- full (write-block-events / write-tx-events): one inbound event per
(peer, hash) — every peer we received the announcement from.
- first (write-first-*-event): one event per hash, earliest sighting.
Unify the write path: WriteBlockHashes -> WriteBlockEvents (pure append,
caller-gated) and add the symmetric WriteTransactionEvents; the sensor
picks the hash set via eventHashes(all, firstSeen, full, first).
WriteBlockHashFirstSeen is now the Datastore-only first-seen stamp
(ClickHouse/json/nodb no-op). tx first-seen is tracked at
announcement via a new Conns.announcedTxs LRU (the tx analog of the
blocks cache). The fetch/broadcast dedup is unchanged, so the efficiency
posture is unaffected; full per-peer streams remain opt-in.
Adds handler-level tests (recordingDB double) asserting full=per-peer,
first=once, neither=none for both blocks and txs, plus an eventHashes
unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
minhd-vu
changed the title
Rename the inner err in the deferred db.Close() (sensor.go) and the test's Close check to cerr so `shadow` (go vet) stops flagging them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
praetoriansentry
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a ClickHouse backend for
polycli p2p sensor(a cheaper alternative to GCP Datastore), and reworks the block/transaction event model so the full (per-peer) and first-seen flags are actually distinct.Part of the sensor Datastore→ClickHouse migration (Datastore ~$14–15k/mo, dominated by read-before-write dedup; ClickHouse is append-only + columnar). Pairs with sensor-network-tools#102 (schema/consumers/local stack) and polygon-infrastructure#1786 (infra).
ClickHouse backend (
p2p/database/clickhouse.go)rowBatcher(buffered channel + background flusher): the sensor hot path never blocks, and a slow/unavailable DB drops rows rather than exhausting memory. No read-before-write.util.Ecrecover), consistent with the data-analysis tools.Close()(added to theDatabaseinterface): drains the batchers before closing the connection, so buffered rows aren't lost on shutdown.%werror wrapping, LZ4 compression + connection-pool defaults.blocks.is_parentcolumn;transactions.first_seendropped (earliest first-seen is derived fromtransaction_events, mirroringblock_first_seen).Event model (both backends)
Previously
--write-{block,tx}-eventsand--write-first-{block,tx}-eventproduced identical output — the fetch-dedup cache collapsed both to one event per first-seen hash. Now:--write-*-events): one inbound event per (peer, hash) — every peer we received the announcement from.--write-first-*-event): one event per hash, earliest sighting.Unified to
WriteBlockEvents/WriteTransactionEvents(pure append, caller-gated via a sharedeventHasheshelper). Tx first-seen is tracked at announcement via a newConns.announcedTxsLRU (the tx analog of the blocks cache). The fetch/broadcast dedup is unchanged, so the efficiency posture is unaffected; per-peer streams are opt-in.Testing
make build,go vet,gofmt, andgo test ./p2p/...pass. New handler-level tests assert full=per-peer / first=once / neither=none for both blocks and txs, plus aneventHashesunit test. Verified end-to-end against Polygon mainnet via the local stack (sensor-network-tools#102): block data consistent with Datastore (0 missing / 0 mismatched), per-peer events (~7.5 peers/block, ~10 peers/tx), andClosedrains cleanly.Notes
none.🤖 Generated with Claude Code