WIP: statistics, executor: collect singleton sketches for row sampling

[0xPoe]

What problem does this PR solve?

Issue Number: ref #67449

Problem Summary:

Row-sampling analyze needs singleton sketches to improve NDV estimation from distributed samples.

What changed and how does it work?

The pkg/statistics half of this work is split out into #68499. That commit stays at the base of this PR; the executor wiring is stacked on top.

• Bump tipb to include row-sample singleton sketch fields.
• Collect and serialize singleton sketches in row sampler.
• Use per-worker singleton sketches to estimate NDV during analyze stats building.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manual test:

• make bazel_prepare
• make lint

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Improve NDV estimation for row-sampling ANALYZE by collecting singleton sketches.

Summary by CodeRabbit

• New Features

  • Added FM-sketch based NDV estimation to ANALYZE and a singleton-sketch layer for tracking values seen once in samples.

• Improvements

  • Histograms may adopt higher sketch-derived NDV estimates (excluding special indexes) to improve distinct-value accuracy.
  • Sampling now preserves and merges sketch state and exposes an NDV sampling rate.

• Tests

  • New unit and sampling tests validate sketch NDV estimation and serialization/merge.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[tiprow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds singleton-sketch collection during row sampling, refactors FM-sketch hashing helpers, accumulates per-worker sketches, computes per-column NDV estimates (skipping special indexes), and applies those estimates to refine histogram NDVs; includes serialization/merge updates and tests.

Changes

Singleton Sketch Collection and NDV Estimation for Row Analysis

Layer / File(s)	Summary
Data Shape & Structure pkg/statistics/row_sampler.go	baseCollector gains SingletonSketches, singletonBuilders, and SketchSampleCount fields to track singleton-value hashes and sketch sample counts alongside FM sketches.
Singleton Sketch Builder Implementation pkg/statistics/row_sampler.go	Internal singletonSketchBuilder tracks hash values appearing once vs. multiple times via dynamically-expanded bitmaps, with methods to insert datum/row values and convert state to an FMSketch.
Collector Initialization & Constants pkg/statistics/row_sampler.go	NewReservoirRowSampleCollector and NewBernoulliRowSampleCollector allocate new singleton sketch slices; DefaultNDVSampleRate and RowSampleBuilder.NDVSampleRate control probabilistic sketch collection after warm-up.
Sketch Collection & Sampling pkg/statistics/row_sampler.go	RowSampleBuilder.Collect populates singleton builders, conditionally enables sketching based on sample count and rate, increments SketchSampleCount, and calls BuildSingletonSketches to materialize singleton sketch results.
Column & Group Collection with Sketch Integration pkg/statistics/row_sampler.go	collectColumns and collectColumnGroups now accept collectSketch flag and insert values into both FM sketches and singleton builders; BuildSingletonSketches derives group singleton sketches and copies single-column results where applicable.
Hashing Logic Refactoring pkg/statistics/fmsketch.go	Extracts hashing into standalone hashDatum and hashRow helpers that return (uint64, error); InsertValue and InsertRowValue now call these helpers then insertHashValue.
Protobuf Serialization & Deserialization pkg/statistics/row_sampler.go	ToProto serializes SingletonSketches and SketchSampleCount; FromProto restores FMSketches, SingletonSketches, resets singletonBuilders, and restores sketch sample counts.
Collector Merging with Sketch Support pkg/statistics/row_sampler.go	ReservoirRowSampleCollector.MergeCollector and BernoulliRowSampleCollector.MergeCollector merge SketchSampleCount and invoke mergeSingletonSketches to combine singleton sketches alongside FM sketches.
Sketch Accumulation in buildSamplingStats pkg/executor/analyze_col_sampling.go	buildSamplingStats accumulates deep-copied per-worker sketch data and sample-sizing metadata into nodeNDVSketches, nodeSingletonSketches, and related accumulators for NDV estimation.
NDV Estimation Helpers pkg/executor/analyze_col_sampling.go	Adds copySketches and estimateNDVsBySketch to compute per-column NDV estimates from root FM sketch NDV, node NDV sketches, singleton sketches, and sample metadata, skipping special indexes.
Histogram NDV Refinement Integration pkg/executor/analyze_col_sampling.go	subBuildWorker signature updated to accept estimateNDVs []int64; after histogram construction it may override hist.NDV when the sketch-based estimate exceeds the histogram-derived NDV.
Unit Tests pkg/executor/analyze_utils_test.go, pkg/statistics/sample_test.go	Adds TestEstimateNDVsBySketch, mustBuildFMSketch, and SubTestRowSampleSingletonSketches to validate NDV estimation, singleton sketch construction, and proto round-trip behavior.
Build & Dependency Pins go.mod, DEPS.bzl, cmd/tidb-server/BUILD.bazel, pkg/executor/builder.go, pkg/store/mockstore/unistore/cophandler/analyze.go	Bumps tipb module/pin (and adds replace to 0xPoe fork), adjusts Bazel test shard_count, propagates NDV sample rate into tipb AnalyzeColumnsReq and into cophandler full-sampling builder.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• pingcap/tidb#68158: Bumps tipb dependency to include singleton sketch fields that this PR serializes and transmits via row sample collectors.
• pingcap/tidb#66747: Changes related to analyze executor build flow and handling of per-worker collectors/sketches; related to worker signature/collector handling edits.
• pingcap/tidb#67518: Refinements to the GEE-based NDV estimation helpers used by this PR.

Suggested reviewers

• mjonss
• time-and-fate
• henrybw
• terry1purcell

Poem

🐰 I hopped through sampled rows tonight,
Collected hashes in soft moonlight,
Once-seen friends I marked with care,
NDVs tallied from sketches fair,
Histograms tidy — rabbit's delight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.75% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main changes: adding singleton sketch collection for row sampling in statistics and executor packages, accurately summarizing the primary objective.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The pull request description follows the required template with all critical sections completed: issue reference, problem summary, detailed explanation of changes, comprehensive checklist coverage, and appropriate release note.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Timed out fetching pipeline failures after 30000ms

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pantheon-ai]

@0xPoe I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

_{ℹ️ Learn more details on Pantheon AI.}

[pantheon-ai]

Review Complete

Findings: 0 issues
Posted: 0
Duplicates/Skipped: 0

_{ℹ️ Learn more details on Pantheon AI.}

[pantheon-ai]

✅ Code looks good. No issues found.

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

pkg/statistics/sample_test.go (1)

275-316: ⚡ Quick win

Please cover the merge path too.

This subtest only exercises build + proto round-trip. The analyze path consumes singleton sketches after FromProto() and MergeCollector(), so a deterministic case where the same value is singleton in two children and must disappear after merge would protect the new behavior much better.

As per coding guidelines, "Prefer extending existing test suites and fixtures over creating new scaffolding."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/statistics/sample_test.go` around lines 275 - 316, The test
SubTestRowSampleSingletonSketches only checks Build + proto round-trip; add a
merge/analysis path to cover the case where the same value is singleton in two
child collectors and must be removed after MergeCollector/Analyze. Create two
ReservoirRowSampleCollector instances via NewReservoirRowSampleCollector, feed
them deterministic rows so a particular value is singleton in both, call
BuildSingletonSketches on each, use ToProto/FromProto or directly call
MergeCollector to combine them, then run the same analysis path that consumes
singleton sketches and assert that the merged collector no longer treats that
value as singleton (NDV decreases/removes it) and that downstream
FromProto/MergeCollector round-trip preserves this behavior; reuse the existing
rows/colGroups and assertions in SubTestRowSampleSingletonSketches rather than
adding new scaffolding.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@DEPS.bzl`:
- Around line 6585-6591: The Bazel fetch is failing because the module zip for
com_github_pingcap_tipb-v0.0.0-20260414032333-da912b84de6f referenced in
DEPS.bzl (sha256 =
"68768a27ed6c35716fcb01a0b4a15ff13e5c1a5dc11acc7a3d44ba02a2742077", strip_prefix
= "github.com/pingcap/tipb@v0.0.0-20260414032333-da912b84de6f") is not available
on the configured mirrors (the listed URLs), so either upload the module zip for
that pseudo-version to all mirrors (http://bazel-cache.pingcap.net,
http://ats.apps.svc, https://cache.hawkingrei.com,
https://storage.googleapis.com/pingcapmirror) and confirm go.sum contains the
matching hash, or revert/remove this DEPS.bzl entry until the artifact is
mirrored; after uploading or reverting, re-run the CI pipeline to verify the
fetch succeeds.

In `@pkg/executor/analyze_col_sampling.go`:
- Around line 965-973: The code currently computes a single aggregated
sampleSize from nodeSketchSampleCounts/nodeSampleSizes and passes it to
EstimateNDVByGEE(), but sketches are populated per-slice (collectColumns() may
skip nulls), so you must track and pass a per-slice sample count: change the
loop that builds nodeSketchSampleCounts to also compute per-index
actualSampleCounts[i] (the number of rows that contributed to each sketch, using
the same logic collectColumns() uses to skip nulls), and replace uses of the
aggregated sampleSize with actualSampleCounts[i] when computing
sampleNDV/singletonItems and when calling EstimateNDVByGEE() (also apply the
same fix in the similar block around lines 992-999). Ensure variable names
referenced are nodeSketchSampleCounts, nodeSampleSizes, sampleSize,
collectColumns(), EstimateNDVByGEE(), sampleNDV, and singletonItems so the
correct per-slice counts are used.

In `@pkg/statistics/row_sampler.go`:
- Around line 435-450: The current mergeSingletonSketches uses MergeFMSketch
(union), which incorrectly treats "seen-once" items as still singletons after
merge; instead preserve singleton sketches at the original child-collector
granularity by changing the storage and merge behavior: stop calling
MergeFMSketch and instead append a copy of each incoming singletonSketch to a
per-child list (e.g., change baseCollector.SingletonSketches from []*FMSketch to
[][]*FMSketch or otherwise store sketches by child index) inside
mergeSingletonSketches (use singletonSketch.Copy()), and update
buildSamplingStats/estimateNDVsBySketch consumers to iterate the per-child
sketches rather than a single merged sketch so singletonItems aren’t
double-counted.
- Around line 485-487: When deserializing singleton sketches, FMSketchFromProto
leaves maxSize == 0 which later causes shrink logic/corruption when those
sketches are merged; after calling FMSketchFromProto(pbSketch) in the loop that
populates s.SingletonSketches, set the returned sketch's maxSize to the expected
non-zero capacity (e.g. the same maxSize used for new sketches in this package
or derived from s or pbCollector) so that MergeFMSketch and
mergeSingletonSketches operate on a properly initialized sketch; update the loop
that builds s.SingletonSketches to normalize maxSize on each sketch returned by
FMSketchFromProto before appending.

---

Nitpick comments:
In `@pkg/statistics/sample_test.go`:
- Around line 275-316: The test SubTestRowSampleSingletonSketches only checks
Build + proto round-trip; add a merge/analysis path to cover the case where the
same value is singleton in two child collectors and must be removed after
MergeCollector/Analyze. Create two ReservoirRowSampleCollector instances via
NewReservoirRowSampleCollector, feed them deterministic rows so a particular
value is singleton in both, call BuildSingletonSketches on each, use
ToProto/FromProto or directly call MergeCollector to combine them, then run the
same analysis path that consumes singleton sketches and assert that the merged
collector no longer treats that value as singleton (NDV decreases/removes it)
and that downstream FromProto/MergeCollector round-trip preserves this behavior;
reuse the existing rows/colGroups and assertions in
SubTestRowSampleSingletonSketches rather than adding new scaffolding.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: f100b334-ed63-4074-9cae-e68df8946d0b

📥 Commits

Reviewing files that changed from the base of the PR and between 33ae9e3 and 1d78d22.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (7)

• DEPS.bzl
• go.mod
• pkg/executor/analyze_col_sampling.go
• pkg/executor/analyze_utils_test.go
• pkg/statistics/fmsketch.go
• pkg/statistics/row_sampler.go
• pkg/statistics/sample_test.go

[codecov]

Codecov Report

❌ Patch coverage is 24.31507% with 221 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.1863%. Comparing base (2b9527b) to head (7957a28).
⚠️ Report is 34 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #68157        +/-   ##
================================================
- Coverage   77.2806%   75.1863%   -2.0943%     
================================================
  Files          2010       2040        +30     
  Lines        555351     578396     +23045     
================================================
+ Hits         429179     434875      +5696     
- Misses       125252     142082     +16830     
- Partials        920       1439       +519     

Flag	Coverage Δ
integration	42.8624% <24.3150%> (+3.0694%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	60.4679% <ø> (ø)
parser	∅ <ø> (∅)
br	48.7418% <ø> (-14.2673%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[0xPoe]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign time-and-fate for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/executor/OWNERS
• pkg/executor/test/analyzetest/OWNERS
• pkg/planner/OWNERS
• pkg/statistics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@0xPoe: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-mysql-client-test	7957a28	link	true	/test pull-mysql-client-test
idc-jenkins-ci-tidb/mysql-test	7957a28	link	true	/test mysql-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.