Repository navigation
fix(reader): honour bloom_filter_on_read - #524
Merged
anoop-narang merged 1 commit intoOct 12, 2026
Merged
anoop-narang merged 1 commit into
anoop-narang merged 1 commit into
Conversation
The morsel reader loaded every row group's bloom filters whenever the scan had a pruning predicate, ignoring the session's `datafusion.execution.parquet.bloom_filter_on_read`. DataFusion's own parquet opener skips them when the option is off, so a caller that turns it off to save the per-file bloom reads got them anyway once the scan went through liquid cache. Carry the option from the source's TableParquetOptions into the morselizer, skip the bloom-filter load when it is off, and count the remaining row groups as matched by bloom filters, as DataFusion does.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #524 +/- ##
=======================================
Coverage ? 81.15%
=======================================
Files ? 80
Lines ? 13453
Branches ? 13453
=======================================
Hits ? 10918
Misses ? 2212
Partials ? 323 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The liquid parquet reader ignores
datafusion.execution.parquet.bloom_filter_on_read. Whenever a scan has a pruning predicate, it loads and applies every remaining row group's bloom filters, even with the option set tofalse.With
ParquetSource, this scan reads no bloom filters. Once the scan goes throughLiquidParquetSource, it reads them anyway, androw_groups_pruned_bloom_filterreports row groups pruned by a filter the session turned off.Why it matters
Each bloom filter is a separate ranged read per row group and predicate column, and the reads are not cached across queries. Callers turn the option off when bloom checks cannot pay for themselves — for example, many small files where file-level pruning has already selected exactly the files that match. Today they have no way to avoid those reads.
The option was honoured in the original opener and dropped in d7f0741 ("remove some flags"). If leaving it out was deliberate, I'm happy to hear the reasoning; otherwise this restores it.
The fix
LiquidParquetSourcealready carries the session'sTableParquetOptions. Passglobal.bloom_filter_on_readinto the morselizer and skip the bloom-filter load when it is off. As in DataFusion's own opener, the remaining row groups are then counted as matched by bloom filters, so the pruning metrics stay consistent between the two sources.Tests
The existing
prunes_row_group_with_bloom_filtertest becomes a helper parameterised on the option, and a new test covers the off case:prunes_row_group_with_bloom_filter(on): 1 row, 1 row group pruned, 1 matched.skips_bloom_filter_when_bloom_filter_on_read_is_off(off): 1 row, 0 pruned, 2 matched. The predicate alone still produces the right answer.With the test applied to
mainand the fix left out:With the fix,
cargo test -p liquid-cache-datafusionpasses (63 tests).