Skip to content

fix(reader): honour bloom_filter_on_read - #524

Merged
anoop-narang merged 1 commit into
datafusion-contrib:mainfrom
hotdata-dev:upstream/honor-bloom-filter-on-read
Oct 12, 2026
Merged

anoop-narang merged 1 commit into
datafusion-contrib:mainfrom
hotdata-dev:upstream/honor-bloom-filter-on-read

Conversation

@anoop-narang

Copy link
Copy Markdown
Collaborator

The liquid parquet reader ignores datafusion.execution.parquet.bloom_filter_on_read. Whenever a scan has a pruning predicate, it loads and applies every remaining row group's bloom filters, even with the option set to false.

let config = SessionConfig::new()
    .set_bool("datafusion.execution.parquet.bloom_filter_on_read", false);
let ctx = SessionContext::new_with_config(config);
// register a parquet table `t` with bloom filters, then:
// SELECT * FROM t WHERE a = 2

With ParquetSource, this scan reads no bloom filters. Once the scan goes through LiquidParquetSource, it reads them anyway, and row_groups_pruned_bloom_filter reports row groups pruned by a filter the session turned off.

Why it matters

Each bloom filter is a separate ranged read per row group and predicate column, and the reads are not cached across queries. Callers turn the option off when bloom checks cannot pay for themselves — for example, many small files where file-level pruning has already selected exactly the files that match. Today they have no way to avoid those reads.

The option was honoured in the original opener and dropped in d7f0741 ("remove some flags"). If leaving it out was deliberate, I'm happy to hear the reasoning; otherwise this restores it.

The fix

LiquidParquetSource already carries the session's TableParquetOptions. Pass global.bloom_filter_on_read into the morselizer and skip the bloom-filter load when it is off. As in DataFusion's own opener, the remaining row groups are then counted as matched by bloom filters, so the pruning metrics stay consistent between the two sources.

Tests

The existing prunes_row_group_with_bloom_filter test becomes a helper parameterised on the option, and a new test covers the off case:

  • prunes_row_group_with_bloom_filter (on): 1 row, 1 row group pruned, 1 matched.
  • skips_bloom_filter_when_bloom_filter_on_read_is_off (off): 1 row, 0 pruned, 2 matched. The predicate alone still produces the right answer.

With the test applied to main and the fix left out:

$ cargo test -p liquid-cache-datafusion --lib bloom

test optimizers::tests::prunes_row_group_with_bloom_filter ... ok
test optimizers::tests::skips_bloom_filter_when_bloom_filter_on_read_is_off ... FAILED
assertion `left == right` failed
  left: 1
 right: 0

With the fix, cargo test -p liquid-cache-datafusion passes (63 tests).

The morsel reader loaded every row group's bloom filters whenever the
scan had a pruning predicate, ignoring the session's
`datafusion.execution.parquet.bloom_filter_on_read`. DataFusion's own
parquet opener skips them when the option is off, so a caller that
turns it off to save the per-file bloom reads got them anyway once the
scan went through liquid cache.

Carry the option from the source's TableParquetOptions into the
morselizer, skip the bloom-filter load when it is off, and count the
remaining row groups as matched by bloom filters, as DataFusion does.
@codecov-commenter

codecov-commenter commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@8225b68). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #524   +/-   ##
=======================================
  Coverage        ?   81.15%           
=======================================
  Files           ?       80           
  Lines           ?    13453           
  Branches        ?    13453           
=======================================
  Hits            ?    10918           
  Misses          ?     2212           
  Partials        ?      323           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anoop-narang
anoop-narang merged commit 53414cd into datafusion-contrib:main Oct 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants