fix(optimizer): keep scans that read virtual columns on ParquetSource

[anoop-narang]

Queries that read a DataFusion virtual column fail once the liquid cache is on.

SELECT file_row_index() AS pos, id FROM t

Plain DataFusion returns each row's position in its file. Through LiquidCacheLocalBuilder it errors:

SchemaError("Unable to get field named \"__datafusion_file_row_index\". Valid fields: [\"id\"]")

Why

file_row_index() is answered by the parquet reader: projection pushdown rewrites it into a row-number virtual column (arrow's RowNumber extension type) on the scan's TableSchema, and ParquetSource asks the parquet reader to produce it. Table providers can declare such columns directly too, for example to apply positional deletes.

LocalModeOptimizer swaps every parquet scan to LiquidParquetSource. The liquid read path carries the TableSchema across but never produces a virtual column, so the batch comes back without it and resolving the column fails.

The fix

convert_parquet_scan declines the swap when the scan reads a virtual column, so the scan stays on ParquetSource and logs a BYPASS line at info. The check covers what the scan actually reads (the projection, unioned with the pushed-down filter), not what the table declares, so a table carrying a row-number column keeps the cache for queries that never read it. A scan with no projection reads the whole table schema and is declined.

Declining rather than producing positions in the liquid reader is deliberate for now: a row number is an absolute position in the file, and a plausible-but-shifted one would silently attach a delete or lineage record to the wrong row. Generating them in the liquid reader would be the follow-up that lets these scans use the cache.

The filter half of the check is defensive. DataFusion 55's ParquetSource::try_pushdown_filters already refuses predicates that reference a virtual column, so today it should not trigger; it covers a source that does push one down, since the liquid reader would fail rewriting it against the file schemas.

Tests

• datafusion-local/src/tests/virtual_column.rs: runs the file_row_index() query above (12000 rows, spanning batches) through plain DataFusion and through the liquid cache, twice, and compares results.
• optimizers unit tests: the guard keys on what is read, not what is declared (projection, filter, and no-projection cases), and convert_parquet_scan keeps a virtual-column scan on ParquetSource with an ordinary scan as a positive control.

With only the new test applied to main:

$ cargo test -p liquid-cache-datafusion-local virtual_column

test tests::virtual_column::file_row_index_matches_datafusion ... FAILED
called `Result::unwrap()` on an `Err` value: ArrowError(SchemaError("Unable to get field named \"__datafusion_file_row_index\". Valid fields: [\"id\"]"), Some(""))

With the fix, liquid-cache-datafusion (62) and liquid-cache-datafusion-local (32) pass, and clippy is clean with -D warnings.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@5237fdd). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #523   +/-   ##
=======================================
  Coverage        ?   81.16%           
=======================================
  Files           ?       80           
  Lines           ?    13414           
  Branches        ?    13414           
=======================================
  Hits            ?    10888           
  Misses          ?     2201           
  Partials        ?      325           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.