You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
coerce_int96 leaves INT96 in two-level and legacy Parquet lists as nanoseconds #26463
With datafusion.execution.parquet.coerce_int96 set, INT96 timestamps inside lists that do not use the three-level LIST layout stay Timestamp(Nanosecond, None). That covers Hive's two-level lists (repeated int96 bag, how Hive writes array<timestamp>), bare repeated int96 fields, and parquet-mr Avro's legacy lists whose repeated group is named array. Values outside 1677-09-21 to 2262-04-11 then overflow on decode, which is what coerce_int96 is meant to prevent: 0001-01-01T01:02:03.123456 reads back in 1754.
To Reproduce
use std::sync::Arc;use arrow::datatypes::TimeUnit;use datafusion_datasource_parquet::Int96Coercer;use parquet::arrow::parquet_to_arrow_schema;use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};let descr = SchemaDescriptor::new(Arc::new(parse_message_type("message m { optional group hive (LIST) { repeated int96 bag; } optional group three (LIST) { repeated group list { optional int96 element; } } }",).unwrap(),));let arrow_schema = parquet_to_arrow_schema(&descr,None).unwrap();let coerced = Int96Coercer::new(&descr,&arrow_schema,&TimeUnit::Microsecond).coerce().unwrap();println!("{:?}", coerced.field_with_name("hive").unwrap().data_type());println!("{:?}", coerced.field_with_name("three").unwrap().data_type());
hive keeps Timestamp(Nanosecond, None) while three becomes Timestamp(Microsecond, None).
Expected behavior
Every INT96 leaf coerces to the requested unit, whatever the list encoding.
Additional context
coerce_int96_to_resolution_impl in datasource-parquet/src/schema_coercion.rs finds INT96 leaves by rebuilding each Arrow leaf's Parquet path, and assumes a list's path is <list>.list.<element>. For hive it looks up hive.list.bag, which no column has, since the path is hive.bag. It also never descends LargeList or FixedSizeList.
Willingness to contribute
I plan to contribute a fix for this bug independently
Activity