Skip to content

coerce_int96 leaves INT96 in two-level and legacy Parquet lists as nanoseconds #26463

Open

Description

@ranflarion

Describe the bug

With datafusion.execution.parquet.coerce_int96 set, INT96 timestamps inside lists that do not use the three-level LIST layout stay Timestamp(Nanosecond, None). That covers Hive's two-level lists (repeated int96 bag, how Hive writes array<timestamp>), bare repeated int96 fields, and parquet-mr Avro's legacy lists whose repeated group is named array. Values outside 1677-09-21 to 2262-04-11 then overflow on decode, which is what coerce_int96 is meant to prevent: 0001-01-01T01:02:03.123456 reads back in 1754.

To Reproduce

use std::sync::Arc;
use arrow::datatypes::TimeUnit;
use datafusion_datasource_parquet::Int96Coercer;
use parquet::arrow::parquet_to_arrow_schema;
use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};

let descr = SchemaDescriptor::new(Arc::new(
    parse_message_type(
        "message m {
          optional group hive (LIST) { repeated int96 bag; }
          optional group three (LIST) { repeated group list { optional int96 element; } }
        }",
    )
    .unwrap(),
));
let arrow_schema = parquet_to_arrow_schema(&descr, None).unwrap();
let coerced = Int96Coercer::new(&descr, &arrow_schema, &TimeUnit::Microsecond)
    .coerce()
    .unwrap();
println!("{:?}", coerced.field_with_name("hive").unwrap().data_type());
println!("{:?}", coerced.field_with_name("three").unwrap().data_type());

hive keeps Timestamp(Nanosecond, None) while three becomes Timestamp(Microsecond, None).

Expected behavior

Every INT96 leaf coerces to the requested unit, whatever the list encoding.

Additional context

coerce_int96_to_resolution_impl in datasource-parquet/src/schema_coercion.rs finds INT96 leaves by rebuilding each Arrow leaf's Parquet path, and assumes a list's path is <list>.list.<element>. For hive it looks up hive.list.bag, which no column has, since the path is hive.bag. It also never descends LargeList or FixedSizeList.

Willingness to contribute

I plan to contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions