ToSubstraitType.convert (spark/src/main/scala/io/substrait/spark/ToSubstraitType.scala:158-159) and ToSubstraitLiteral (spark/src/main/scala/io/substrait/spark/expression/ToSubstraitLiteral.scala:106-109) match the default interval types only:
case DayTimeIntervalType.DEFAULT => Some(creator.intervalDay(Util.MICROSECOND_PRECISION))
case YearMonthIntervalType.DEFAULT => Some(creator.INTERVAL_YEAR)
YearMonthIntervalType.DEFAULT is YearMonthIntervalType(YEAR, MONTH), so a narrowed type — YearMonthIntervalType(YEAR, YEAR), which Spark's DDL and parser produce for INTERVAL YEAR — falls through to case _ => None / case _ => null and the conversion fails with Unable to convert the type interval year. The same holds for DayTimeIntervalType(HOUR, HOUR) and the rest of the day-time field combinations.
Why the mapping would be lossless
Substrait has no notion of interval start/end fields. interval_year is a years/months pair and interval_day<P> is days/seconds/subseconds; a narrowed Spark type carries the same physical value (a months Int, a micros Long) and only restricts which fields Spark will render. So converting YearMonthIntervalType(YEAR, YEAR) to interval_year loses only the rendering hint, not the value.
The asymmetry only bites Spark-authored plans: ToSparkType always emits .DEFAULT on the way in, so a round trip through Substrait normalizes the fields and never produces a narrowed type of its own.
Open question
Whether normalizing to the default on the way out is acceptable, or whether the field restriction should be preserved somehow. Substrait has nowhere to put it, so the realistic choice is between converting and normalizing (a Spark-visible behaviour change on round trip: INTERVAL YEAR comes back as INTERVAL YEAR TO MONTH) and continuing to reject. Rejecting is arguably the honest option, in which case the fix is a clearer message than the generic one — the current failure gives no hint that the fields are the problem rather than the type.
Found while reviewing #1140.
ToSubstraitType.convert(spark/src/main/scala/io/substrait/spark/ToSubstraitType.scala:158-159) andToSubstraitLiteral(spark/src/main/scala/io/substrait/spark/expression/ToSubstraitLiteral.scala:106-109) match the default interval types only:YearMonthIntervalType.DEFAULTisYearMonthIntervalType(YEAR, MONTH), so a narrowed type —YearMonthIntervalType(YEAR, YEAR), which Spark's DDL and parser produce forINTERVAL YEAR— falls through tocase _ => None/case _ => nulland the conversion fails withUnable to convert the type interval year. The same holds forDayTimeIntervalType(HOUR, HOUR)and the rest of the day-time field combinations.Why the mapping would be lossless
Substrait has no notion of interval start/end fields.
interval_yearis a years/months pair andinterval_day<P>is days/seconds/subseconds; a narrowed Spark type carries the same physical value (a monthsInt, a microsLong) and only restricts which fields Spark will render. So convertingYearMonthIntervalType(YEAR, YEAR)tointerval_yearloses only the rendering hint, not the value.The asymmetry only bites Spark-authored plans:
ToSparkTypealways emits.DEFAULTon the way in, so a round trip through Substrait normalizes the fields and never produces a narrowed type of its own.Open question
Whether normalizing to the default on the way out is acceptable, or whether the field restriction should be preserved somehow. Substrait has nowhere to put it, so the realistic choice is between converting and normalizing (a Spark-visible behaviour change on round trip:
INTERVAL YEARcomes back asINTERVAL YEAR TO MONTH) and continuing to reject. Rejecting is arguably the honest option, in which case the fix is a clearer message than the generic one — the current failure gives no hint that the fields are the problem rather than the type.Found while reviewing #1140.