fix(spark)!: derive the dialect from core's model and the runtime's functions

[nielspardon]

The generator restated things it could derive, and each restatement was a way for the published spark_dialect.yaml to drift without a test failing.

Aggregate and window functions came from the collection merged with spark.yml while the runtime converters bind against the standard extensions only, so an aggregate added to spark.yml would be advertised and then fail with "Unable to find binding for call". All three sections now come from the collections SparkExtension hands the runtime, passed to the generator explicitly.

The dependencies block was an unsorted Map, so its order was an artifact of one Scala version's String hashing; it is now built as a SortedMap from the URNs the emitted functions actually reference. A hand-maintained URN-to-alias map returned "" for an unmapped URN, emitting a source that dangled against dependencies while still validating against the schema, which declares source as a plain string; the alias is now derived from the URN's last segment, and two URNs deriving the same alias fail rather than one silently displacing the other.

The dialect is emitted through io.substrait.dialect.Dialect rather than a parallel set of Scala case classes that typed enums as String, so the generator can no longer express a dialect core would reject, and a dialect-schema field core gains no longer has to be added a second time before Spark can express it.

The published file is unchanged apart from key order: core's field order, dependencies sorted, and max_precision ahead of system_metadata. Comparing the parsed models cannot catch an ordering change, since Dialect.dependencies is a Map, so the published text is now compared as text — which in turn needs the file declared as a Test input so that editing it invalidates the tests.

Worth knowing while reviewing: the aggregate/window fix has no test that can fail today. With spark.yml declaring no aggregate or window function, the merged collection and the standard collection are indistinguishable, so reverting that wiring alone breaks nothing observable. The guard checks the advertised aggregates and windows against DefaultExtensionCatalog.DEFAULT_COLLECTION directly rather than against what the generator was handed, so it arms the moment spark.yml gains one — the same moment the bug would go live. dependencies is still derived from function URNs only, so a USER_DEFINED supported type would need its own alias folded in; the dangling-source test covers type sources too, so that would fail rather than ship.

Two unreachable branches in the function probe went with the rewrite, one of them a println aimed at the same System.out that main writes the dialect to.

Closes #1087

BREAKING CHANGE: io.substrait.spark.utils.Dialect, SupportedType, TypeMetadata, FunctionMetadata and SupportedFunction are removed; the dialect is modelled by io.substrait.dialect.Dialect and friends. DialectGenerator.generate() returns io.substrait.dialect.Dialect, and the DialectGenerator class now takes the scalar, aggregate and window function collections it generates from.

[alexandrefimov]

Read this against main at 961f83e, which now includes #1128 — that is where the one thing I would act on comes from.

max_precision for the three temporal types is the last restatement left in supportedTypes, and it is the kind this PR is about. Some(9) says what Util.MICROSECOND_PRECISION and the conversion guard already say, and they now say 6: #1128 pinned the type conversions at exactly microseconds and fed the generator from that constant. Rebasing conflicts in DialectGenerator.scala and spark_dialect.yaml, and resolving either in this branch's favour puts the 9 back — I tried it, and ./gradlew dialect regenerates the file with max_precision: 9 in three places, which fails core's SparkDialectParseTest.parsesPrecisionTypes with expected: <6> but was: <9>. CI catches it, so this is a heads-up for the rebase rather than a defect in what is here.

Separately, I checked the premise of the aggregate/window fix, since you note it has no test that can fail today. It holds: on main the generator reads COLLECTION.aggregateFunctions() where COLLECTION = EXTENSION_COLLECTION.merge(SparkImpls), while toAggregateFunction and toWindowFunction are built from EXTENSION_COLLECTION alone. So the advertised set really is the wider one, and the scalar side was already consistent because the generator used SparkScalarFunctions, which is the merged one on both sides.

[nielspardon]

Rebased onto 961f83e and resolved both conflicts in #1128's favour: supportedTypes reads Util.MICROSECOND_PRECISION and carries your comment, so ./gradlew dialect emits max_precision: 6 in all three places. The regenerated file's content now matches main's exactly — the only difference left is the key ordering this PR introduces (core's field order, dependencies sorted, max_precision ahead of system_metadata). Your suggestion applies as-is apart from scalafmt reflowing two of the three calls, which are over 100 columns on one line.

Thanks for checking the aggregate/window premise independently — that was the part I could not pin with a test.

[alexandrefimov]

Re-read this after the rebase. Checked the claim about the published file by regenerating rather than by reading: ./gradlew :spark:spark-4.0_2.13:dialect rewrites spark_dialect.yaml byte for byte, and a normalised comparison against main's copy is identical, so the only difference really is key order.

Two things inline, plus one that would not anchor because the line falls between the diff hunks. In DialectSuite, generate validated YAML asserts only that the file exists, and main calls f.createNewFile() before it constructs the writer — so exists() holds even if nothing was written, and the test passes whatever the generator emits. Comparing the file's content against published would make it cover the path it is named for. (The temp file is deleted on entry but not on exit, so build/tmp/test/dialect.yaml outlives the run.)

None of the three is blocking.

[alexandrefimov]

Re-checked this on top of current main (934a60e), since the branch sits on 961f83e: the merge is clean — only the three build.gradle.kts files auto-merge — and ./gradlew build on the merged tree passes, with DialectSuite's ten tests green on all three variants. That includes the byte-for-byte comparison, so the published file matches what the generator emits at this main.

The precision question from my first pass is settled by the branch as it stands: supportedTypes reads Util.MICROSECOND_PRECISION, and the published file carries max_precision: 6 in all three places.

The three things I left earlier are non-blocking and independent of each other: PathSensitivity.NONE on the new inputs.file, the name of SparkAggregateFunctions/SparkWindowFunctions sitting next to the merged SparkScalarFunctions, and generate validated YAML, which asserts only that the file exists — main creates it before writing, so it passes whatever is written; the new byte-for-byte test covers the content, which leaves that one covering the CLI path alone.

[nielspardon]

All three applied in 350ce87.

PathSensitivity.NONE on the new inputs.file: -Dorg.gradle.caching.debug=true now prints IGNORED_PATH{...spark_dialect.yaml=IGNORED / 98ceb602...} for publishedDialect instead of ABSOLUTE_PATH.

Renamed to StandardAggregateFunctions / StandardWindowFunctions rather than commenting the lines — both are new in this branch, so it costs nothing beyond it, and the name carries further than a comment does.

generate validated YAML now reads the file back and compares it to published, and deletes it on the way out. Checked the premise the way you'd want rather than by reading: dropping out.write(yaml) from main makes the rewritten test fail where the old one passed. It's the only test covering the CLI write path, so it's named for that now.

Filing the two you set aside separately: the dialect task's missing outputs, and COLLECTION having no main-source consumer left.

[nielspardon]

Filed the two follow-ups: #1233 (the dialect task's missing outputs) and #1234 (COLLECTION without a main-source consumer).

#1233 turned out sharper than the ordering question it started as. Because dialect declares no output, Gradle fingerprints spark_dialect.yaml at whatever state the test task finds it in — so --parallel :spark:spark-4.0_2.13:dialect :spark:spark-3.4_2.12:test on a hand-edited file passes and banks that pass under the edited file's fingerprint. Re-applying the same edit then gets UP-TO-DATE and a green build on content that fails under --rerun-tasks. Repro steps are in the issue, along with your dependsOn-defeats-the-comparison point, which is why the ordering half needs mustRunAfter rather than the obvious fix.