Fix column lineage transformation details always appearing as null

[kalra-mohit]

Marquez's LineageEvent.ColumnLineageInputField (used to deserialize the OpenLineage ColumnLineageDatasetFacet) only ever captured namespace/name/field — no transformations property existed on it at all. Meanwhile the write path, OpenLineageDao.upsertColumnLineage, only read transformationDescription/transformationType off the output column (ColumnLineageOutputColumn), which the current OpenLineage spec marks deprecated: true.

Modern producers report transformation details per input field instead, nested under inputFields[].transformations[] — exactly what's in the issue's example:

"customer_full": {
  "inputFields": [
    {
      "namespace": "bookstore2", "name": "customers", "field": "customer_email",
      "transformations": [{ "type": "DIRECT", "subtype": "TRANSFORMATION", "description": "concat(...)" }]
    }
  ]
}

Since the model had nowhere to deserialize that array into, and the write path never looked at the deprecated output-column fields for producers that only emit the new shape, transformation_description/transformation_type landed as null for anyone using the current facet format — reproducing the bug as reported (a second user hit the same thing in the issue thread).

flowchart TD
    E["OpenLineage event:
inputFields[].transformations[] (current spec)"]
    E --> Old["Before: ColumnLineageInputField has no transformations field"]
    Old --> O1["Write path only reads deprecated
outputColumn.transformationDescription/Type"]
    O1 --> R1["null/null persisted whenever producer
only emits the new per-field format"]
    E --> New["After: transformationOf(inputField, outputColumn)"]
    New --> O2{"inputField.transformations present?"}
    O2 -->|yes| R2["Use that field's own transformation
description/type"]
    O2 -->|no - legacy producer| R3["Fall back to deprecated
outputColumn description/type"]

Loading

Fix:

• Added transformations (List<Transformation>) to ColumnLineageInputField, with a new Transformation class (type/subtype/description/masking) mirroring the spec. Kept the existing 3-arg constructor working unchanged so no call site needed to move — verified by compiling all 8 existing usages untouched.
• Added OpenLineageDao.transformationOf(inputField, outputColumn) — a small pure function that prefers the first entry of that specific input field's own transformations list, and falls back to the deprecated output-column-level fields for producers still on the old format.
• upsertColumnLineage now resolves the transformation per matched input field via transformationOf and groups input fields by their resolved (description, type) before calling the existing batch ColumnLineageDao.upsertColumnLineageRow(...) once per group. That persists a distinct transformation per input/output edge, which column_lineage and the read path already supported — without touching ColumnLineageDao's public signature, so ColumnLineageDaoTest didn't need to change either.

Testing: new OpenLineageDaoColumnLineageTransformationTest (unit, no DB) exercises transformationOf directly against the issue's exact scenario, plus multi-field, fallback-to-deprecated, both-formats-absent, and multiple-transformations-on-one-field cases. Did the fail-then-pass check by hand: temporarily reverted transformationOf to always read the deprecated fields, saw 3 of 5 tests fail with null description/type, restored the fix, all 5 passed.

Also added OpenLineageDaoTest#testUpdateMarquezModelDatasetWithColumnLineageFacet_usesNewTransformationsFormat, a full DB-backed test that builds a facet using only the new inputFields[].transformations[] format (deprecated fields deliberately left unset, matching real modern producers) and checks the persisted row has a non-null description/type.

./gradlew :api:testUnit passes (123 tests, including the 5 new ones). Couldn't run the Postgres-backed OpenLineageDaoTest locally — same Testcontainers/Docker API mismatch I keep hitting in this sandbox (bundled docker-java wants API v1.32, the local engine needs >= v1.40; confirmed pre-existing by hitting it on unmodified RunDaoTest too). :api:compileTestJava passes, so the new integration test at least compiles clean against the real classes. Would appreciate CI running the DB suite here.

Known gap I'm leaving in (marked with a comment in the code)

While working on this I found a real edge case in upsertColumnLineage that's worth surfacing outside a code comment. When a single output column's input fields span 2+ distinct transformation groups, upsertColumnLineageRow(...) gets called once per group — but that method's existing implementation always returns all column_lineage rows currently persisted for that (outputDatasetVersionUuid, outputDatasetFieldUuid) pair, not just the rows the current call wrote. So across multiple groups, the same row can show up more than once in the List<ColumnLineageRow> this method returns.

It has no production impact right now: the only consumer of that return value, DatasetRecord#getColumnLineageRows(), isn't read anywhere in production code — only in tests, and this PR's own new test only exercises the single-group case, so it doesn't trigger the duplication. The rows actually persisted in column_lineage are correct either way; this only affects the in-memory list handed back up the call stack.

I left a // NOTE: comment right above the relevant code explaining this, in case someone adds a real consumer of getColumnLineageRows() later — they'd want to either dedupe or change upsertColumnLineageRow's contract to return only newly-written rows. Not fixing it here since it's inert today and out of scope for the null-transformation bug this PR is actually about.

Fixes #3100.

[kalra-mohit]

@merobi-hub whenever you get a chance to take a look, happy to address any feedback!