[yyy1000]

Summary

When a view projects a leaf out of a nullable struct — SELECT job.region AS job_region — the generated Avro schema declares the flattened column non-nullable if the leaf is required inside its parent. The view still returns null on every row where the parent struct is absent, so the published schema is stricter than the data: an Avro writer fails on those rows, and consumers built against the previously nullable column break.

This is a pre-existing bug in shared view code, introduced with visitFieldAccess in #83 / #110 (2021). It is not specific to any one schema source — see "Impact on the Hive path" below.

Root cause

Two nullability facts are in play, and both matter:

Fact	Meaning
job is nullable	the struct may be absent
job.region is required	if a struct exists, region is populated

Flattening collapses two levels into one column, so that column must carry the union of both:

nullable(a.b.c) = nullable(a) OR nullable(a.b) OR nullable(a.b.c)

Only the last term was evaluated. getFieldFromTopSchema calls extractIfOption(topSchema) on entry and again at the end of every loop iteration, discarding each ancestor's ["null", …] branch; handleFieldAccess then copies the leaf's schema up verbatim. By the time the leaf is located, the only record that the parent could be absent is gone.

Note the leaf's required is not pushed onto the parent — the parent is erased and the leaf takes its place. Selecting a struct wholesale (SELECT job) was always correct: the envelope survives and carries the absent case.

This has been observed on real views whose base tables carry a nullable struct with required leaves, where the leaf is never null while its parent is present — so the leaf's required is accurate, and it is precisely the flattened column that has to widen.

Impact on the Hive path

The buggy code is shared, and views resolving through dali.row.schema are shielded only by an accident of input: ToNullableSchemaVisitor force-converts that schema to all-nullable, so no required leaf ever reaches the traversal.

That shield does not apply to tables whose partner is avro.schema.literal, since MergeHiveSchemaWithAvro.primitive keeps a leaf non-nullable when the partner says so. The new tests deliberately run through that path and fail without the fix.

The fix

• getFieldFromTopSchema returns the resolved field plus a nullableAncestor flag, set wherever an extractIfOption strips a null branch — before the loop and on every iteration, so intermediate structs, array elements and map values all count.
• handleFieldAccess applies makeNullable when any ancestor was optional, then reorderOptionIfRequired so a leaf carrying a default stays valid Avro (["string","null"], since Avro requires the default to match the first union branch).
• SchemaUtilities gains an appendField overload taking an explicit schema; the existing 3-arg form delegates to it, so no existing caller changes behaviour.

Updated golden file

testProjectStructInnerField-expected.avsc has encoded this bug since #83 (May 2021), asserting Struct_Col.Bool_Field → "boolean" against a nullable Struct_Col. The view's WHERE Struct_Col IS NOT NULL does not justify it: LogicalFilter is documented in RelToAvroSchemaConverter as a nullability pass-through, so the output is identical with or without the predicate. Updated to the correct expectation.

Tests

Five new tests in ViewToAvroSchemaConverterTests, all running through the Hive path with avro.schema.literal:

Test	Covers
testFlattenedFieldFromNullableStructIsNullable	main case, plus a control that a required leaf under a non-nullable struct stays required
testFlattenedFieldFromNullableGrandparentIsNullable	nullable grandparent, non-nullable parent
testFlattenedFieldFromNullableArrayAndMapIsNullable	array-element and map-value traversal
testFlattenedFieldFromNullableStructKeepsDefaultValid	required leaf with a default; asserts branch order and a parser round-trip
testSelectingNullableStructWholesaleKeepsRequiredLeaf	SELECT job unchanged — no over-nullification

With the main-code change reverted, 4 of the 5 fail. Full ./gradlew test green; spotless clean.

The change has additionally been validated against a large corpus of existing views, where it corrected exactly the affected flattened columns and left every other schema unchanged.

[aastha25]

Excellent find, this latent bug has existed since schema inference was first introduced here. Nice work on the fix too: it lives in the shared traversal code, so it applies uniformly to both the existing schema-inference path and the new one, with no branch-specific special-casing needed.

Just one comment on testing coverage -
Current tests only prove 1-hop ( Job.Region ) and 2-hop/grandparent ( Wrapper.Mid.Leaf ) propagation. Could you add a 3+ level case, e.g.  A(required) → B(nullable) → C(required) → leaf(required) , asserting the flattened leaf column comes out nullable? That's the only way to confirm  nullableAncestor ||= ...  inside the loop actually accumulates across iterations, rather than just happening to work for 2 hops.

Note that testSelectDeepNestStructFieldFromDeepNestComplex  looks like it already covers this (3 levels deep) but doesn't: its base table ( basedeepnestedcomplex ) has no  avro.schema.literal , so it falls back to  TypeInfoToAvroSchemaConverter(mkFieldsOptional=true) , which force-nullifies every field — leaves included — regardless of ancestor nullability. That test would still pass even if  nullableAncestor  tracking were deleted entirely, so it isn't real evidence of 3+-level correctness.