core: Expand.deriveRecordType omits the duplicate-index column

[alexandrefimov]

Both normative texts put a column after the expand fields. physical_relations.md, Expand Operation, gives the Direct Output Order as "The expand fields followed by an i32 column describing the index of the duplicate that the row is derived from"; the comment on ExpandRel in algebra.proto says "an extra int64 field is emitted which contains a zero-indexed ordinal corresponding to the duplicate definition". They disagree about the width, which is what substrait#714 is open on, and they agree that the column is there.

Expand.deriveRecordType() maps the fields and appends nothing:

public Type.Struct deriveRecordType() {
  Type.Struct initial = getInput().getRecordType();
  return TypeCreator.of(initial.nullable())
      .struct(getFields().stream().map(ExpandField::getType));
}

Observed

An ExpandRel over a two-column read (i64, i64?) carrying two consistent_field definitions, one per input column:

• derived: Struct{nullable=false, fields=[I64{nullable=false}, I64{nullable=true}]}
• expected: those two fields and the index column after them

A second plan, identical except that its second definition is a switching_field over the same two columns, derives the same two-column struct.

Reproduced at fff639064df794840db36fffcd881c09100e23df (v0.103.0); main at ee9f3d20a has the same method.

Whether that column is i32 or int64 is the open question in substrait#714, so a fix here has to choose ahead of it. For what it is worth as a second reading rather than as an argument, substrait-python 0.31.0 appends it as a required i32, at type_inference.py:839.

Reproducer

Convert this plan through ProtoPlanConverter and read the root's record type. The switching-field variant replaces the second consistentField with {"switchingField": {"duplicates": [<field 0>, <field 1>]}}.

{
  "relations": [
    {
      "root": {
        "input": {
          "expand": {
            "common": {
              "direct": {}
            },
            "input": {
              "read": {
                "common": {
                  "direct": {}
                },
                "baseSchema": {
                  "names": [
                    "c0",
                    "c1"
                  ],
                  "struct": {
                    "types": [
                      {
                        "i64": {
                          "nullability": "NULLABILITY_REQUIRED"
                        }
                      },
                      {
                        "i64": {
                          "nullability": "NULLABILITY_NULLABLE"
                        }
                      }
                    ],
                    "nullability": "NULLABILITY_REQUIRED"
                  }
                },
                "namedTable": {
                  "names": [
                    "t_rn"
                  ]
                }
              }
            },
            "fields": [
              {
                "consistentField": {
                  "selection": {
                    "directReference": {
                      "structField": {}
                    },
                    "rootReference": {}
                  }
                }
              },
              {
                "consistentField": {
                  "selection": {
                    "directReference": {
                      "structField": {
                        "field": 1
                      }
                    },
                    "rootReference": {}
                  }
                }
              }
            ]
          }
        }
      }
    }
  ],
  "version": {
    "minorNumber": 102,
    "producer": "case-corpus"
  }
}