equality_ids written as list<long>, but spec and all other implementations use list<int>

Apache Iceberg version

main (development)

Please describe the bug 🐞

equality_ids is declared as a list of long in the manifest-entry Avro schema, but the Iceberg spec, the Java reference implementation, and iceberg-cpp all specify int. pyiceberg is the only implementation that disagrees, so manifests it writes are rejected outright by spec-conformant readers.

Evidence

source	element type of field 136
Iceberg spec, `format/spec.md` ("Manifests" table)	`list<136: int>`
Java, `api/src/main/java/org/apache/iceberg/DataFile.java`	`ListType.ofRequired(136, IntegerType.get())`
iceberg-cpp, `src/iceberg/manifest_entry.h`	`iceberg::int32()`
pyiceberg, `pyiceberg/manifest.py:298` and `:393`	`LongType()`

Both the v1 and v2 manifest-entry schemas are affected. Note that pyiceberg's own type hint already says int:

# pyiceberg/manifest.py:528
def equality_ids(self) -> list[int] | None:

So only the Avro schema disagrees; the Python side already treats these as ints.

Impact

A spec-conformant reader cannot read a pyiceberg-written manifest at all. iceberg-cpp fails with:

Cannot read Iceberg type: int from Avro type: "long"

because its kInt case accepts only AVRO_INT (Avro permits int → long promotion, not the reverse). Any Java or C++ consumer of pyiceberg-written metadata hits the same wall.

This is not limited to equality-delete tables: the schema is written into every manifest, so an ordinary append-only table with no delete files is equally unreadable.

Reproducer

import fastavro

# any table written by pyiceberg
with open("<some>-m0.avro", "rb") as f:
    schema = fastavro.reader(f).writer_schema

# -> equality_ids: ["null", {"element-id": 136, "type": "array", "items": "long"}]
#    spec says items should be "int"

The obvious fix breaks backwards compatibility

Changing LongType() → IntegerType() in those two places makes pyiceberg unable to read every manifest it has previously written:

ResolveError: Cannot promote long to int

Two reasons compound here:

pyiceberg/schema.py registers promote for IntegerType (int → long is legal) but has no LongType registration, so long → int falls through to the generic case and raises (schema.py:1660).
MANIFEST_ENTRY_SCHEMAS is used for both directions — the writer takes file_schema from it and the reader takes record_schema from it — so changing the constant changes the read path too.

Verified empirically: patching field 135's element type to IntegerType and then reading a manifest written by the current release raises the above.

Possible directions

A correct fix has to write int while continuing to read long. Options, roughly:

Split the write and read schemas so the writer emits the spec type while the reader keeps accepting the historical one. Probably the cleanest, and it changes no promotion rules.
Allow long → int narrowing for this field specifically. Smaller diff, but it widens the resolver in a direction the Avro and Iceberg promotion rules deliberately exclude.

Deliberately not proposing a patch — which trade-off is acceptable seems like a maintainer call, and there may be a preferred pattern for schema-compatibility shims that I am not aware of.

Environment

pyiceberg 0.11.1 (current release) and main, both affected
Cross-checked against the Iceberg spec, the Java reference implementation, and iceberg-cpp
Found while making iceberg-cpp read a pyiceberg-written table

Status checks

Still present on main. Verified against main on 2026-08-24: manifest.py:298 and :393 both still declare LongType(). Separately, no commit in the repository's history mentions equality_ids at all, and the recent commits touching manifest.py are unrelated (manifest cache configuration, min sequence number, ManifestEntry.snapshot_id setter, lint rules).

No prior report found, though this is the weaker of the two claims. Searching this repository for equality_ids returns 13 issues and PRs, all of which mention it incidentally (equality-delete tests, manifest entry status, the files metadata table) rather than discussing its declared type. Searches for "Cannot promote long to int" and "Cannot read Iceberg type" return nothing. It is entirely possible this has been raised somewhere I did not look — the dev mailing list or Slack, or phrased in a way my queries missed.

Related but distinct: #3744 (Metadata inspection APIs fail with struct.error after int→long / float→double type promotion) also concerns int/long handling, but it is about promotion of user table columns breaking inspect.* on pre-promotion data files. It does not involve the manifest-entry schema or equality_ids, so the two should not be conflated.

Willingness to contribute

I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time

Thanks for reporting the issue! This is definitely a serious bug. Lets see how we can best fix forward.

I took a look at this in depth. We are in fact using long type for equality_ids which contradicts the spec. The good thing is, we dont write equality_ids yet, so only the schema is wrong. And it looks like the java implementation can tolerate this difference in schema (which is why we didnt catch it in the first place). Cpp's validation is stricter so this was caught.

I think the best way to fix forward is

change equality_ids to int so we stop producing wrong schema
allow pyiceberg to tolerate equality_ids with long. this will allow us to read back avro files already written
implement rewrite_manifest so that users can correct the avro files. this will write the correct schema and allow Cpp reader to work

I considered adding this quirk to the spec's implementation note, but i think its best if we just handle this internally in pyiceberg and not burden every implementation with this caveat.

EDIT:
Here is the more-detailed proposed long-term fix:

Write equality_ids as int from now on.
int is the type required by the Iceberg specification, so this prevents PyIceberg from creating any more manifests with the wrong schema. Older PyIceberg versions can still read these new manifests.
Always read old manifests that use long for equality_ids.
Existing PyIceberg manifests may use long, so PyIceberg should retain permanent read support for this old format. This compatibility should apply only to data_file.equality_ids, rather than allowing long → int conversions everywhere and potentially hiding other schema problems.
Offer a maintenance operation to rewrite old manifests.
Users who need strict-reader compatibility, such as with iceberg-cpp, can use it to replace the legacy manifests used by the current table state with new ones that use equality_ids: int. The operation should write new manifest files and commit a new snapshot; it must not modify existing manifests in place because Iceberg manifests are immutable. Rewriting manifests referenced by every historical snapshot is a larger metadata migration and can be considered separately.

PyIceberg does not currently write equality delete files, so equality_ids is normally empty in the data itself. However, the incorrect type is still part of every manifest schema, which is why this affects all manifests.

We should not add a flag to continue writing long. It would keep producing invalid Iceberg metadata, while permanent read support preserves access to existing manifests.

Tolerate legacy long equality_ids in resolver and write int in manife…

Manifest: Use int for equality_ids in manifest schema per Iceberg spec (

Tolerate legacy long equality_ids in resolver and write int in manife…

@haoxu0

Thanks @haoxu0 for reporting this, and it is a serious issue indeed. I think the approach that @kevinjqliu is suggesting is the right direction. When we coerce int into long, we should check for overflows (and fail).

@kevinjqliu

@kevinjqliu @Fokko Thanks. Happy to help with the fix — just let me know.

Yeah, I think that'll be good to avoid writing these in the future. I think we should still give a deprecation warning on manifests with longs for equality_ids and eventually deprecate the functionality.

I was back and forth about whether to emit deprecation warning. I think the emitting a warning might be too noisy since all existing avro manifests from pyiceberg will trigger it

btw we're iterating on a fix in #3842

Write equality_ids as list<int> and keep reading legacy list<long> ma…

feat: add rewrite_if predicate to RewriteManifests

last step is to implement the rewrite_manifest procedure, i'll take a look at #3631

Manifest: Use int for equality_ids in manifest schema per Iceberg spec (

Navigation Menu

Repository navigation

equality_ids written as list<long>, but spec and all other implementations use list<int> #3840

Apache Iceberg version

Please describe the bug 🐞

Evidence

Impact

Reproducer

The obvious fix breaks backwards compatibility

Possible directions

Environment

Status checks

Willingness to contribute

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

equality_ids written as list<long>, but spec and all other implementations use list<int> #3840

Description

Apache Iceberg version

Please describe the bug 🐞

Evidence

Impact

Reproducer

The obvious fix breaks backwards compatibility

Possible directions

Environment

Status checks

Willingness to contribute

Activity

kevinjqliu commented on Aug 25, 2026

kevinjqliu commented on Aug 31, 2026

Fokko commented on Sep 3, 2026

haoxu0 commented on Sep 3, 2026

rambleraptor commented on Sep 4, 2026

kevinjqliu commented on Sep 4, 2026

kevinjqliu commented on Sep 4, 2026

kevinjqliu commented on Sep 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions