[Bug] [Connector-V2] [Doris] LARGEINT is declared as DECIMAL(20,0), so non-Doris sinks fail on values over 20 digits

Search before asking

I had searched in the issues and found no similar issues.

What happened

The Doris connector maps LARGEINT to DecimalType(20, 0) (AbstractDorisTypeConverter#convert). Doris LARGEINT is a 128-bit signed integer, so its values range from -2^127 to 2^127-1, which is up to 39 digits. The declared precision of 20 is too small.

The source reader itself returns the full value; only the declared schema is wrong. Any downstream that trusts the schema therefore creates a column that is too narrow. When a value with more than 20 digits arrives, the job fails.

I reproduced this end to end with the 3.0.0 release against Doris 4.0.6. The mapping code is unchanged since #6354, so dev behaves the same. The source table has 5 rows:

id	value	digits
1	12345	5
2	99999999999999999999	20
3	100000000000000000000	21
4	170141183460469231731687303715884105727 (max)	39
5	-170141183460469231731687303715884105728 (min)	39 + sign

Job	Result
Doris → Console	All 5 values printed intact. The reader is fine.
Doris → Doris, auto-create	OK. `reconvert()` special-cases `sourceType = largeint` and recreates a `largeint` column.
Doris → MySQL 8 (JDBC), auto-create	Target column created as `decimal(20,0)`. The job fails on row 3: `Out of range value for column 'v'`, and 0 rows are written. Non-strict `sql_mode` fails the same way, because Connector/J reports the truncation.

Doris → Doris only works because of the sourceType special case in reconvert(). Every other sink gets DECIMAL(20,0). ARRAY<largeint> has the same problem (DorisTypeConverterV2, new DecimalArrayType(new DecimalType(20, 0))). MAX_DORIS_LARGEINT_TO_VARCHAR_LENGTH = 39 is defined in the converter but is not used.

Fix options. The full LARGEINT range needs 39 digits. DecimalType itself has no cap, but sink converters do: MySqlTypeConverter.MAX_PRECISION = 65, whereas Doris and Oracle cap at 38.

Map to DECIMAL(39,0). This is exact and keeps the numeric type. Sinks that allow 39 digits, such as MySQL, are correct. How sinks capped at 38 handle a 39-digit decimal still needs checking for each one.
Map to STRING with length 39. This is lossless everywhere, but downstream gets a string column instead of a number.

Either way, the existing sourceType = largeint special case for Doris → Doris would stay. I lean towards option 1, since it fixes the reproduced failure without changing the logical type, but I'd appreciate a maintainer's preference before I open a PR.

SeaTunnel Version

3.0.0 (also present on dev)

SeaTunnel Config

-- Doris 4.0.6
CREATE TABLE seatunnel_probe_tmp.li_src (id INT, v LARGEINT)
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1');
INSERT INTO seatunnel_probe_tmp.li_src VALUES
 (1, 12345), (2, 99999999999999999999), (3, 100000000000000000000),
 (4, 170141183460469231731687303715884105727), (5, -170141183460469231731687303715884105728);

env {
  parallelism = 1
  job.mode = "BATCH"
}
source {
  Doris {
    fenodes = "doris-fe:8030"
    query-port = 9030
    username = "root"
    password = ""
    database = "seatunnel_probe_tmp"
    table = "li_src"
  }
}
sink {
  Jdbc {
    url = "jdbc:mysql://mysql:3306/probe?useSSL=false&allowPublicKeyRetrieval=true"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "root"
    password = "***"
    generate_sink_sql = true
    database = "probe"
    table = "li_dst"
    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
  }
}

Running Command

./bin/seatunnel.sh --config doris_to_mysql.conf -m local

Error Exception

-- auto-created target table
CREATE TABLE `li_dst` (`id` int DEFAULT NULL, `v` decimal(20,0) DEFAULT NULL)

Caused by: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'v' at row 3
Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Out of range value for column 'v' at row 3
ErrorCode:[COMMON-10], ErrorDescription:[Flush data operation that in sink connector failed]

Zeta or Flink or Spark Version

Zeta (local mode)

Java or Scala Version

Java 8 (1.8.0_412)

Screenshots

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

One boundary detail for the string option: -2^127 is -170141183460469231731687303715884105728, which needs 40 characters including the minus sign. DECIMAL(39,0) is enough because decimal precision excludes the sign; a bounded VARCHAR fallback needs 40, not 39. The existing MAX_DORIS_LARGEINT_TO_VARCHAR_LENGTH = 39 comment also misses that sign, so it should not be reused as the fallback width unchanged.

I checked the converters on dev at b0f6feddf. The Oracle path answers part of the open downstream question: OracleTypeConverter.reconvert() clamps precision above 38 and adjusts scale, so DECIMAL(39,0) becomes NUMBER(38,0). It therefore still cannot represent the full LARGEINT range. The Doris scalar path preserves the original sourceType = LARGEINT, as you noted.

For the numeric fix, it would help to keep the source-schema correction separate from a promise of lossless conversion to every sink. Boundary coverage should include both signed extrema and the ARRAY<largeint> element precision; any bounded string fallback should check the negative extremum as well. Happy to review your patch once the mapping is agreed. This is source and numeric-boundary verification, not another live Doris/Oracle reproduction.

The source-schema defect is confirmed on current dev: AbstractDorisTypeConverter declares LARGEINT as DECIMAL(20,0), while the Doris-to-Doris special case only protects that one target path. A generic sink that relies on the logical schema can therefore receive an undersized decimal.

Please use the exact source representation, DECIMAL(39,0), for scalar LARGEINT and the corresponding array element type. Do not silently change the source schema to STRING or clamp it to 38 digits to accommodate a target-specific limit. Target connectors that cannot represent DECIMAL(39,0) need their own explicit compatibility behavior; this PR should not promise lossless delivery to every target.

Please include source-schema tests for 21-digit values and both signed 128-bit extrema, array coverage, and an end-to-end Doris-to-MySQL auto-create regression showing that the full positive range is no longer narrowed. Also retain the existing Doris-to-Doris source-type preservation. The 40-character requirement applies only to a future bounded-string fallback because the minus sign is not decimal precision. Please link the focused PR here.

@DanielLeens

@DanielLeens I implemented the DECIMAL(39,0) mapping as agreed, and before opening the PR I hit an engine-level limit that I think needs your call.

Spark can't represent DECIMAL(39,0). On my fork's CI run (https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/38510021504), doris-connector-it passes on Zeta and every Flink container, but fails on Spark 2.4.6 and 3.3.0, on both JDK 11 and 17:

org.apache.spark.sql.AnalysisException: decimal can only support precision up to 38
  at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:54)
  at org.apache.seatunnel.translation.spark.utils.TypeConverterUtils.convert(TypeConverterUtils.java:111)
  ...
  at org.apache.seatunnel.translation.spark.source.SeaTunnelSourceTable.schema(SeaTunnelSourceTable.java:100)

The failure happens while the Spark translation layer builds the source table schema, before any row is read. So with this change alone, any Spark job that reads a Doris table with a LARGEINT column fails, even when every value is small. Today the same job runs, and only values over 20 digits are a problem. On Spark this would be a regression, not a fix. The existing DorisIT and DorisMultiReadIT cases hit it too, not just the new Doris-to-MySQL E2E.

This isn't a target-connector limit, so the "targets handle their own compatibility" split doesn't quite cover it. The two options I see:

Handle it in the Spark translation layer (TypeConverterUtils): map a SeaTunnel decimal with precision > 38 to Spark StringType and convert values accordingly, or fail with an explicit error that names the column. That touches shared code used by every connector, so I'd do it as a separate PR that lands first, then the LARGEINT fix on top.
Keep the Doris fix as is and declare Spark unsupported for LARGEINT: disable the affected cases on Spark containers with @DisabledOnContainer and document the limitation. Smaller, but it knowingly breaks LARGEINT reads on Spark that work today.

I lean towards 1, since 2 trades a silent narrowing for a hard failure on Spark. I'd like your view before going either way. Clamping to 38 only on Spark is the other obvious workaround, but it goes against what you asked for here, so I've left it out.

Thanks for the detailed report and reproduction.

The source-side analysis holds: AbstractDorisTypeConverter#convert declares LARGEINT as DecimalType(20, 0), which cannot hold the 128-bit range, and the Doris → Doris path only works because of the sourceType = largeint special case in reconvert(). Mapping scalar LARGEINT and the ARRAY<largeint> element type to DECIMAL(39,0), while keeping the Doris → Doris preservation, still looks like the right source-schema correction. That does not promise lossless delivery to every sink; targets that cannot represent 39 digits should handle that explicitly in their own converters rather than by narrowing the source schema.

If you run into any engine- or sink-level precision limits while implementing this, please describe them in the PR (which component raises the error and at which stage) so we can decide how to handle them, rather than silently clamping. Please link the PR here once it is open.

[Fix][Connector-V2][Doris] Declare LARGEINT as DECIMAL(39, 0)

@SEZ9

Thanks @SEZ9! Opened #12787 with the DECIMAL(39, 0) mapping. As suggested, I didn't clamp anything: the PR description lists the Spark limit (TypeConverterUtils fails at source schema translation with decimal can only support precision up to 38, which affects every Doris table with a LARGEINT column on Spark 2.4/3.3) and two options for handling it. @DanielLeens could you take a look and pick one?

The DECIMAL(39,0) source-schema correction and the Spark limitation must be kept separate, but the limitation cannot be hidden by disabling Spark tests or by silently clamping to 38 digits. Either approach would leave a data-correctness regression unowned.

Please do not add an implicit DecimalType(>38) -> StringType fallback to the shared Spark translator in #12787. That is a cross-connector schema-contract change and needs its own compatibility proposal, tests, EN/ZH documentation, and release-note decision. Conversely, #12787 should not merge while it makes every Spark read of a Doris LARGEINT schema fail before data is read.

The next step is a dedicated Spark compatibility issue linked from #12787 that compares the viable explicit policies (a documented string representation, a clear preflight failure, or another preserved-schema approach) and covers the actual row-conversion path, not only schema construction. Keep #12787 focused on the Doris source representation and its Zeta/Flink coverage until that dependent Spark policy is agreed.

@DanielLeens

Thanks @DanielLeens, agreed on keeping the two separate. I opened #12814 for the Spark policy. It compares a documented string representation, a clear preflight failure, and a schema-preserving approach that reuses the existing TIMESTAMP_TZ metadata pattern, and lists the schema, row write/read and nested-type paths that any policy has to cover. It also notes that the same limit already affects other sources today (PostgreSQL NUMERIC(p > 38) passes through unchanged, MySQL DECIMAL(65) is clamped to DECIMAL(38, 18)).

#12787 now links to it, stays focused on the Doris source representation and Zeta/Flink coverage, and will not merge before the Spark policy is settled. In the meantime I'll address the incompatible-changes.md entry and the ARRAY<largeint> matching in #12787.

@zhengxiang378928908-code

@zhengxiang378928908-code Sorry for the delayed answer to your direct question: I'm not picking either of the two options as written. Option 1 (an implicit DecimalType(>38) -> StringType fallback in the shared Spark TypeConverterUtils) is a cross-connector schema-contract change and shouldn't ride along with the Doris fix, and option 2 (disabling the affected cases on Spark containers) would hide a regression that today's DecimalType(20,0) mapping doesn't have. The failure you reproduced in https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/38510021504 (decimal can only support precision up to 38 at source schema translation, on Spark 2.4.6 and 3.3.0) needs its own explicit policy, which is exactly what the dedicated Spark compatibility issue you opened afterwards is for. So the path you landed on in your last comment is the right one, and thanks for also noting that PostgreSQL NUMERIC(p > 38) and the MySQL DECIMAL(65) clamp are already affected — that confirms the policy belongs at the Spark translation layer, not in the Doris connector.

Where things stand from my side:

Agreed: the DECIMAL(39, 0) mapping for LARGEINT in AbstractDorisTypeConverter, with no clamping, is the source representation. Zeta and Flink passing on your fork is the coverage I expect the PR to carry.
Left in the PR: the incompatible-changes.md entry and the ARRAY<largeint> matching so ARRAY<LARGEINT> gets the same DECIMAL(39, 0) element type. Please ping me once both are pushed and I'll review.
Spark policy issue: it stays the blocker for merging the PR. Whichever of the three approaches you compared (documented string representation, preflight failure, or the schema-preserving TIMESTAMP_TZ-style metadata pattern) ends up preferred, please make sure the proposal covers the row write/read and nested-type paths, not only schema construction. If you already have a leaning among the three, say so there and I'll weigh in.

[Fix][Connector-V2][Doris] Match LARGEINT source types by element type

[Fix][Connector-V2][Doris] Declare LARGEINT as DECIMAL(39, 0)

[Fix][Connector-V2][Doris] Match LARGEINT source types by element type

[Fix][Connector-V2][Doris] Declare LARGEINT as DECIMAL(39, 0)

[Fix][Connector-V2][Doris] Match LARGEINT source types by element type

[Fix][Connector-V2][Doris] Declare LARGEINT as DECIMAL(39, 0)

[Fix][Connector-V2][Doris] Match LARGEINT source types by element type

Navigation Menu

Repository navigation

[Bug] [Connector-V2] [Doris] LARGEINT is declared as DECIMAL(20,0), so non-Doris sinks fail on values over 20 digits #12753

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Zeta or Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] [Connector-V2] [Doris] LARGEINT is declared as DECIMAL(20,0), so non-Doris sinks fail on values over 20 digits #12753

Description

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Zeta or Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

Activity

CryoThrust commented on Oct 13, 2026

DanielLeens commented on Oct 13, 2026

zhengxiang378928908-code commented on Oct 15, 2026

SEZ9 commented on Oct 15, 2026

zhengxiang378928908-code commented on Oct 15, 2026

DanielLeens commented on Oct 15, 2026

zhengxiang378928908-code commented on Oct 15, 2026

SEZ9 commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions