Repository navigation
Merge v1.5-variegata into main - #149
Merged
benfleis merged 20 commits intoOct 14, 2026
Merged
Conversation
Previously tests relied on hand-generated source tables. Those have changes, and tests broke. Update tests to match existing sources, and slightly tweak test runner Makefile for flexibility with uv. (I already manually updated source tables to match.)
…kdb#85) fix duckdb handling of ratified commits in managed tables In order to handle visible ratified commits within a UC managed table, delta-kernel requires the `max_catalog_version` parameter. Get this, and pass it through to delta+kernel. Depends on duckdb/duckdb-delta#319 New test validates against both spark- and duckdb- committed tables. Also: - update delta git ref - bump MainDistributionPipeline config
* fix commit backfilling Backfilling now fully implemented, with one shortcoming. Backfills now occur at attach time; all outstanding ratified commits will be backfilled, and signalled as moved. When a commit occurs, its ratified commit will remain in the backfill list, to be moved and signalled at the _next_ attach. This deferred strategy is required with current DuckDB due to lack of credentials post commit. Addressing this cleanly likely requires some changes to DuckDB's credential handling, and remains future work. * format * cleanups - DRY the backfill function, and use NULL_IF_EXISTS
…review (duckdb#89) Bumps to v1.5.4 so that we can push a v1.5.4 variant with version=`protocol_preview`
Also backport uc-api logging, and hand-tweak tests to avoid read-write pollution (which is structurally fixed in head)
Update UC catalog-managed path to delta.yaml v1 protocol
Migrate the CMT commit path off the legacy delta/preview/commits endpoints
to delta.yaml v1, and harden the surrounding commit/backfill state.
API
- GetCommits/PostCommit -> LoadTable/UpdateTable; add-commit now carries
assert-table-uuid + assert-etag requirements.
- Split table credentials: managed via delta/v1 .../credentials, external via
temporary-table-credentials (keyed on IsCatalogManaged, was IsCCV2).
- Parse v1 {"error":{...}}; surface HTTP status + body on failure.
- Fix etag JSON escaping (double-quoted -> HTTP 400).
Backfill
- Remove the deferred pending-backfill queue; backfill now runs synchronously
at attach from LoadTable's returned commits.
- Skip backfill on read-only attaches (a RO attach must not mutate _delta_log/);
reads still resolve via log_tail + max_catalog_version.
- latest-table-version is optional per spec: fall back to max commit version
(0 would hide staged commits); assert + log incoherent responses.
Concurrency
- Fuse etag + backfilled_version into MutexProtected<CommitState>, closing a
torn-read/unlocked-read race in the commit callback (new uc_mutex_protected.hpp).
Tests / infra
- scripts/oss_uc_local/seed_tables seeds duck.{managed,external}.id_name_{ro,rw}.
- New managed_catalog.test (CMT read+write); uc_catalog_write.test rewritten to
assert UpdateTable is NOT called for plain EXTERNAL tables.
- Databricks: new catalog_managed_delta_read.test + Spark/DuckDB staged-commit
fixtures; write_catalog_managed.test backfill assertions; option-name updates.
- Semantic request/response logging (uc_logging.hpp); tests assert on log lines.
- test/sql/README.md tracks remaining gaps (OSS+S3, async, races).
ParseColumnDefinition parsed type_precision/type_scale with TryGetNumFromObject's default fail_on_missing=true, but the shared getter treats a JSON null the same as absent -- so a null threw "Invalid field found while parsing field: type_precision" while binding the /tables response, before any Delta file is touched. Delta-Spark reports type_precision/type_scale as JSON null for non-DECIMAL columns (only DECIMAL carries a real value); UC CLI-created tables write 0 and don't trigger it. The crash fired on any listing/read of such a catalog (SHOW ALL TABLES / DESCRIBE / SELECT). Note: actual testing is difficult in current repository, but was done locally against not-yet-landed testing framework. That test is contained here, and will harmlessly skip until the framework lands.
Current OSS UC tests hard-ref 'http://localhost:8080' (or similar). Replace with env var UC_TEST_ENDPOINT to allow flexbility here.
As fixed in main / duckdb#113 the test space had to be rebuilt. This adjusts refs to work against the rebuild, and adds a one-off script to manually recreate the `source` schema needed by some tests.
fix: backport the Databricks interop fixes to v1.5 Backports: - duckdb#82 - "Fix: unknown type timestamp_ntz" - duckdb#96 - "rename CommitState -> UCCommitState, avoid conflict w/ duckdb's" - duckdb#107 - "test: validate the delta column-mapping and partition-value write fixes through UC" - duckdb#110 - "fix: handle schema divergence, make delta log SoT" - duckdb#115 - "test: offline timestamp_ntz coverage; clean up schema-handling tests" - duckdb#122 - "fix: Keep the internal Delta catalog a transaction has already read" - duckdb#125 - "fix: do not crash on duckdb_indexes()" Adjustments: - v1.5 has no python test driver, so their tests are rewritten as sqllogictest against a local OSS server. duckdb#107's char/varchar write check runs against Databricks instead. - Time travel through the UC catalog path gets its own test.
[AUTOMATED_BUMP] Bump to v1.5.6
Every v1.5 fix on this branch already landed on main, in its own form or as the original the branch backported, so the merge only connects the histories.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connects the v1.5 release branch's history to main, resolving every conflict to main's side: each fix on
v1.5-variegataalready landed on main, either in its own form or as the original the branch backported. The merged tree is identical to main's.Resolves duckdblabs/duckdb-internal#11382.