Repository navigation
feat: CREATE TABLE in a Unity Catalog schema now creates a catalog-managed Delta table. - #139
Merged
Merged
Conversation
benfleis
force-pushed
the
feat/uc-create-table
branch
2 times, most recently
from
October 8, 2026 12:37
f32272f to
8493c51
Compare
A managed table is created in three steps, as UC's Delta protocol has it: ask UC to stage a table, write its version 0 through UC's committer, then register what was written. - `createStagingTable` gives the location, the table id and what the table has to be. Its required properties, and its required protocol features as `delta.feature.<name> = supported`, become part of the CREATE the delta catalog runs: without `catalogManaged` among them kernel refuses to commit through a catalog committer at all. - The staging location is attached as a delta catalog with `parent_commit`, so version 0 takes the same path every later commit does, and is detached again whatever happens. - `createTable` describes the table from its own log -- schema, partition columns, protocol, configuration and the commit timestamp -- so the catalog cannot end up describing something other than what was committed. - Credentials for the staging location are vended and scoped to it, except for local storage, which needs none. UC's suggested properties are not applied yet.
…applied A suggestion is UC saying how it would write the table, so a created table now carries them -- with two exceptions. Column mapping is skipped: the writer maps top-level, unpartitioned columns only, so taking that suggestion by default would leave nested and partitioned tables impossible to create. And a `delta.*` key kernel does not accept at creation is skipped rather than refused, which keeps a suggestion UC adds later from breaking CREATE; each skip is logged. A table created against the OSS server now carries row tracking, deletion vectors, in-commit timestamps and v2 checkpoints, and reads and writes through the catalog.
The body creates a managed table, asserts the three calls the flow is made of (stage, write version 0, register), then inserts, reads, and reads again from a connection that only ever saw the catalog's copy. Two departures from the bodies next to it, both because the table must not exist when the test starts: - No `@requires`: that provisions a table. The driver picks a name instead, one no earlier run used, since DROP TABLE is not supported yet and a fixed name would collide with the run before. It deletes the table afterwards through UC's REST API. - The endpoint comes from `UC_ENDPOINT` rather than being written into the body, so a run that reaches the server by another address can say so. It defaults to what the other bodies hardcode.
…CREATE Four defects on the path a CREATE TABLE in a Unity Catalog schema takes, found by a conformance review against the managed-tables spec. - Nested columns reached the catalog in the Delta log's spelling, so a list or map was registered without its element type and refused. UC's API spells multi-word keys with dashes, and a schema's only multi-word keys are the parameters of array and map -- that is the whole of the difference. - A CREATE invalidated the schema's cache by destroying its entries while the internal Delta databases those entries own stayed attached, so any table the session had already read became unusable. The set is marked stale instead. - A taken name arrived as an HTTP 409. The staging call classifies it, so IF NOT EXISTS is a no-op and a bare CREATE reports DuckDB's own "already exists". The name is deliberately not checked locally first: the catalog is the only authority, it rejects a duplicate before allocating anything, and a cached answer could turn IF NOT EXISTS into a silent no-op for a table that is already gone. - A registration whose reply was lost reported failure for a table that does exist. The spec's recovery is to load the name and match its table-uuid against the staged id; a match now completes the CREATE with a warning naming the original error. Classifying the conflict needed the transport to hand one chosen failure status back as a response -- until now every UC error body sat unreachable behind an exception.
Two sections on the catalog-managed CREATE body. A list, a map and a nested struct, created on an attachment that has already read another table, so one section covers both the wire spelling of nested types and a CREATE under live entries. Then the same name a second time, with IF NOT EXISTS and without. The driver names the extra table and deletes both afterwards, since DROP is still unsupported and a fixed name would collide with the run before.
The cases call `SerializeFiltersToIRC` and the position-delete filter directly, and link_extension_libraries contributes only the statically linked extensions -- so in a build where unity_catalog is loadable, which is every build outside the CI container, the runner failed to link. Naming the archive resolves it there and changes nothing where the extension is already static.
They are, in the loop below the one this comment sits on. The sentence described the state before that loop existed; the policy it got wrong, down to the column-mapping exception, is already spelled out where the suggestions are read.
Both sets were written into the same option map, required first, so a key UC named in both went into version 0 with the value it merely prefers. The catalog is then told what the log says, which is how the required value was meant to be guaranteed, and a server that validates its own requirement answers 400 -- after version 0 is on disk and with a staging table left registered. Writing the required set last costs nothing: UC 0.5.1 names no key in both, so this only decides what happens when one does. The protocol features are unaffected either way, since `delta.feature.*` is not a key the suggestion loop will apply.
benfleis
force-pushed
the
feat/uc-create-table
branch
from
October 13, 2026 21:41
a9c36c9 to
a3af4ea
Compare
…eat/uc-create-table
- IF NOT EXISTS is a no-op only when the name is taken. Any other catalog error raised while staging used to be swallowed the same way, leaving the user believing a table existed. - A 409 from the staging call that is not UC's error body (a proxy in front of the server, say) is reported as the HTTP failure it is, instead of as a missing `table-id` field. - When version 0 cannot be written or registered, a failure to detach the staging database is logged rather than thrown, so the user sees the error that stopped the create. - Property values UC sends as JSON numbers or booleans are kept as their text, because Delta stores every property as a string. They used to be dropped without a word, which for a required property meant writing version 0 without it. Only null still means "no opinion"; an object or array is refused. Comments now cite the managed-tables protocol by name rather than a file outside the repository, show the 409 body UC actually sends, and state once, where the log is read, why UC is told what version 0 says.
The runner already links the extension's archive by name, so naming it again added nothing.
…succeeds - IF NOT EXISTS now has to be decided by the catalog: the name is staged a second time and nothing is registered. Unchanged columns alone would also pass for a local check of the loaded tables, which skips the catalog altogether. - The column-mapping suggestion is skipped on purpose, and the log line that says so is asserted. - Read-only attachments, OR REPLACE, CREATE AS SELECT, PARTITIONED BY, NOT NULL, PRIMARY KEY and TIME are refused, each with the message a user sees. - The driver takes the server's address from the fixture, so attaching to a running server (--existing-service) works without setting UC_ENDPOINT.
The create path treats a failed register call as a success when the name now loads with the id it was staged under, and as the real error otherwise. That is the only code that turns a reported failure into a reported success, and nothing exercised it. Against the mock catalog: - a 5xx and a connection closed before the reply, with the table registered, both count as created; - a table that is not registered, or is registered under another id, reports the register call's own error rather than the lookup made after it. The mock gains a hook that answers a request with any status, or closes the connection unanswered.
… server UC spells the parameters of array and map with dashes and silently drops keys it does not know, so a camelCase key reaches it as a missing field. The conversion was file-local, which left a live server per nested shape as the only way to test it. It is now reachable from the Catch runner, and the cases cover flat columns, array and map parameters, a map of a struct of an array of decimals, field names that happen to look like keys, and a schema that does not parse.
UC registers the table during the CREATE itself, so rolling back the transaction leaves it in place, empty and usable. Spark offers no multi-statement transactions to compare with, so this is the extension's own behaviour; the test makes a change to it a deliberate one.
benfleis
changed the title
CREATE TABLE in a Unity Catalog schema now creates a catalog-managed Delta table.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CREATE TABLEin a Unity Catalog schema now creates a catalog-managed Delta table.A create: stages the table with UC, writes version 0 through the catalog's committer, then registers the
table as version 0 describes it, so the catalog always matches the log.
except column mapping, which would break nested and partitioned tables.
IF NOT EXISTSis decided by the catalog, not by tables already loaded.OR REPLACE,AS SELECT,PARTITIONED BY, table options,constraints, and types Delta cannot store.
PARTITIONED BYandlocationfollow in the stacked PR.Tests: OSS UC end-to-end (
test/oss_local/table-cmt/create_table), a mock catalog for the recoverypath, and Catch cases for the nested-type wire format.