DataFusion-DuckLake

A DataFusion extension for reading and writing DuckLake catalogs. DuckLake is an integrated data lake and catalog format that stores metadata in a SQL database and data as Parquet files on disk or object storage.

The goal of this project is to make DuckLake a first-class, Arrow-native lakehouse format inside DataFusion.

This project is maintained by Hotdata with support from the community. Come talk to us on the Hotdata Discord.

• 📦 crates.io: https://crates.io/crates/datafusion-ducklake
• 📖 API docs: https://docs.rs/datafusion-ducklake
• 🧩 Feature & backend support: see COMPATIBILITY.md
• 💬 Project chat: DataFusion+DuckLake Discord — development and usage discussion
• 🧡 Meet the team: Hotdata Discord

---

Quick start

Add the crate:

cargo add datafusion-ducklake

The default build includes the SQLite catalog backend. Every other catalog backend, and write support for all of them, is opt-in via feature flags; applications configure their object store implementation directly. See COMPATIBILITY.md for the full matrix.

The DuckDB backend statically compiles DuckDB, which is by far the slowest dependency in this crate to build, so it is not a default:

# Cargo.toml — read (and write) DuckDB-backed catalogs
[dependencies.datafusion-ducklake]
version = "0.9"
features = ["duckdb-bundled"]   # add "write-duckdb" to write them

# Cargo.toml — read PostgreSQL catalogs
# (to write them too, use features = ["write-postgres"])
[dependencies.datafusion-ducklake]
version = "0.9"
default-features = false
features = ["metadata-postgres", "tls-rustls-aws-lc-rs"]

metadata-postgres, multicatalog-postgres, and write-postgres do not select a TLS provider. Plain local connections work without one. For TLS, also enable one of tls-native-tls, tls-rustls-aws-lc-rs, or tls-rustls-ring on datafusion-ducklake.

The examples below also use datafusion, object_store, and url directly — add them to your [dependencies] as well (this crate does not re-export them). The write example additionally uses sqlx (with its postgres and runtime-tokio features) to open the connection pool.

Run a query against an existing PostgreSQL catalog with the bundled example:

cargo run --example basic_query --features metadata-postgres -- \
  "postgresql://user:password@localhost:5432/database" "SELECT * FROM main.users"

Configure object_store directly for local, S3, or MinIO data files. Applications can instead register another DataFusion ObjectStore, such as an OpenDAL‑backed connector.

DataFusion enables local filesystem support. S3 and MinIO applications must enable object_store/aws themselves. That feature enables Ring through its HTTP client. If another dependency enables AWS‑LC, install the intended process‑wide Rustls CryptoProvider before creating TLS clients.

(The example also accepts DuckDB, SQLite, and MySQL connection strings with the matching metadata-* feature — see COMPATIBILITY.md.)

---

Reading a catalog

Register a DuckLakeCatalog with a SessionContext and query it with normal SQL as catalog.schema.table:

use datafusion::execution::runtime_env::RuntimeEnv;
use datafusion::prelude::*;
use datafusion_ducklake::{DuckLakeCatalog, PostgresMetadataProvider};
use object_store::ObjectStore;
use object_store::aws::AmazonS3Builder;
use std::sync::Arc;
use url::Url;

// (inside an async fn)
// Read metadata from a PostgreSQL catalog
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;

// Register object stores for any non-local data (S3 / MinIO)
let runtime = Arc::new(RuntimeEnv::default());
let s3: Arc<dyn ObjectStore> = Arc::new(
    AmazonS3Builder::new()
        .with_endpoint("http://localhost:9000") // MinIO endpoint
        .with_bucket_name("ducklake-data")
        .with_access_key_id("minioadmin")
        .with_secret_access_key("minioadmin")
        .with_region("us-west-2") // any region works for MinIO
        .with_allow_http(true)    // required for http:// endpoints
        .build()?,
);
runtime.register_object_store(&Url::parse("s3://ducklake-data/")?, s3);

let catalog = DuckLakeCatalog::new(provider)?;
let ctx = SessionContext::new_with_config_rt(
    SessionConfig::new().with_default_catalog_and_schema("ducklake", "main"),
    runtime,
);
DuckLakeCatalog::register(&ctx, "ducklake", catalog);

let df = ctx.sql("SELECT * FROM ducklake.main.my_table").await?;
df.show().await?;

Keeping a table's reads across queries

By default, every scan of a table reads the catalog again: the file list, the inlined rows and the inlined deletions. It also reads again each delete file that it reaches. A caller that keeps built tables across queries, for example in a cache keyed by table and snapshot, can keep these reads instead. Both memos are off by default.

A CatalogReadMemo keeps the catalog reads. Bind it to the provider once per query. The bind reads the catalog head (the newest snapshot id). The bound provider serves the reads while the head stays where the memo was filled. Do not keep a bound provider past its query:

use datafusion_ducklake::CatalogReadMemo;

let memo = CatalogReadMemo::new(CatalogReadMemo::DEFAULT_MAX_BYTES);
// For each query:
let provider = memo.bind(Arc::clone(&provider))?;
let table = cached_table.with_metadata_provider(Arc::new(provider));
// or, also for each query: DuckLakeCatalog::new(provider)?

The memo is not DuckLake behavior. Official DuckLake lists a scan's files again in every transaction. The memo is correct only because every commit that changes a listing also moves the head. These cases do not move the head:

• The low-level MetadataWriter API changes the catalog at a snapshot that create_snapshot made earlier. A memo filled between the two misses the change until the next commit.
• Expiring snapshots and cleanup commit nothing. Official DuckLake refuses a read at an expired snapshot. Without a memo, this crate returns no rows for it. With a memo, a fill kept before the expire returns the old rows until cleanup deletes their files, and then the read fails.

A commit to any table moves the head and empties the memo. On a catalog that commits between most queries, the memo costs more than it saves. See CatalogReadMemo for the details.

DuckLakeReadOptions keeps the delete positions that a table's scans read. A delete file is never changed after it is written, so this memo needs no check. The memo lives with the table, so it helps only a caller that keeps the table. A table that a catalog lookup builds for each statement starts with an empty memo:

use datafusion_ducklake::DuckLakeReadOptions;

let table = cached_table.with_read_options(DuckLakeReadOptions::memoized());

---

Writing a catalog

Writes of at most data_inlining_row_limit rows (default 10, as in official DuckLake) are stored in the catalog rather than in Parquet: SQL writes, DuckLakeTableWriter writes and streaming sessions (all but begin_write_with_embedded_rowid and begin_write_to_path), and keyed mutations. Set the limit to 0 to write every insert to Parquet. Deletes built by hand from data files (files_matching, resolve_positions) miss inlined rows; use DuckLakeTable::keyed_mutation, delete_positions or delete_rowids. See the inlined storage limits.

PostgreSQL has two writers, both behind the write-postgres feature:

• PostgresSingleCatalogMetadataWriter — the standard, spec-compliant single-catalog layout. Same catalog shape as the SQLite and MySQL writers, so the catalog is readable (and writable) by other DuckLake implementations including DuckDB's ducklake extension. SQL CREATE TABLE and INSERT INTO both work. Prefer this one.
• PostgresMetadataWriter — the experimental multi-catalog layout described in its own section, for hosting many catalogs in one database. Library-specific, not in the DuckLake spec, and no CTAS.

use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, PostgresMetadataProvider, PostgresSingleCatalogMetadataWriter,
};
use std::sync::Arc;

// Bootstrap the standard DuckLake tables and point the catalog at its data root
let writer = PostgresSingleCatalogMetadataWriter::new_with_init(
    "postgresql://user:pass@localhost:5432/db",
).await?;
writer.set_data_path("/abs/path/to/data")?;

// SQL planning requires the target schema to exist
let snapshot = writer.create_snapshot()?;
writer.get_or_create_schema("main", None, snapshot)?;
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), Arc::new(writer))?;
let ctx = SessionContext::new();
DuckLakeCatalog::register(&ctx, "ducklake", catalog);
ctx.sql("CREATE TABLE ducklake.main.events (id BIGINT)").await?.collect().await?;

Non-empty CREATE TABLE AS SELECT is rejected before publishing metadata. After creating an empty table, reopen the catalog before using INSERT INTO ... SELECT to populate it.

The multi-catalog path instead looks like this — tables are created through the writer API (no CTAS), then appended to with SQL:

use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, DuckLakeTableWriter, MulticatalogManager, MulticatalogProvider,
    PostgresMetadataWriter, initialize_multicatalog_schema,
};
use sqlx::postgres::PgPoolOptions;
use std::sync::Arc;

let pool = PgPoolOptions::new().connect("postgresql://user:pass@localhost:5432/db").await?;

// One-time: bootstrap the multi-catalog tables, then create a named catalog
initialize_multicatalog_schema(&pool).await?;
let catalog_id = MulticatalogManager::new(pool.clone()).create_catalog("my_catalog").await?;

// Create a table by writing the first batch through the table writer
let writer = Arc::new(PostgresMetadataWriter::with_pool(pool.clone(), catalog_id).await?);
writer.set_data_path("/abs/path/to/data")?;
let object_store: Arc<dyn object_store::ObjectStore> =
    Arc::new(object_store::local::LocalFileSystem::new());
let table_writer = DuckLakeTableWriter::new(writer.clone(), object_store)?;
table_writer.write_table("public", "events", &[batch]).await?; // `batch` is your RecordBatch

// Now append with SQL, reading the same catalog back through MulticatalogProvider
let provider = MulticatalogProvider::with_pool(pool.clone(), "my_catalog").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), writer)?;
let ctx = SessionContext::new();
DuckLakeCatalog::register(&ctx, "ducklake", catalog);
ctx.sql("INSERT INTO ducklake.public.events VALUES (1, 'a')").await?.collect().await?;
ctx.sql("SELECT count(*) FROM ducklake.public.events").await?.show().await?;

Writer output is configurable (Parquet compression, row-group sizing by row count and byte size). See DuckLakeTableWriter for the writer options.

Writing to a standard, single-catalog DuckLake store is supported through DuckdbMetadataWriter, SqliteMetadataWriter, MySqlMetadataWriter, and PostgresSingleCatalogMetadataWriter with their matching write-* features. See tests/it/sql_write_tests.rs and tests/it/postgres_single_catalog_write_tests.rs.

---

Partitioning

Partition a table by one or more columns (optionally through a transform) so that queries filtering on a partition column skip whole files:

// Declare the partition scheme *before* loading data, then INSERT as usual.
execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET PARTITIONED BY (region, year(sale_date))",
)
.await?;

Writes split rows into one Parquet file per partition value; reads then prune non-matching files automatically. Supported transforms are identity (the raw value) and year/month/day/hour; pruning currently applies to identity and year (month/day/hour are recorded but not yet used to skip files). Partitioned writes work on every writable backend — SQL INSERT/UPDATE, the low-level write entry points, the streaming session, compaction, and promote all honour the live spec — and read + pruning work on all backends. RESET PARTITIONED BY turns it off. See COMPATIBILITY.md and tests/it/partition_write_tests.rs.

---

Sort order

Order the rows inside each written file so that per-file statistics stay tight and range-filtered scans skip more:

execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET SORTED BY (sale_date DESC NULLS LAST)",
)
.await?;

The spec is recorded in the catalog (ducklake_sort_info / ducklake_sort_expression) and applied on insert, inlined rows included (with the table's stored sort_on_insert option off, only to rows written to Parquet while inlining is enabled, as official DuckLake does), to the inlined-data flush whatever sort_on_insert says, and to compaction output; rows are sorted before the partition split, so each per-partition file stays a sorted subsequence. UPDATE rewrites are not sorted, as in official DuckLake. SET SORTED BY records bare-column keys; a key DuckDB recorded as another expression orders inserts, flushes and compaction (evaluated with DataFusion). RESET SORTED BY turns it off.

---

Multi-catalog (PostgreSQL, experimental)

A single PostgreSQL metadata store can host multiple independent DuckLake catalogs — useful for multi-tenant deployments or keeping many logical lakehouses in one database.

⚠️ Experimental and library-specific. This multi-catalog layout is not part of the DuckLake specification and is not (yet) supported or accepted upstream. Catalogs written this way are read back only through this crate's MulticatalogProvider — they are not interchangeable with a standard single-catalog DuckLake store. The API and on-disk/in-catalog layout may change, so treat it as a preview. PostgreSQL writes no longer require this path — use PostgresSingleCatalogMetadataWriter for the spec-compliant layout.

• Create and manage catalogs with MulticatalogManager (feature write-postgres): initialize_multicatalog_schema bootstraps the shared tables, then create_catalog, and drop_table_in_catalog manage their contents.
• Read a specific catalog with MulticatalogProvider::with_pool(pool, "name") (feature multicatalog-postgres), which plugs into a DuckLakeCatalog like any other metadata provider.

See examples/multicatalog_write.rs for an end-to-end walkthrough (bootstrap → create catalogs → write → read back).

---

Maintenance

The maintenance API handles lakehouse upkeep from Rust: expiring old snapshots, cleaning up superseded files, and reclaiming orphaned files. The concrete entry points are backend-gated (write-sqlite / write-postgres). DROP TABLE is available through MetadataWriter. See examples/maintenance_demo.rs and examples/orphan_cleanup_demo.rs.

Compaction

Two explicit, triggered operations on DuckLakeTable rewrite a table's data files into a better physical layout without changing its logical rows:

• merge_adjacent_files(state, MergeOptions) coalesces several small files (of the same schema version) into fewer larger ones. A merged file spanning multiple origin snapshots is written as a DuckLake partial data file (preserving each row's original rowid and origin snapshot), so time travel and change feeds are unaffected.
• rewrite_data_files(state, RewriteOptions) rewrites a file whose deleted fraction exceeds a threshold (default 0.95), dropping its deleted rows.

Both commit atomically in one snapshot and coexist with concurrent appends; superseded files are scheduled for deletion and reclaimed later by cleanup_old_files. See examples/compaction_demo.rs.

---

Compatibility

For the full breakdown of catalog backends, object stores, types, capabilities, and current limitations, see COMPATIBILITY.md.

A few highlights worth knowing up front:

• Reads and writes work on DuckDB, SQLite, PostgreSQL, and MySQL. PostgreSQL supports the standard single-catalog layout, recommended by default, and an experimental multi-catalog layout.
• Object stores: local filesystem, S3-compatible (S3, MinIO), and any other object_store backend registered in the RuntimeEnv under the data_path scheme (gs://, az://, ...).
• A catalog from DuckLakeCatalog::new or with_writer reads the latest snapshot at each lookup. Register it with DuckLakeCatalog::register to keep each statement on one snapshot.
• Snapshots can be selected through DuckLakeCatalog (by id or timestamp) or per query with ducklake_table_at; DataFusion does not support AS OF syntax.
• Comments and tags are readable on every backend through the information_schema views object_tags and column_tags. SQLite and both PostgreSQL layouts can write them, for example with COMMENT ON through execute_ducklake_sql.
• Table partitioning: read + file pruning on all backends; partitioned writes on every writable backend.
• Filter pushdown reaches the catalog: per-column statistics narrow the file-listing query itself, so a selective scan no longer lists every file. A few type/backend combinations decline it and fall back to in-memory pruning — see COMPATIBILITY.md.
• Data inlining: rows inlined by DuckDB's ducklake extension are read and counted by COUNT(*) on every backend. Writes inline batches of at most data_inlining_row_limit rows (default 10, as in official DuckLake), through SQL and DuckLakeTableWriter alike, and UPDATE and DELETE inline small changes as official does. The single-catalog PostgreSQL writer does not inline: its SQL writes and staged write transactions go to Parquet, and it supports neither UPDATE nor DELETE. Row-lineage scans and change feeds include inlined rows — see COMPATIBILITY.md.
• Schema evolution: recursive list/struct/map columns, literal column defaults, and per-file map_by_name name mappings are honoured across scans, writes, and change feeds.
• Read-only DuckLake views are exposed across metadata backends.
• ORDER BY col LIMIT n prunes data files whose statistics cannot beat the running Top-N boundary, and an unfiltered COUNT(*)/MIN/MAX folds to a literal from catalog statistics.

---

Running the tests

The test fixtures are built by driving the real DuckLake extension through a bundled DuckDB, and those modules are gated on metadata-duckdb. Since that is not a default feature, a bare cargo test compiles only a small fraction of the suite:

cargo test --features duckdb-bundled            # the suite as CI runs it, minus Docker
cargo test --features "duckdb-bundled write-sqlite metadata-postgres metadata-mysql"

The PostgreSQL, MySQL, and multi-catalog suites need Docker (via testcontainers). Note that --all-features enables skip-tests-with-docker, which ignores those suites rather than running them, so a green --all-features run is not evidence that they passed.

Project status

This project is in alpha and evolving alongside DataFusion and DuckLake. APIs may change as core abstractions are refined. See CHANGELOG.md for release history. Feedback, issues, and contributions are welcome.