Glonce

DuckDB's DuckLake extension reads and writes an open SQL and Parquet lakehouse format

infrastructureThe Glonce editorsHacker News🔥52Glonce rating52Impact45Novelty55Relevance75Surprise30Players40Credibility90
DuckDB's DuckLake extension reads and writes an open SQL and Parquet lakehouse format

The duckdb/ducklake GitHub repository holds the DuckLake extension for DuckDB. Its README describes DuckLake as an open Lakehouse format built on SQL and Parquet. The design splits the work in two: metadata lives in a catalog database, and the data itself lives in Parquet files. The extension lets DuckDB read and write data from DuckLake directly.

Installation is one command, INSTALL ducklake;. The latest development version comes from the nightly channel with FORCE INSTALL ducklake FROM core_nightly;. Once installed, a DuckLake database is attached with the ATTACH syntax, and tables can then be created, modified and queried with standard SQL.

The README walks through a short example. ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/') stores the metadata in a DuckDB database file called metadata.ducklake and the data as Parquet files in the file_path directory. The example creates a table my_table with an integer id and a varchar val, inserts two rows (1, 'Hello') and (2, 'World'), then runs an UPDATE that changes the second row's val to 'DuckLake'.

Two features follow from the same example. Time travel: querying the table with AT (VERSION => 2) returns the data as it was before the UPDATE, with 'World' still in the second row. Schema change: ALTER TABLE ... ADD COLUMN new_column VARCHAR adds a column that shows NULL for both existing rows. Change tracking: calling my_ducklake.table_changes('my_table', 2, 2) returns rows with snapshot_id, rowid and change_type columns; for snapshot 2 it lists the two inserts.

The rest of the README is for contributors. To build, run git submodule init and git submodule update --recursive, then make (or make GEN=ninja release to build with multiple cores). The submodules are pinned to the DuckDB version recorded in .github/duckdb-version, which CI builds against; make pull moves them to the tip of their branch instead, where the build can fail. The bundled shell runs from ./build/release/duckdb. External contributions are welcome, and the active development branch is main, which all contributions should target.

The test suite runs through ./build/release/test/unittest, either whole, per file, or by pattern. Other configurations cover running DuckDB core tests with DuckLake as the storage backend, running DuckLake tests with PostgreSQL as the catalog database (this requires a running PostgreSQL), running them with SQLite as the catalog database, and running tests with deletion vectors enabled.

Key facts

  • DuckLake is an open Lakehouse format built on SQL and Parquet: metadata sits in a catalog database, data sits in Parquet files.
  • The DuckLake extension lets DuckDB directly read and write DuckLake data; it installs with INSTALL ducklake; and the development version comes from core_nightly.
  • The README example shows standard SQL on an attached DuckLake database, time travel with AT (VERSION => 2), an ADD COLUMN schema change, and change tracking via table_changes.
  • The test suite can be run with PostgreSQL or SQLite as the catalog database, and with deletion vectors enabled.
  • Contributions should target the main branch; building with make pull can fail because it moves submodules to the tip of their branch.

Why it matters

Lakehouse formats keep table data in open files and track it with metadata. DuckLake takes a specific route: the metadata goes into an ordinary catalog database and the data stays in Parquet files, with SQL as the interface. Because the extension works inside DuckDB, the same engine can create, update and query these tables, and the README shows snapshots, time travel and change tracking working from plain SQL statements. This is data infrastructure rather than AI news.

Who it affects

Anyone who already uses DuckDB and wants lakehouse-style tables with versioned snapshots. Contributors to the DuckLake extension are also addressed directly: the README welcomes external contributions and points them to the main branch.

How to use it

Run INSTALL ducklake; in DuckDB. For the latest development build, run FORCE INSTALL ducklake FROM core_nightly;. Then attach a database, for example ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/'); and use ordinary SQL for CREATE TABLE, INSERT, UPDATE and ALTER TABLE. Query an older snapshot with AT (VERSION => 2), and inspect changes between snapshots with table_changes. To build from source, initialise and update the submodules, then run make; the resulting shell is ./build/release/duckdb. The README points to the DuckLake website and a Usage guide for more.

How solid is it

The source is the project's own README, so it describes intended behaviour and shows example output rather than independent results. The example is concrete and internally consistent, and the repository ships a test suite with several configurations: PostgreSQL and SQLite as catalog databases, DuckDB core tests with DuckLake as the storage backend, and deletion vectors enabled. No benchmarks, no comparison with other Lakehouse formats, and no release date or version number are given in the text.

Risks and caveats

The text does not say whether the extension is production-ready or stable. The development version from core_nightly and the main branch are the active development lines. Building against the tip of the submodule branches with make pull can fail; the pinned versions in .github/duckdb-version are what CI builds against. The text does not state which catalog databases are supported beyond DuckDB, and PostgreSQL and SQLite appear only as test configurations.