Glonce

DuckDB adds async I/O, up to 3.7x faster on S3

infrastructureThe Glonce editorsHacker News🔥54Glonce rating54Impact65Novelty58Relevance88Surprise50Players60Credibility87
DuckDB adds async I/O, up to 3.7x faster on S3

DuckDB is adding asynchronous I/O for reading Parquet and CSV files over the network, the kind of read that happens when querying data stored remotely, for example in S3. The feature targets a specific failure mode: queries where synchronous I/O cannot saturate the available network bandwidth, which the DuckDB team says is typical of EC2/S3 compute-storage setups. It can already be tried today in DuckDB's v2.0.0-dev preview builds, and it becomes the default behavior once the general v2.0 release ships, scheduled for fall 2026.

The change tracks a shift in how DuckDB is actually used. For most of its history the engine ran locally, reading straight off a machine's own SSD; pushing filters and projections down let it prune early and read only what a query needed, so I/O rarely became the bottleneck. That assumption broke down as DuckDB increasingly queries large datasets stored remotely, such as data lakes built on DuckLake, and, since May 2026, as it can run as a server over the Quack protocol. In a typical data lake setup, the data sits in blob storage such as S3 while a separate machine, an EC2 instance in the same region, processes it. Latency and bandwidth dominate there. If the database cannot issue enough concurrent requests to use the available bandwidth, worker threads sit idle, waiting on remote reads instead of doing work.

Mechanically: a Parquet scan splits into per-row-group jobs, and each job issues one or more byte-range fetch requests. With synchronous I/O, the worker thread that issues a fetch blocks until the bytes arrive before it can decode anything. DuckDB's async I/O separates the two jobs across two thread pools. REGULAR threads, by default one per CPU thread, do the real work: decoding, joins, aggregations, picking up I/O only when idle. ASYNC threads are dedicated mostly to blocking I/O, since a thread waiting on a network response uses almost no CPU; DuckDB defaults to four times the system's thread count for this pool, capped at 256 threads total. To keep the ASYNC threads busy, DuckDB reads ahead: fetch tasks for upcoming jobs are scheduled before a worker actually asks for that data, so fetching for the next job overlaps with decoding the current one instead of waiting its turn.

The read-ahead queue fills cooperatively. Any regular worker looking for scan work tops it up, to a limit set either by a fixed number of slots or by a memory budget; a new job, one row group for Parquet, one scan boundary for CSV, gets created and its fetch tasks go straight onto the ASYNC pool. A worker claims the oldest job in the queue: if its I/O already finished, it decodes immediately; if not, it parks that task and does other pipeline work until the job's last fetch task unblocks it. Because more in-flight prefetching holds more memory, a read_ahead_depth setting governs the budget. The default, -1, is unlimited but bounded dynamically by DuckDB's shared temporary memory manager, the same one that splits memory across concurrent joins, sorts and window operators; under heavy memory pressure from those operators, the queue can shrink to admitting one job at a time, close to synchronous behavior, and expand again once the pressure eases. A positive number fixes the cap at that many jobs regardless of memory pressure, and 0 turns read-ahead off entirely. Async I/O currently covers Parquet and uncompressed, seekable UTF-8 CSV files; DuckDB's native format and JSON are still to come, with no date given.

To measure the effect, the DuckDB team ran TPC-H Query 6 at scale factor 100 with the data on S3, comparing DuckDB v1.5.5, the latest stable release, against v2.0.0-dev with async I/O. The lineitem table in the dataset held 600,037,902 rows; the Parquet version of the file was about 22 GB, split into roughly 4,880 row groups averaging about 122,880 rows each. Compute ran on one EC2 r7i.16xlarge instance, 64 vCPUs and 512 GB of RAM, in the same region as the S3 bucket. External file caching was disabled, so every run read straight from S3, and each configuration ran five times, with the team reporting the mean.

The numbers moved together. Mean runtime dropped from 8.230 seconds on v1.5.5 to 2.844 seconds on v2.0.0-dev with async I/O left at its default, untuned settings, almost 3x faster. A second run, tuned specifically for that machine, read-ahead capped at 64 in-flight jobs, async_threads set to 48, and adjusted HTTP retry settings (http_retries=8, http_retry_wait_ms=50, http_retry_backoff=2), finished in 2.227 seconds: 21.7% faster than the untuned async run, and about 3.7x faster than v1.5.5. Network throughput told the same story. On the test's 25 Gbit/s link, v1.5.5 stayed around 5 Gbit/s, since its synchronous reads could not keep enough requests in flight; the untuned async run used the link far more effectively, approaching and at times touching its limit; the tuned run, with fewer but 'hotter' connections and cheap retries, kept variance to a minimum and stayed close to full saturation throughout. Throughput was measured by sampling the network interface's received-byte counter every 50 milliseconds.

One overhead shows up in every run regardless of tuning: a few hundred milliseconds pass before the first bump in network traffic, taken up by opening the DuckDB connection, the initial TLS handshake and opening the file, followed by another gap for downloading and processing the Parquet file's footer before the main transfer begins. The team says this startup cost is something it still intends to investigate and optimize before the v2.0 release. The post also states it will cover CSV benchmarks alongside the Parquet ones, but the captured source text cuts off mid-sentence during the network-throughput discussion, before any CSV results or the post's conclusion appear, so those numbers are not available here.

Key facts

  • DuckDB is adding asynchronous I/O for remote Parquet and CSV reads, usable now in v2.0.0-dev preview builds and becoming the default once v2.0 ships in fall 2026.
  • The engine splits work across two thread pools: REGULAR workers that decode, join and aggregate, and ASYNC workers dedicated to blocking I/O, sized by default at four times the system's thread count and capped at 256, with a read-ahead queue that prefetches upcoming jobs to keep both busy.
  • On a TPC-H Query 6 benchmark over S3 (SF100 scale, a 600,037,902-row lineitem table, one EC2 r7i.16xlarge instance), mean runtime fell from 8.230 seconds on DuckDB v1.5.5 to 2.844 seconds with default async settings, almost 3x faster, and to 2.227 seconds with settings tuned for the machine, about 3.7x faster and 21.7% faster than the untuned async run.
  • Network throughput improved to match: v1.5.5 topped out around 5 Gbit/s on the benchmark's 25 Gbit/s link, while the tuned async run stayed close to fully saturating it.
  • A read_ahead_depth setting controls how much data gets prefetched, since holding too much in flight risks out-of-memory issues if decoding lags behind the network; it defaults to an unlimited depth bounded by DuckDB's shared memory manager but can be fixed to a set number of jobs or turned off entirely.

Why it matters

DuckDB built its reputation on being a fast, local database engine, reading straight off a machine's own SSD, where pushing filters and projections down let it prune early and rarely wait on I/O. That no longer holds now that DuckDB is increasingly used against data stored remotely: large-scale data lakes such as DuckLake, and, since May 2026, DuckDB running as a server over its own Quack protocol. In that setup, data sits in S3 while a separate machine, typically an EC2 instance in the same region, processes it, and if the engine cannot issue enough concurrent requests to use the available bandwidth, worker threads sit idle waiting on reads instead of doing work. Asynchronous I/O is DuckDB's answer: separate the threads that fetch data from the threads that decode and process it, and start fetching ahead of demand so the two overlap. The team's own benchmark shows the gap is not marginal: on a 25 Gbit/s link, DuckDB v1.5.5's synchronous reads used only around 5 Gbit/s of it, while the tuned async version stayed close to full saturation.

Who it affects

The change matters most to people running DuckDB against data that lives somewhere other than local disk: data lakes on S3 or similar blob storage, DuckLake users, and anyone using DuckDB's server mode over the Quack protocol, introduced in May 2026. Workloads that already read from a local SSD see little from this, since that path was already low-latency and high-bandwidth; the bottleneck this fixes is specific to network-bound, EC2-and-S3-style compute-storage setups. The post names no specific companies or production deployments already relying on the feature, only DuckDB's own internal benchmark; it describes a change to the engine itself, not an adoption story.

How to use it

Asynchronous I/O can be tried today in DuckDB's v2.0.0-dev preview builds and becomes the default once the general v2.0 release ships, scheduled for fall 2026. It currently covers Parquet files and uncompressed, seekable UTF-8 CSV files; support for DuckDB's native format and for JSON is still to come, with no date given. The main control is the read_ahead_depth setting, changed with a SET statement, for example SET read_ahead_depth = 5;. Its default, -1, is unlimited but dynamically bounded by DuckDB's shared temporary memory manager; a positive number fixes the cap at that many jobs regardless of memory pressure; and 0 turns read-ahead off, so each scan schedules I/O only for its own job. The team's fastest benchmark run also tuned async_threads, http_retries, http_retry_wait_ms and http_retry_backoff, though the post does not say whether those specific values become new defaults. The source gives no pricing or licensing figures.

How solid is it

This is DuckDB's own engineering blog, written throughout in first person plural with no individual author named in the captured text. The benchmark is self-run and self-compared: DuckDB's team measured its own upcoming release against its own previous stable release, v1.5.5, on one EC2 instance of their choosing, running one query, TPC-H Query 6, at one scale factor, SF100, averaged over five executions. That is a disclosed methodology, not a bare claim: instance type, dataset size, row counts, cache settings and iteration count are all stated. It is not independently reproduced, though, and the standout 3.7x figure came from a configuration hand-tuned for that specific machine and network, not from the untuned default, which reached almost 3x on its own. The post also says it will benchmark CSV files alongside Parquet, but the captured source text cuts off mid-sentence during the network-throughput discussion, before any CSV results or the post's conclusion appear, so those figures cannot be reported here. On Hacker News, the post had drawn 125 points from 8 comments, a strong score for a specialized systems write-up with comparatively little discussion attached.

Risks and caveats

The feature is still pre-release: general availability is scheduled for fall 2026, and what exists now is a v2.0.0-dev preview build, so settings and defaults could still change before then. Prefetching trades memory for speed, and the DuckDB team flags the downside directly: if decoding is slower than the network, prefetched data can accumulate and risk out-of-memory errors, which is why the default read-ahead budget is tied to DuckDB's shared memory manager. The tradeoff cuts the other way too, since heavy memory pressure from other operators can shrink the queue to one job at a time, and the speed gains shrink with it. Format coverage is partial for now, with DuckDB's native format and JSON not yet supported. The headline 3.7x figure required manual tuning of read-ahead depth and HTTP retry settings for one specific machine and network; the default, untuned configuration reached a smaller, still substantial, almost 3x. The team also flags its own unexplained startup latency, a few hundred milliseconds of connection and file-opening overhead before a query's data actually starts moving, as something still to be optimized before release.

“Asynchronous I/O should have the largest effect when the latency of synchronous requests prevents us from using the available remote bandwidth.”

— the DuckDB team, in the post