Pandas loses to Polars, DuckDB by up to 19x in 1-billion-row test

Eddie Atkinson, adapting a talk given at Latency Conference, argues that Pandas' own inefficiency is what pushes teams into distributed systems such as Spark, Databricks, Snowflake or Dask long before their workloads actually need one. The claim: those systems are mostly unnecessary, well marketed 'silver bullets' for a scale of data almost nobody has. To back this, the post leans on a 2024 Amazon paper, 'Why TPC is not enough: An analysis of the Amazon Redshift fleet', which published telemetry on real query run times and table sizes across Amazon's own Redshift fleet. Working from that data, and adding two assumptions of its own, an average Redshift row of 1KB and a cluster of 10 machines each pulling 8GB/s from S3, the post calculates that 94.68% of tables in the fleet hold under 100GB, and 86.9% of queries touch 80GB or less. It puts the practical ceiling for 'Medium Data' at around 100GB, a size single-machine tools can absorb, and credits Jordan Tigani of MotherDuck with a separate deep dive into the same dataset, while flagging that MotherDuck sells DuckDB hosting and 'some scepticism is perhaps warranted'.

The two alternatives proposed are Polars, a Rust-based DataFrame library with a Pandas-like API, and DuckDB, an in-process analytical database described as 'essentially SQLite for analytics'. To compare them with Pandas, the post uses the 1 Billion Row Challenge, a competition to compute the min, mean and max of a billion-row weather-station CSV as fast as possible; the fastest entry accepted in the original challenge ran in 1.5 seconds, in Java, on a bare-metal Hetzner AX161 server with 32 cores and 128GB of RAM. Atkinson reran the same workload on an AWS m7a.8xlarge cloud instance with the same 32 cores, 128GB of RAM and an AMD CPU, matching the original spec; the post explains the switch away from bare metal as down to being 'a serial procrastinator, a skinflint', combined with Hetzner's requirement that renters build a track record before it will rent out its larger boxes. Over 30 timed repetitions, after two warmup runs, the median results were stark: Pandas took 4 minutes 28 seconds and a peak 38.12GB of memory; Polars finished in 5.04 seconds using 18.02GB; DuckDB finished in 5.19 seconds using just 1.93GB. The post's own summary: Polars and DuckDB used 2x and 19x less memory than Pandas respectively, landing 'within striking distance' of a hand-tuned Java program. DuckDB's showing is called 'flawless', delivered 'with very little code and no tuning'; Polars' memory use is said to still look a little high, suspected to mean not all of its computation streamed. CPU use tells the same story: Pandas stayed near 113.0%, essentially one core, while Polars reached 3202.60% and DuckDB 3174.64%, evidence both parallelised the work across the machine's 32 cores automatically, something Pandas' single-threaded, eager execution does not do.

The same benchmark was repeated on ordinary laptop hardware, a Framework 13 with an Intel i5-1135G7, 8 cores and only 16GB of RAM, to see how the tools behave in a local development loop rather than in production. There, Pandas took 12 minutes 15 seconds and stayed near 110% CPU use, its 15.67GB working set forcing 21.02GB into swap on a machine with only 16GB of RAM to begin with. Polars finished in 39 seconds at up to 765.75% CPU, using 15.22GB of memory and barely touching swap (35.85MB); DuckDB finished in 47 seconds at up to 807.0% CPU, using only 546.87MB of memory and no swap at all, close to saturating all 8 cores. The post's verdict: Pandas 'runs like molasses and guzzles memory', while the other two stayed comfortably inside the laptop's limits.

Switching tools, the post argues, need not mean rewriting everything, because Polars and DuckDB both support Apache Arrow, the columnar in-memory format created by Wes McKinney, who is also Pandas' original creator. Pandas has supported Arrow since its 2.0 release in April 2023, though it does not create Arrow-backed DataFrames by default; a developer has to explicitly set dtype_backend to 'pyarrow'. Because all three tools can share Arrow-backed data without copying it in memory, the post says teams can mix and match, swapping in Polars or DuckDB for just the slow part of a pipeline.

To make that mix-and-match case concretely, a second, real-world example follows: about 3GB of NYC taxi trip data in monthly Parquet files, used to check whether cash payments fell as a share of all rides during what the post labels 'the pandemic' (2019 to 2022). On the same laptop, four combinations of reader and compute engine were timed. A pure-Pandas pipeline, reading and computing in Pandas, took a median 41.88 seconds; letting DuckDB do the reading while Pandas computed took 28.39 seconds; the reverse, Pandas reading and DuckDB computing, took 29.25 seconds; a pure-DuckDB pipeline, read and compute both in DuckDB, was fastest at 21.70 seconds and used only 216.76MB of memory against Pandas' 14.52GB. Cash usage did decline over the period checked, but the post is explicit that 'correlation is not causality' and says no conclusion should be drawn from that particular finding; it exists purely to exercise the benchmark.

The post closes by listing reasons the reader should not simply take its argument on faith: Atkinson is one person running open-source benchmarks that anyone can inspect and challenge; teams heavily invested in the Pandas ecosystem may still find the switching cost too high, though the post argues that bar has dropped since the talk was first given; and Pandas itself keeps improving, however slowly, given its central place in the ecosystem. The post also flags that a separate, more standard TPC-H benchmark it links to was run by Coiled, a company selling hosted Dask, a competitor to Polars and DuckDB for mindshare, and says that history calls for caution without claiming it makes the numbers wrong. On the remaining choice between Polars and DuckDB, no fixed answer is offered, only that data engineers tend to gravitate to DuckDB's SQL interface and software engineers to Polars' API, and that the decision should follow a team's own workload and preference. The bottom line: the real mistake is not underusing distributed systems, but reaching for one, and its complexity, purely because Pandas performs badly, when the actual odds of needing one in the long run are, in the post's words, 'pretty small'.

Key facts

Working from Amazon's 2024 paper on its own Redshift fleet, the post calculates that 94.68% of tables hold under 100GB and 86.9% of queries touch 80GB or less, the numbers the post uses to argue most 'Big Data' is really 'Medium Data'.
In the 1 Billion Row Challenge benchmark on a 32-core, 128GB AWS instance, Pandas took a median 4 minutes 28 seconds and 38.12GB of memory, against 5.04 seconds and 18.02GB for Polars and 5.19 seconds and 1.93GB for DuckDB.
On a 16GB-RAM laptop, Pandas needed 12 minutes 15 seconds and spilled 21.02GB into swap, while Polars finished in 39 seconds with 35.85MB of swap and DuckDB in 47 seconds with none.
On a roughly 3GB NYC taxi dataset, a pure-DuckDB read-and-compute pipeline finished in 21.70 seconds versus 41.88 seconds for pure Pandas, with mixed Pandas/DuckDB combinations landing at 28 to 29 seconds.
Author Eddie Atkinson recommends Polars or DuckDB for 'Medium Data' instead of jumping to Spark, Databricks, Snowflake or Dask, arguing genuine need for a distributed system is rare.

Why it matters

Once Pandas starts struggling, the usual next step is a full distributed system: Spark, Databricks, Snowflake or Dask. This post, based on a Latency Conference talk by Eddie Atkinson, argues that jump is usually premature and expensive. Amazon's own telemetry on its Redshift fleet suggests most real workloads are far smaller than 'Big Data': the post's reading of a 2024 Amazon paper puts 94.68% of Redshift tables under 100GB and 86.9% of queries at 80GB or less. If that holds, single-machine tools like Polars and DuckDB, not clusters, are the right fix for most teams hitting the 'Pandas cliff'.

Who it affects

Data engineers, analysts and developers running Pandas on datasets from tens of gigabytes up, especially anyone about to be pushed toward Spark, Databricks, Snowflake or Dask purely because Pandas has become slow or memory-hungry. The post also names, and flags for potential bias, MotherDuck, which sells DuckDB hosting, and Coiled, which sells hosted Dask, both cited for benchmark data relevant to the comparison.

How to use it

Polars and DuckDB both read and write Apache Arrow, the columnar format created by Wes McKinney, Pandas' original creator; Pandas itself has supported Arrow since version 2.0 in April 2023, though only if a developer sets dtype_backend to 'pyarrow', since that is not the default. Because all three can pass Arrow-backed data between each other without copying it, the suggested path is incremental: swap just the slow step of an existing Pandas pipeline, reading a file or running one aggregation, for the DuckDB or Polars equivalent, rather than rewriting everything at once. The post's own tests show mixed pipelines, one tool reading, another computing, landing close to a pure-DuckDB pipeline's speed and well ahead of pure Pandas.

How solid is it

The benchmark figures are the post's own, produced with a hand-rolled tool that samples CPU and memory every 50 milliseconds across 30 repetitions after two warmup runs; only the median per metric is reported, with no variance or range given. The headline Redshift percentages, 94.68% and 86.9%, rest on two assumptions the post supplies itself, an average row size of 1KB and a per-machine throughput of 8GB/s sourced to an 'admittedly outdated' benchmark, not on any threshold Amazon itself published. The post names Jordan Tigani of MotherDuck for a related analysis of the same Redshift dataset, and separately flags that a linked TPC-H comparison was run by Coiled, a hosted-Dask vendor competing with Polars and DuckDB for mindshare, urging scepticism of both, 'including of me'.

Risks and caveats

The post does not disclose its own author's employer or any conflict of interest, even though it explicitly raises that question about MotherDuck and Coiled. Teams heavily invested in the Pandas ecosystem may still face a real switching cost. The one added real-world example, that cash payments fell as a share of NYC taxi rides between 2019 and 2022, is flagged by the post itself as correlation rather than causation, with an explicit warning against drawing conclusions from it. And the closing question, Polars or DuckDB, is left open: the post says the answer depends on a team's workload, experience and preference, not on a fixed technical verdict.

“I posit that most workloads will never justify those systems, they are just well marketed 'silver bullets'.”

— Eddie Atkinson