BM25 beats neural RAG methods once corpora scale up, study finds

Retrieval augmented generation (RAG) covers several different approaches: lexical retrieval, dense embedding search, graph based indexing, and agentic search that lets a model explore documents step by step. Researcher Pengyu Wang and co-authors note that these paradigms are usually tested on different benchmarks at a single corpus size, so it has been unclear how their accuracy and cost trade off as the underlying document collection grows. To answer that, the team ran a controlled study that varies corpus size across 28 strictly nested tiers spanning roughly a 450 fold range, while keeping the same questions and the same fixed set of relevant and adversarial documents unchanged at every tier. All methods were tested under one reader model and one judging protocol, measuring official accuracy, the tokens spent on construction and on querying, and latency.

The result is a scale dependent crossover rather than a single winner. File-System Agent, an agentic search method that explores documents sequentially, leads at the smallest shared tiers. But at that smallest tier, the bedrock corpus, it spends 39 times more query tokens than the other methods, and its advantage shrinks as the search space grows. At around 10 million corpus tokens, BM25 overtakes File-System Agent and stays ahead at every larger shared tier, with the margin approaching 20 points at full scale. BM25 also anchors the low cost end of the accuracy to cost frontier, since it needs no LLM based construction step at all. Dense retrieval stays efficient but is less accurate throughout. Graph based RAG runs into construction bottlenecks before it even reaches deployment scale, and even its more scalable variants remain below BM25 at every shared tier tested.

The authors conclude that as a corpus grows, global candidate ranking becomes increasingly favored over sequential exploration: lexical retrieval, in their words, is the strongest scalable default, while agentic reasoning works best applied after ranked discovery rather than used in place of it. The paper had drawn 37 points and 3 comments on Hugging Face Papers within 38 hours of posting, a modest early reception rather than broad peer validation.

Key facts

A controlled study varies corpus size across 28 strictly nested tiers spanning roughly 450x, holding the same questions and the same fixed set of relevant and adversarial documents constant, under one reader model and one judging protocol.
File-System Agent, an agentic search method, leads at the smallest shared tiers but spends 39 times more query tokens than the other methods at that bedrock tier, and its lead erodes as the corpus grows.
Around 10 million corpus tokens, BM25 overtakes File-System Agent and leads at every larger shared tier, with the gap approaching 20 points at full scale.
Dense retrieval stays efficient but less accurate; graph based RAG hits construction bottlenecks before deployment scale, and its scalable variants still trail BM25.
The authors conclude lexical retrieval (BM25) is the strongest scalable default for RAG, with agentic reasoning best used after ranked retrieval rather than instead of it.

Why it matters

RAG systems are usually benchmarked on different datasets at one fixed corpus size, so it has been unclear how each retrieval paradigm's accuracy and cost actually scale as a document collection grows. This study holds the questions and the core document set fixed while varying corpus size across a roughly 450 fold range, which isolates that scaling effect directly. The finding that a decades old lexical method, BM25, overtakes newer agentic and graph based approaches once the corpus is large enough is a concrete, practical result for anyone deciding how to architect a RAG pipeline meant to grow.

Who it affects

Engineers and researchers building or scaling retrieval augmented generation systems, particularly teams choosing between agentic search, dense embeddings, graph based indexing, and classic lexical retrieval for document collections that are expected to keep growing.

How to use it

The study points to a practical default: for large or growing corpora, plain BM25 retrieval performs best per unit of cost and needs no LLM based construction step. Agentic, step by step search such as File-System Agent is worth using at small scale or as a reasoning layer applied after BM25 has already narrowed the candidates, not as a replacement for ranked retrieval once the corpus is large.

How solid is it

The design is a controlled experiment: 28 strictly nested corpus size tiers spanning about 450x, the same fixed set of relevant and adversarial documents and the same questions at every tier, and a single reader model and judging protocol used across all methods, which isolates corpus size as the variable being tested. It measures official accuracy, construction and query tokens, and latency directly rather than by proxy. The paper had 37 points and 3 comments on Hugging Face Papers 38 hours after posting, which reflects early attention rather than independent replication.

Risks and caveats

The results come from one reader model, one judging protocol, and one fixed set of questions and documents, so they may not generalize to other domains, corpora, or reader models. The crossover point is specific to this setup, roughly 10 million corpus tokens, and the reported near 20 point margin applies only at the largest tested scale; at smaller scales the agentic method still leads. Graph based RAG's scalable variants are not eliminated by the study, only shown to trail BM25 at the shared tiers tested, and the paper does not claim BM25 wins at every possible scale or task.

“Corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.”

— the paper