Glonce

LASAGNA study of 450 RAG setups finds strong techniques often fail to stack

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact50Novelty55Relevance95Surprise45Players10Credibility80
LASAGNA study of 450 RAG setups finds strong techniques often fail to stack

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine many parts: embedding, chunking, retrieval, reranking, query augmentation, context compression and agentic verification. Yet, the paper says, these techniques are typically evaluated one at a time. Three questions stay open: do isolated gains survive when techniques are combined, do retrieval quality and answer faithfulness favour the same design choices, and do conclusions from small corpora hold at large scale?

To answer them, the authors pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. They test 450 configurations across eight benchmarks, with corpora ranging from 5K to 200K documents.

Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose. In the example given, three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and for faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale.

From this the authors conclude that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. They call on researchers to do so and supply LASAGNA to simplify these experiments.

Key facts

  • The paper tests 450 RAG pipeline configurations across eight benchmarks, with corpora from 5K to 200K documents.
  • Three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together.
  • Retrieval and faithfulness gains stack differently, and optimizing one can hurt the other.
  • The best combination of techniques changes as the corpus grows; a small-corpus winner may not stay best at scale.
  • The authors release LASAGNA, an open-source framework for controlled RAG evaluation, and urge testing new techniques with the full stack.

Why it matters

RAG pipelines now stack many components, and new techniques are usually shown to help one at a time. This paper's central claim is that such isolated gains are not reliable predictors of realistic performance. The sharpest example: three retrieval techniques whose individual gains predict +47% together deliver +17%. If that holds more widely, a technique's headline number says less about a real system than it appears to.

Who it affects

Researchers who propose new RAG techniques are the paper's direct target: the authors call on them to test with the full surrounding stack. Teams building RAG pipelines are also affected, since the findings concern design choices across embedding, chunking, retrieval, reranking, query augmentation, context compression and agentic verification, and how those choices behave as a corpus grows from 5K to 200K documents.

How to use it

LASAGNA is described as an open-source framework for controlled RAG evaluation, supplied to simplify experiments that test techniques together rather than alone. The paper's practical advice is to test new RAG techniques with the full surrounding stack, since gains measured in isolation are not reliable predictors of realistic performance.

How solid is it

This is a preprint abstract-level account. The scale is substantial: 450 configurations, eight benchmarks, corpora from 5K to 200K documents, and three hypotheses declared in advance. The summary does not name the metric or baseline behind the +47% and +17% figures, the three retrieval techniques combined, or the eight benchmarks, so the headline example cannot be checked from this text alone.

Risks and caveats

The +47% versus +17% result is one example of non-composition, not a general rate. The source does not say which compositions win at which corpus size, only that the best one changes, and the corpus sizes tested stop at 200K documents. The statement that the small-corpus winner "may not remain best" is itself hedged, so the findings are best read as a warning against assuming isolated gains carry over.