MEM-SQL memory method raises Text-to-SQL accuracy on new HARE benchmark

Text-to-SQL systems turn natural-language questions into SQL queries over increasingly complex databases. The authors of a new paper argue that widely used benchmarks evaluate each question on its own. That overlooks a common setting: earlier queries over the same database can supply joins or calculations for a new question.
To measure this, they introduce HARE (History-AwaRe Text-to-SQL BEnchmark), which tests the accuracy and cost of adapting historical SQL. HARE has 1,600 requests, derived from a fixed pool of 823 verified queries across four datasets. Its annotations record which components each request preserves and which edits it requires, so whole-query adaptation can be told apart from partial reuse.
The method is MEM-SQL (Memory-EMpowered Text-to-SQL). It keeps three kinds of stored material: whole-query templates, reusable SQL blocks, and mappings from domain concepts to schema elements. MEM-SQL uses a memory layer only if every requirement assigned to that layer has a match above the layer's threshold. With On-demand Schema Linking (OSL), it looks at the schema only when no layer qualifies or when the generator judges the retrieved evidence insufficient.
Across six language models on HARE, mean execution accuracy rises from 37.1% for Direct Generation without history to 50.0% for MEM-SQL. On average that comes with 76.6% fewer input tokens and 68.8% lower latency.
The analysis also finds a weak spot. Requests that reuse several verified queries remain a bottleneck, with accuracy falling from 56.7% to 29.7%. The drop is sharpest when reused components must be placed in a new query structure or combined with a shared join.
Finally, the authors applied OSL to three other memory designs. It consistently reduced generation cost, but its effect on accuracy depended on the memory design. Their conclusion: verified history can make generation both more accurate and cheaper, but its value depends on the evidence retained and on when the system consults the schema.
Key facts
- HARE is a new Text-to-SQL benchmark of 1,600 requests derived from a pool of 823 verified queries across four datasets, built to test reuse of historical SQL.
- MEM-SQL stores whole-query templates, reusable SQL blocks and concept-to-schema mappings, and consults the schema only when needed (On-demand Schema Linking).
- Across six language models, mean execution accuracy rises from 37.1% (Direct Generation without history) to 50.0% (MEM-SQL).
- MEM-SQL uses 76.6% fewer input tokens and has 68.8% lower latency on average.
- Requests that reuse several verified queries stay hard: accuracy falls from 56.7% to 29.7%.
Why it matters
Most Text-to-SQL benchmarks treat every question as independent, yet in practice a database often already has verified queries whose joins or calculations a new question could reuse. HARE targets that setting and measures both accuracy and cost. The reported result is that history can help on both counts: higher mean execution accuracy (37.1% to 50.0%) alongside 76.6% fewer input tokens and 68.8% lower latency on average.
Who it affects
The work is aimed at people building Text-to-SQL systems over databases where verified queries already exist, and at researchers who design benchmarks and memory mechanisms for such systems. The six language models and the four datasets are not named.
How to use it
The practical idea is to keep verified history in three forms (whole-query templates, reusable SQL blocks, and mappings from domain concepts to schema elements) and to use a layer only when every requirement assigned to it matches above a threshold. The schema is then consulted only when no layer qualifies or the generator finds the evidence insufficient. No release date, code or dataset availability is stated.
How solid is it
This is a preprint, and the numbers come from the authors' own benchmark, which they built alongside the method. The accuracy figures are means across six language models, and no per-model results are given. The 76.6% and 68.8% reductions are averages.
Risks and caveats
Requests that reuse several verified queries remain a bottleneck, with accuracy falling from 56.7% to 29.7%, most sharply when reused components must go into a new query structure or be combined with a shared join. The benefit of OSL on accuracy depends on the memory design: applied to three other designs it consistently cut generation cost, but accuracy gains varied. No figures are given for how much OSL changes accuracy in those three designs. The value of history also depends on which evidence is retained.
“Verified history can thus make generation both more accurate and cheaper, but its value depends on the evidence retained and when the system consults the schema.”
— Paper abstract, MEM-SQL