Glonce

RENDER benchmark shows memory format alone swings LLM scores by up to 72 points

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact70Novelty75Relevance95Surprise60Players25Credibility80

Researchers introduce RENDER, a benchmark control built to answer a question that memory and retrieval-augmented generation (RAG) evaluations usually skip: does it matter how the answering model's input is packaged, separate from what information it contains? Most evaluations treat the rendering of a conversation history, whether as a memory entry, a summary, a typed record, or a raw excerpt, as an incidental implementation detail. RENDER fixes the underlying conversation and varies only the reader-facing artifact, so any score difference can be traced to formatting rather than content. The method combines a five-level packet ladder, which localizes exactly when answer-bearing content enters the model's input, with deterministic templates that approximate four real-world styles: ChatGPT-style memory entries, LangChain-style summaries, MemGPT-style typed records, and raw conversation excerpts. The team ran this design on 500 questions from the LongMemEval benchmark across nine language models. Packets built to a matched information budget and resolved into a clean form beat recency-truncated raw dialogue, meaning the tail end of a raw conversation cut to fit a length limit, by 42.4 to 72.6 points. Among the four deployed-style templates alone, the gap between the best-performing and worst-performing format for a given model ranged from 24.6 to 48.8 points. Under the paper's primary scoring method, ChatGPT-style entries produced higher point estimates than raw conversation on 7 of the 9 models tested. Rescoring the same outputs with a judge model preserved the overall positive effect of well-formatted packets, though significance varied model by model rather than holding uniformly. The most striking single result: three of the nine models scored 0 percent when the same facts were presented as formal ledger-style packets, but answered correctly 45.4 to 53.4 percent of the time when the identical facts appeared as natural-language memory entries. The authors report that the effect holds up when retrieval noise is added and also transfers to a second benchmark, HotpotQA, not just LongMemEval. Their conclusion is that memory and RAG evaluations should report, or actively control, the reader-facing artifact format, since otherwise the same underlying knowledge and the same model can look dramatically more or less capable purely as an artifact of how the text was packaged.

Key facts

  • RENDER is a benchmark control that fixes the conversation content and varies only the format (memory entry, summary, typed record, or raw excerpt) shown to the model.
  • Tested on 500 LongMemEval questions across nine models, using a five-level packet ladder plus four deterministic templates: ChatGPT-style entries, LangChain-style summaries, MemGPT-style typed records, and raw conversation.
  • Matched-budget resolved packets beat recency-truncated raw dialogue by 42.4 to 72.6 points; the best-worst spread among deployed-style templates alone is 24.6 to 48.8 points per model.
  • Under the primary scorer, ChatGPT-style entries beat raw conversation on 7 of 9 models; three models that scored 0 percent on formal ledger packets scored 45.4 to 53.4 percent on the same facts in natural-language form.
  • The effect survives added retrieval noise and transfers to the HotpotQA benchmark, leading the authors to argue evaluations should report or control the input format.

Why it matters

Memory and RAG evaluations are widely used to rank how well language models recall and use past conversation or retrieved context, and those rankings feed decisions about which model or memory system to deploy. RENDER's core finding is that the format used to present the same underlying facts, independent of the facts themselves, can swing a model's measured accuracy by tens of points. A model is not simply 'good' or 'bad' at memory in the abstract; it is good or bad at memory rendered a particular way, and two evaluations of the same model using different formatting conventions can produce contradictory verdicts about its capability.

Who it affects

The result concerns anyone building or evaluating memory-augmented or RAG-based LLM systems: teams designing chat memory features (the kind that produce ChatGPT-style entries), developers building on frameworks like LangChain that generate summaries, systems built on MemGPT-style typed records, and researchers who benchmark model memory using raw conversation excerpts. It also affects anyone reading published memory or RAG benchmark results, since a model's ranking may depend heavily on which template the benchmark happened to use.

How to use it

RENDER is presented as a benchmark control rather than a product, meaning its practical use is methodological: evaluators can adopt its packet ladder and template set to test whether their own memory or RAG scores are sensitive to formatting before drawing conclusions about a model's capability. Teams building memory systems can also use the finding directly by choosing natural-language-style entries over strict formal ledger formats, since the paper shows the same facts scored far higher for several models when presented in natural language rather than formal typed records.

How solid is it

The evaluation spans 500 questions from an established benchmark, LongMemEval, across nine different models, giving the core comparison reasonable breadth. The authors also checked robustness in two ways: rescoring outputs with a separate judge model, which preserved the positive aggregate effect of good formatting even though model-specific statistical significance varied, and testing whether the effect held under added retrieval noise and on a second, different benchmark, HotpotQA, where it did transfer. The abstract does not specify which nine models were used, what the judge-based rescoring method measures beyond the primary scorer, or what the deterministic templates look like in detail beyond their names, so the precise mechanics behind the point gaps are not fully visible from the summary alone.

Risks and caveats

Because judge-based rescoring showed mixed, model-specific significance rather than a uniform effect, the magnitude of the formatting advantage is not guaranteed to hold identically across every model or deployment, and practitioners should not assume a single template choice will produce the same size of gain everywhere. The paper's own framing is a caution rather than a fix: it argues evaluations should report or control the rendering format, which implies that a large share of existing memory and RAG benchmark results, run without that control, may already carry unacknowledged formatting bias.