Glonce

Paper "You don't need long context" says compaction keeps LLM accuracy

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance92Surprise45Players12Credibility65
Paper "You don't need long context" says compaction keeps LLM accuracy

A paper titled "You don't need long context" challenges the case for long-context language models. Its starting point is a cost observation: later tokens are costlier to generate than earlier tokens. On frontier open weight models, the authors say, the disparity in cost may rise to as much as 7x.

The central claim is empirical. Through frequent compaction, the authors show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when they maintain short contexts. In other words, a model that keeps its working context short by repeatedly compacting it does about as well as one that carries everything along.

The authors then push the idea to an extreme. They demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when the model takes at most one action per context. Based on that result, they outline the concept of semantic state space models (SSSMs).

The paper's conclusion is worded as a question, not a refutation: its findings question the need for long-context language models. The text available is the paper's abstract only, so the mechanics of compaction, the definition of SSSMs beyond the name, and the accuracy figures are not covered here.

Key facts

  • The paper argues that frontier language models sacrifice little to no task accuracy when frequent compaction keeps their contexts short, across a broad range of long-horizon and long-input tasks.
  • Its cost argument: later tokens are costlier to generate than earlier ones, and on frontier open weight models the disparity may rise to as much as 7x.
  • On long-horizon software engineering tasks, performance does not collapse even when the model takes at most one action per context.
  • From that result the authors outline the concept of semantic state space models (SSSMs).
  • The authors say their findings question the need for long-context language models.

Why it matters

Long context windows are usually treated as a capability to scale. This paper pulls the other way: if frequent compaction preserves accuracy, a model may not need to carry a long context at all. The authors tie this to cost, since later tokens are costlier to generate than earlier ones, with the gap reaching as much as 7x on frontier open weight models. The paper also sketches a new concept, semantic state space models (SSSMs), built on its software engineering result.

Who it affects

The argument speaks to anyone who builds or runs language models on long-horizon tasks or long inputs, where context length drives both capability and generation cost. The software engineering experiment points at agent-style work in particular, since it counts actions taken per context.

How to use it

The abstract gives no recipe. What it does offer is a direction: keep contexts short through frequent compaction instead of relying on a long window. The abstract does not describe how compaction works or what SSSMs are beyond the name, so anyone wanting to apply the idea would need the full paper.

How solid is it

The claims are the authors' own, stated in the abstract of a paper on Hugging Face's papers page. The source here is the abstract only. No authors or institutions are named. No specific models, benchmarks or task names are given. No accuracy figures are given; "little to no" accuracy loss is not quantified. The 7x cost disparity is stated only for frontier open weight models and as an upper bound ("as much as"), with no context lengths given for the comparison.

Risks and caveats

"Little to no" loss is a general statement across a broad range of tasks, and the abstract names none of them, so it is hard to tell where the result might break down. The 7x figure is a ceiling, not a typical value. The abstract does not claim long-context models are useless; it says the findings "question the need" for them. The software engineering result shows performance does not collapse at one action per context, which is a weaker claim than matching full-context accuracy.

“Our findings question the need for long-context language models.”

— From the paper's abstract, "You don't need long context"