Prefix Sliding can make reasoning models 3x faster without retraining

Test-time scaling improves a language model's performance by giving it extra compute at inference time, for example letting it reason for longer before answering. The standard way to do this keeps the model's entire reasoning trace in memory through full attention, which the authors say makes the hard tasks that need long thinking prohibitively expensive. Investigating that cost, the authors find that most intermediate reasoning tokens lose importance as the model keeps reasoning, which calls into question whether keeping all of them in memory is worth what it costs.

Based on that finding, the authors propose Prefix Sliding, a method that discards reasoning tokens once they fall outside two zones it keeps in memory: a prefix holding the key instructions and tools available to the model, and a window of the last few thousand tokens, which is the reasoning the model is actively working on right now. Because both zones stay fixed in size, the total memory Prefix Sliding needs stays capped no matter how long the model goes on reasoning, which the authors say is what makes efficient long-horizon test-time scaling possible.

The paper reports two separate results from two different setups. First, without any additional training, applying Prefix Sliding to an existing model can make it 3x faster while maintaining performance, a training-free result. Second, training a model with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens, a separate, trained setup. The abstract does not say the trained version is also 3x faster, or that the untrained version reaches traces beyond a hundred thousand tokens; each number belongs to its own setup.

In ablation studies, the authors report that Prefix Sliding outperforms two alternative approaches: summarizing the intermediate reasoning tokens instead of discarding them, and a vanilla sliding window that does not keep a separate prefix. The text states only the direction of that result, not a numeric margin, and it does not name the benchmark, dataset or model family used for either the ablations or the 3x speedup figure. The authors say their code is public, at https://github.com/Muennighoff/prefix-sliding. The abstract text itself does not name the paper's authors or their institutional affiliation; a submitter name is attached to the Hugging Face listing, but it does not appear in the article text.

Key facts

Prefix Sliding discards reasoning tokens once they age out of two fixed zones: a prefix carrying key instructions and tools, and a window of the last few thousand tokens of active reasoning.
Because those two zones stay fixed in size, the method caps total memory use regardless of how long the model reasons, which the authors say enables efficient long-horizon test-time scaling.
Applied to an existing model with no additional training, Prefix Sliding can make it 3x faster while maintaining performance.
Training a model with Prefix Sliding using reinforcement learning can enable scaling to reasoning traces beyond a hundred thousand tokens, a separate result from the untrained 3x speedup.
In ablations, Prefix Sliding outperforms both summarizing intermediate tokens and a vanilla sliding window, though the paper gives no numeric margin for either comparison.

Why it matters

Test-time scaling, letting a model reason longer at inference to solve harder problems, is a central lever for improving reasoning models, but its standard implementation keeps the entire reasoning trace in memory through full attention. The authors argue that makes genuinely hard, long-thinking tasks prohibitively expensive, and they trace the problem to a specific finding: most intermediate reasoning tokens lose importance as reasoning continues, so keeping every one of them in memory buys little. Prefix Sliding turns that finding into a fix by keeping only a fixed prefix of key instructions plus a trailing window of recent reasoning, which caps memory use regardless of how long the model reasons. The result is not a new reasoning ability but a way to make long test-time reasoning computationally sustainable, on models that already exist and, if trained for it, at reasoning lengths well past what full-attention memory would allow.

Who it affects

This is aimed at researchers and engineers who build or run reasoning models where test-time compute, and the memory it consumes, is a real cost: anyone whose inference bill or latency grows with how long a model is allowed to think. It is most directly useful to teams that can apply Prefix Sliding to a model they already have without retraining, and to teams training their own reasoning models who could adopt it during reinforcement learning to reach longer traces. The abstract text does not name the paper's authors or an institution behind the work; a submitter name is attached to the Hugging Face page's metadata, but that name does not appear in the article text itself, so no individual or organization can be credited here.

How to use it

The abstract describes two ways to apply Prefix Sliding. The first needs no retraining at all: applying it to an existing model can make that model 3x faster while maintaining performance. The second is applied during training, using reinforcement learning, and the authors say it can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens, a different setup from the training-free speedup. The authors say their code is public, at https://github.com/Muennighoff/prefix-sliding, so either path can in principle be tried directly against that repository. The text gives no exact size for the prefix, only that it holds the key instructions and tools available to the model, and no precise token count for the trailing window beyond a few thousand tokens, so applying the method exactly as described means reading the code rather than the abstract.

How solid is it

The core evidence is a self-reported claim from the authors: a 3x speedup with maintained performance for the training-free case, and reasoning traces beyond a hundred thousand tokens for the reinforcement-learning case, plus ablations showing Prefix Sliding beats summarizing intermediate tokens and a vanilla sliding window. The abstract names no benchmark, dataset or model family for any of these comparisons, gives no numeric margin for the ablation results, and states no publication venue, peer-review status, or submission date. On Hugging Face's papers board the submission has drawn light attention so far, 4 points and two comments, which reflects visibility rather than independent verification of the results. The authors have made their code public, which is what would let outside researchers check the claims directly.

Risks and caveats

Every headline number in the abstract lacks a stated benchmark or model family to check it against, and the ablation result is given only as a direction, Prefix Sliding beating the two alternatives, with no size given for the gap. The 3x speedup and the beyond-a-hundred-thousand-token scaling come from two different setups, untrained versus trained with reinforcement learning, and the abstract does not say either result also holds for the other setup. The prefix and trailing-window sizes are described only loosely, as key instructions and tools for the prefix and a few thousand tokens for the window, so how much of a model's context budget Prefix Sliding actually needs in practice is not stated. No institution or named author stands behind the claims in the text itself.

“Without training, Prefix Sliding can make existing models 3x faster while maintaining performance.”

— the authors