TTR-Compressor uses test-time regression to compress token sequences
A paper listed on Hugging Face Papers proposes a different way to compress sequences inside language and multimodal models. The authors call sequence compression a fundamental operation and a critical information bottleneck in efficient models, and say that prevailing approaches based on pooling or learned projections remain underperforming.
Their method looks at compression through test-time regression. The input tokens define a regression problem, and the fitted coefficients of that problem form the compressed representation. The regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. The result is TTR-Compressor, which the authors present as a general alternative to conventional pooling and projection-based compression.
They test it in two settings. The first is as the compressor for block-wise sparse attention. There, TTR-Compressor improves RULER performance over attention-based compressors by a large margin, according to the authors. The second is as the visual token merger for multimodal large language models. There it consistently outperforms the standard MLP merger and matches that merger's performance with 55.6% fewer visual tokens.
The authors conclude that these results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models. The source is the abstract only.
Key facts
- TTR-Compressor treats input tokens as a regression problem; the fitted coefficients become the compressed representation.
- The regression problem is learned end-to-end via backpropagation, and its coefficients are computed in closed form during the forward pass.
- As a compressor for block-wise sparse attention, it improves RULER performance over attention-based compressors by a large margin, per the authors.
- As a visual token merger for multimodal LLMs, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens.
Why it matters
Compression of token sequences sits inside efficient language and multimodal models, and the authors describe it as a critical information bottleneck where pooling and learned projections still underperform. This paper offers a different primitive: fit a regression over the tokens and keep the coefficients. If the reported results hold, the same idea serves two quite different jobs, long-context sparse attention and visual token merging.
Who it affects
The work is aimed at people building efficient language models and multimodal large language models, specifically those who use block-wise sparse attention compressors or an MLP merger to shrink visual tokens. No authors or institutions are named in the source.
How to use it
The abstract describes two uses: replace the compressor in block-wise sparse attention, or replace the standard MLP merger that reduces visual tokens in a multimodal LLM. The regression problem is trained end-to-end, so it would be learned along with the model. No code or model release is mentioned.
How solid is it
The evidence here is the authors' own abstract. Their headline figure is relative: TTR-Compressor matches the standard MLP merger with 55.6% fewer visual tokens than that merger uses. The RULER gain is described only as a large margin. No absolute RULER scores or numeric size of the 'large margin' are given, and the abstract does not give the baseline token count.
Risks and caveats
No model names, model sizes, benchmarks for the multimodal evaluation, or datasets are named. No compute, latency or memory savings are stated. The claim that test-time regression is an effective primitive rests on two evaluations reported in a short abstract, so the details need the full paper.
“These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.”
— Paper abstract, Hugging Face Papers