Glonce

LatentPress compresses context into memory tokens, matches raw-context accuracy

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact65Novelty72Relevance98Surprise52Players22Credibility87
LatentPress compresses context into memory tokens, matches raw-context accuracy

Most approaches to compressing a language model's context still convert the result back into text or into images that the model must decode before it can use them, even when the model itself is the only consumer of that context. LatentPress instead writes conversational histories and long documents into continuous memory tokens: a frozen decoder reads these tokens directly through its input-embedding interface, and no text reconstruction happens at inference. A small, reader-matched writer module performs the compression. Only this writer, an adapter of 4.2 million to 26.2 million parameters, about 0.1% of the frozen decoder's size, is trained; the decoder itself is left untouched.

The method compresses content by a factor of 4 to 16 times. On the LongMemEval benchmark, LatentPress reaches 0.504 accuracy at 7.70x compression, close to the 0.490 accuracy of reading the uncompressed evidence directly, and well ahead of a text-summary baseline (0.184 accuracy) and an OCR-based compression baseline (0.426 to 0.312 accuracy). On LongBench-QA, writers trained in-domain match or exceed raw-context reading at 4x to 8x compression; pushed to 16x compression, they fall behind raw-context reading.

The approach is also markedly faster. Writing, meaning compressing, a conversation into memory tokens takes 43 milliseconds, roughly an order of magnitude faster than text summarization or OCR reconstruction. Reading the resulting tokens is 5 to 9 times faster than reading raw context or cached OCR output.

The authors test the interface under two zero-shot transfer settings: from UltraChat to LongMemEval's memory QA task, and from LongMemEval-derived QA to unseen LongBench document domains. They argue this establishes direct soft tokens as a practical, machine-facing context interface that goes beyond text and vision. The implementation of the experiments is available on GitHub.

Key facts

  • LatentPress compresses conversational histories and long documents into continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference.
  • Only a small adapter, 4.2 million to 26.2 million parameters (about 0.1% of the frozen decoder), is trained, while the method achieves 4 to 16 times compression.
  • On LongMemEval, LatentPress reaches 0.504 accuracy at 7.70x compression, versus 0.490 for uncompressed evidence, 0.184 for text summaries, and 0.426 to 0.312 for OCR-based compression.
  • On LongBench-QA, in-domain writers match or exceed raw-context reading at 4x to 8x compression, but trail it at 16x.
  • Writing takes 43 milliseconds per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading the memory tokens is 5 to 9 times faster than raw context or cached OCR.

Why it matters

Systems that keep long conversational histories or feed a model long documents usually pay for that memory either by reprocessing raw text on every turn or by compressing it into a text summary or a rendered image the model has to decode again before it can use it. LatentPress proposes a third path: skip text and vision altogether and hand the model continuous tokens it reads straight through its embedding layer. On LongMemEval, this recovers accuracy close to reading the full uncompressed evidence (0.504 versus 0.490) while cutting the amount of content the model processes by 7.70x, and it does so by training only a small adapter rather than touching the underlying decoder. Near-parity accuracy at several-times compression, with a fraction of a percent of extra trained parameters, is the combination that makes this a notable result rather than an incremental one.

Who it affects

The result is aimed at builders of long-context and memory-augmented language model systems: anything that has to keep a conversational history, a long document, or retrieved evidence around and re-read it repeatedly. It is also directly relevant to teams currently relying on text-summary or OCR-based compression pipelines, since the paper reports both as clearly weaker on accuracy, 0.184 for text summaries and 0.426 to 0.312 for OCR-based compression on LongMemEval, than the continuous-token approach.

How to use it

Applying the method requires access to a frozen decoder's input-embedding interface, plus training a small reader-matched writer module rather than modifying the base model. That writer is the only trained component: 4.2 million to 26.2 million parameters, about 0.1% of the decoder's size. The authors have published the implementation of their experiments on GitHub, at https://github.com/xuyd16ai/context_softtoken_compress.

How solid is it

The claims rest on two established long-context benchmarks, LongMemEval and LongBench-QA, each tested against three baselines: uncompressed raw context, text summaries, and OCR-based compression. The authors additionally test two zero-shot transfer settings, from UltraChat to LongMemEval's memory QA task and from LongMemEval-derived QA to unseen LongBench document domains, meant to show the interface generalizes rather than overfitting to one benchmark. Two gaps limit how far the evidence reaches: the text gives no description of the decoder's architecture, size, or which base model(s) were used, and it makes no comparison to other continuous-memory or soft-token compression methods, only to text summaries, OCR-based compression, and uncompressed context.

Risks and caveats

At the highest compression ratio tested, 16x, LatentPress's writers trail raw-context reading on LongBench-QA, so the accuracy-for-compression tradeoff is not free at every setting; the range the paper reports as matching or exceeding raw-context reading is 4x to 8x. The source also does not say at which compression ratios the OCR-based-compression accuracy range (0.426 to 0.312) was measured, so that particular comparison cannot be pinned to an exact operating point. The text contains no discussion of limitations, failure modes, or safety and societal implications of the approach.

“We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference.”

— the paper's abstract