Declarative Attention cuts KV cache reads by up to 52%

Researchers introduce Declarative Attention (DA), a protocol that lets a language model control its own attention instead of relying on an external mechanism to guess which parts of a long context matter. The problem it targets: language models spend most of their attention on a small fraction of the context, yet global attention layers still read the entire KV cache to generate each token of a reply. If a user asks about a detail buried in a 1M-token conversation, the model scans the full context anyway to answer. A prominent existing fix pre-selects relevant tokens with lightweight proxy scores computed outside the model, but the authors say this extrinsic scoring still costs O(N) per generated step, so it does not remove the underlying scan.

DA takes an intrinsic approach instead: rather than scoring tokens from the outside, it has the model itself declare, within its own chain-of-thought, which parts of the context it needs next. Generation is partitioned into three modes: , which reads the full context; , which reads a specific region; and , which reads only recent output. The inference engine parses these declarations the way it would parse tool calls, and skips reading most of the KV cache accordingly.

The authors evaluated DA zero-shot, with no additional training, on two off-the-shelf models, Gemma-4-31B and Qwen-3.6-27B, across 15 long-context tasks. DA cut the total number of attended tokens during decoding by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, compared with each model's default full-context attention. The accuracy cost was modest: a drop of 1.27 percentage points on Gemma-4-31B and 2.75 percentage points on Qwen-3.6-27B, and the authors report that this gap shrinks as model scale increases. The paper frames DA as opening a new axis for sparse attention, and notes that training a model specifically for DA, rather than applying it zero-shot to an existing model, is a direction left for future work.

Key facts

Declarative Attention (DA) has a model declare, within its own chain-of-thought, whether it needs (full context), (a specific region), or (recent output only) attention, so the inference engine can skip most KV cache reads.
DA was tested zero-shot, with no extra training, on two off-the-shelf models, Gemma-4-31B and Qwen-3.6-27B, across 15 long-context tasks.
It cut total attended tokens during decoding by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B versus each model's default full-context attention.
The accuracy cost was 1.27 percentage points on Gemma-4-31B and 2.75 percentage points on Qwen-3.6-27B, a gap the authors say narrows as model scale increases.
Existing proxy-scoring methods for sparse attention still cost O(N) per decoding step; DA is proposed as an intrinsic alternative that lets the model itself decide what to reread.

Why it matters

Reading the full KV cache on every decoding step is one of the main costs of long-context inference, and it stays expensive even when only a sliver of the context is actually relevant to the current token. Prior sparse-attention approaches attack this from the outside, scoring tokens with a separate lightweight mechanism, but that scoring itself still scales with context length. DA's premise is that the model already knows which region of the conversation it needs; letting it say so directly removes the need for a separate scoring pass and, in the reported experiments, removed roughly a third to half of the attended tokens during decoding.

Who it affects

The direct audience is teams building or serving long-context language models, where KV cache reads are a real latency and compute cost at scale. It also matters to anyone relying on assistants that hold very long conversations or documents in context, since cheaper attention at inference time is what makes that kind of long-context use practical to serve. The paper evaluates existing off-the-shelf models rather than shipping a product, so the immediate audience is inference-engine and model-serving engineers, not end users.

How to use it

DA is applied at inference time: the model is prompted to emit its attention declarations inside its chain-of-thought, and the inference engine parses each declaration like a tool call, then restricts its KV cache read to the declared region before generating the next token. The reported results use it zero-shot, without any additional training of Gemma-4-31B or Qwen-3.6-27B, meaning it was layered onto existing models rather than requiring a new training run. The source gives no information on code availability, licensing, or integration into a specific serving stack.

How solid is it

The evaluation covers 15 long-context tasks under a zero-shot setup on two different model families and sizes, Gemma-4-31B and Qwen-3.6-27B, which gives some evidence the approach is not tied to one architecture. The reported metrics are limited to the reduction in attended tokens and the accuracy change; the source gives no latency, wall-clock speed, or memory measurements, and it does not name the 15 individual tasks. The source text carries no author names, institutions, or affiliations, so those cannot be reported here.

Risks and caveats

The accuracy drop is real, not zero: 1.27 and 2.75 percentage points on the two tested models, even if the authors describe it as modest and shrinking with scale. The token-reduction benefit also varies substantially between the two models tested, 52.0% versus 31.1%, so the gain is not guaranteed to generalize evenly across architectures. All reported results are zero-shot on off-the-shelf models; DA has not been evaluated with a model trained specifically to use it, which the paper itself flags as future work rather than a demonstrated result. The approach also shifts responsibility for deciding what context matters onto the model's own self-report, so its efficiency gains depend on the model correctly judging what it needs to reread.