Glonce

AKV engine lets LLM agents drop and reposition KV cache directly

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact45Novelty60Relevance95Surprise25Players10Credibility70
AKV engine lets LLM agents drop and reposition KV cache directly

A paper listed on Hugging Face Papers tackles a cost problem in long-running LLM agents. As an agent works through a long trajectory of observations and actions, key-value (KV) cache I/O and storage become the dominant cost, and the trajectory can also outgrow the model's maximum context length, so the active cache has to be kept in check.

The usual answer is context engineering: trim or summarize the history. The authors say this has a built-in price. Modifying the trajectory prefix invalidates the cached states that follow it and triggers costly re-prefilling of the context that was kept. Task accuracy can also suffer when too little context is retained in order to limit those re-prefill costs.

The paper proposes a different scheme, cache engineering. It lets an agent control its active KV cache directly, without modifying the trajectory and without re-prefilling retained context. Two operations do the work: drop, which excludes selected cache entries from future attention, and repos, which adjusts the position encodings of the keys that are retained. The authors formalize these operations.

They also identify a serving challenge they call history dependence. Identical token sequences can end up with different cache states, depending on which cache operations were applied along the way, so prefix matching is not enough for correct cache reuse. To handle this, they co-design a serving engine called AKV. It exposes a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration.

The reported results: a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. Running on AKV, it reaches up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. The authors also report that recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Code is in the thu-nics/AKV GitHub repository.

Key facts

  • Cache engineering gives agents two operations on their own KV cache: drop (exclude selected cache from future attention) and repos (adjust retained keys' position encodings), with no change to the trajectory and no re-prefilling.
  • A tool-response rolling-drop policy improves average benchmark scores by 1.7 points and cuts average cost by 27.7% versus conventional threshold-triggered summarization.
  • The AKV serving engine offers a stateless interface compatible with existing serving APIs while managing stateful cache reuse, eviction and restoration internally; it reaches up to 1.2x output throughput and a 1.5x speedup in P95 request latency.
  • Recursive self-improvement of cache policies cuts cost a further 7.1% over expert design.
  • Code is public at github.com/thu-nics/AKV.

Why it matters

For long agent runs, the authors argue KV cache I/O and storage become the dominant cost. Existing fixes such as trimming or summarizing history edit the trajectory prefix, which invalidates cached states and forces re-prefilling. This paper's idea is to let the agent prune its cache in place, so the saved work is not thrown away. The headline claim is a policy that is both cheaper (27.7% lower average cost) and slightly better (1.7 points higher average benchmark score) than threshold-triggered summarization.

Who it affects

Teams building or serving long-horizon LLM agents, where trajectories grow large and cache I/O and storage costs dominate. It also concerns people who run inference engines, since the paper argues that correct cache reuse under these operations needs more than prefix matching.

How to use it

The code is published at https://github.com/thu-nics/AKV. The AKV engine presents a stateless interface compatible with existing serving APIs, so the authors position it as something that fits current serving setups while handling cache reuse, eviction and restoration internally. In the agent itself, the proposed policy uses drop to remove selected cache, such as tool responses, from future attention and repos to fix the position encodings of keys that remain.

How solid is it

This is an author-reported result from a paper listed on Hugging Face Papers. The benchmarks, models and agent tasks used in the experiments are not named in the text. The baseline for the 1.2x throughput and 1.5x P95 latency figures is not stated, and the 1.2x throughput is an "up to" figure, not an average. The scale of the 1.7-point score gain is not given, and neither is which expert design or self-improvement procedure sits behind the 7.1% figure. No peer-review or venue status is stated.

Risks and caveats

History dependence is the central hazard the authors name: identical token sequences can produce different cache states under different operation sequences, so naive prefix-based reuse can be incorrect. That is why a purpose-built engine is needed. The reported gains are compared with one baseline, conventional threshold-triggered summarization, and the headline numbers should be read with the unknowns above in mind, especially the unstated baselines for throughput and latency.

“It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context.”

— Paper abstract, Cache Engineering for Agents