Evolving Infra paper reports up to 4.1x lower LLM serving latency than Dynamo

A paper on Hugging Face presents Evolving Infra, a serving framework built for agentic workloads. The authors start from the premise that agents are a major source of LLM inference demand, and argue that current prefill-decode (PD) disaggregated serving systems handle them poorly. In those systems, some nodes run the prefill stage and others run decode. The authors say such systems offer limited flexibility for adapting compute allocation and KV cache management at runtime, and that they rely on coarse signals that only partly capture how varied and changeable agent workloads are. The result, in the authors' words, is load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse.
Evolving Infra responds in two steps. First, it decouples compute allocation from KV cache management, so a node can switch efficiently between prefill and decode. Second, it builds two components on that design. An online planner matches the PD split to the composition of the workload. An agent-aware distributed KV manager decides what to evict and what to transfer, using signals from the agent's own context management.
For evaluation, the authors ran coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs. They report that Evolving Infra reduces mean end-to-end request latency by up to 4.1x and P95 time to first token (TTFT) by up to 14x, compared with Dynamo, which the paper describes as NVIDIA's production framework. Both figures are best-case maxima, and the abstract gives no per-task or per-model breakdown.
Key facts
- Evolving Infra is an agent-adaptive LLM serving framework that adapts compute allocation and KV cache management to agent workloads at runtime.
- It decouples compute allocation from KV cache management so nodes can switch between prefill and decode, with an online planner setting the PD split.
- An agent-aware distributed KV manager uses signals from agent context management to guide cache eviction and transfer.
- Reported against Dynamo, NVIDIA's production framework: up to 4.1x lower mean end-to-end request latency and up to 14x lower P95 TTFT.
- Tests covered coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs.
Why it matters
The paper's starting point is that agentic workloads are a major source of LLM inference demand, and that serving stacks tuned for ordinary requests fit them badly. The authors say PD-disaggregated systems end up with idle or overloaded prefill and decode nodes, and throw away KV caches that an agent is about to reuse. The proposed fix is to let the serving layer follow what the agent is doing, both in how compute is split and in which cache entries are kept. If the reported gains hold, that is a large margin over a production baseline.
Who it affects
The work is aimed at people who run LLM inference for agent workloads, especially on PD-disaggregated setups like the one Dynamo provides. The evaluated workloads were coding, math and computer-use tasks, so teams serving agents of those kinds are the closest match to what was tested.
How to use it
Nothing in the abstract is a recipe to apply directly. No code release, open-source status or deployment in production is mentioned. What can be taken from it is the design idea: separate compute allocation from KV cache management so nodes can change role, size the prefill-decode split from the live workload mix, and let the agent's context management inform which caches to evict or move.
How solid is it
The figures are the authors' own, taken from the abstract, and they are 'up to' maxima rather than typical or average values. The abstract does not say which of 4.1x or 14x was measured on which model or task. The pairing of 4.1x with mean end-to-end latency and 14x with P95 TTFT follows the order of the sentence. The only comparison named is Dynamo; no comparison against other systems is mentioned. The evaluation scale is up to 256 GB300 GPUs, and no results are given for smaller GPU counts or other hardware.
Risks and caveats
Treat the headline numbers as best cases against a single baseline, reported by the paper's own authors. The abstract names no authors or institutions and gives no submission date. Results for GPU counts below 256 or for hardware other than GB300 are not given, so how the gains carry over to smaller or different clusters is unknown from this text.
“Agentic workloads are a major source of LLM inference demand.”
— Evolving Infra paper abstract