Glonce

AdaptiveSpec tops EAGLE-3 with up to 56% higher throughput

researchThe Glonce editorsarXiv🔥70Glonce rating70Impact68Novelty68Relevance92Surprise48Players38Credibility82

Speculative decoding speeds up large language model inference by having a drafter model propose candidate tokens ahead of time, which the target model then verifies in a single parallel pass instead of generating tokens one at a time. Tree-attention drafters such as EAGLE-3 are widely used for this, but they typically hold two decisions fixed: a strict rule that only accepts a drafted token when it exactly matches what the target model would itself have produced, and a static shape for the tree of candidate tokens being drafted. Earlier work has relaxed each of those two decisions, but only one at a time and under limiting assumptions: training-free lossy verification has needed long draft chains, and adaptive tree shaping has only been explored within a fixed total token budget.

The paper introduces AdaptiveSpec, a training-free method that adjusts both decisions at every decoding step using signals already produced during decoding itself, rather than requiring any additional training. Its per-step margin rule lets a mismatched, draft-proposed token through anyway once the target model's probability on that token, divided by its own top-1 probability, crosses a threshold; the rule works the same way regardless of the draft length or which drafter architecture produced the token. A separate per-step tree policy resizes the draft tree's depth, width and node count from a fused signal combining the drafter's own top-1 confidence with a rolling record of how often draft and target tokens have recently agreed, and unlike prior adaptive-tree-shaping work, it lets the total number of drafted tokens itself grow or shrink rather than only being redistributed within a fixed total. The two adaptations act on separate axes, and the paper reports that their effects compound.

AdaptiveSpec was implemented on SGLang, a production-grade LLM serving engine, and compared against EAGLE-3, which the paper describes as the state-of-the-art autoregressive speculative decoding method. Tested across three target models, DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct and Qwen3-8B, and three benchmarks, GSM8K, MATH-500 and HumanEval, it improves throughput over EAGLE-3 by up to 56% while recovering, in aggregate across those models and benchmarks, between 93% and fully lossless task accuracy.

Key facts

  • AdaptiveSpec is a training-free method that adjusts both a speculative-decoding drafter's verification rule and its draft-tree shape at every step, using signals the decoding process already produces rather than any added training.
  • Its per-step margin rule accepts a mismatched draft token once the target model's probability on that token, divided by its own top-1 probability, crosses a threshold, independent of draft length or drafter architecture.
  • Its per-step tree policy resizes the draft tree's depth, width and node count from the drafter's top-1 confidence and a rolling history of draft-target agreement, letting the total number of drafted tokens itself vary.
  • Implemented on the SGLang serving engine, AdaptiveSpec improves throughput over EAGLE-3, the paper's state-of-the-art baseline, by up to 56%.
  • Across three target models, DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct and Qwen3-8B, and three benchmarks, GSM8K, MATH-500 and HumanEval, it recovers between 93% and fully lossless task accuracy.

Why it matters

Speculative decoding is one of the standard ways to make large language model inference faster without changing the target model itself, but the tree-attention drafters most teams use today, EAGLE-3 among them, lock in a strict verification rule and a fixed draft-tree shape regardless of how the decoding is actually going step to step. AdaptiveSpec shows that both of those choices can be adapted per step, using only signals the decoder is already computing, without any additional training. Its margin rule and its tree-resizing policy operate on separate axes, so their gains compound rather than compete. The result, tested on a production-grade serving engine rather than a simulation, is a throughput gain over what the paper calls the state-of-the-art baseline.

Who it affects

The direct audience is teams operating or building LLM inference serving stacks that already rely on tree-attention speculative decoding, EAGLE-3 among them, since AdaptiveSpec changes how such a system makes its per-step accept and tree-shape decisions without requiring a new model, a new serving engine, or additional training. It also matters to anyone serving math or code workloads specifically, since the paper's accuracy results are measured on GSM8K, MATH-500 and HumanEval, benchmarks built around exactly those tasks, across three target models: DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct and Qwen3-8B.

How to use it

AdaptiveSpec requires no extra training and slots into the two decisions a tree-attention drafter already makes at every step: whether to accept a mismatched draft token, and how large a draft tree to build next. The first is governed by a threshold on the ratio between the target model's probability on the drafted token and its own top-1 probability, though the text does not give a numeric value for that threshold, only that the ratio must exceed it. The second is governed by a combined signal of the drafter's top-1 confidence and its recent draft-target agreement history. The paper's own test of the approach ran on SGLang, without describing hardware or GPU configuration beyond naming that serving engine.

How solid is it

The evidence spans three target models at the same roughly 8-billion-parameter scale, DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct and Qwen3-8B, and three benchmarks, GSM8K, MATH-500 and HumanEval, with the comparison run against EAGLE-3 on the SGLang serving engine. Both headline figures, the up to 56% throughput gain and the 93%-to-lossless accuracy recovery, are given only as ranges aggregated across those models and benchmarks; the text does not break either figure down per model or per benchmark, so it is not possible to tell whether the gain and the accuracy trade-off are spread evenly or concentrated in one case. The paper quantifies its comparison only against EAGLE-3; a comparison against the separate line of adaptive-tree-shaping prior work it mentions is not given a number. The text also does not name the authors or their institutions, or state a submission, publication, or acceptance date.

Risks and caveats

Both headline numbers are boundary values, not typical outcomes: up to 56% is the top of the throughput range, and 93% is the floor of the accuracy-recovery range, so some individual model-benchmark pairs likely land below both. Recovering 93% rather than fully lossless accuracy on the low end of that range also means a real, if partial, accuracy cost is being traded for that throughput on at least some of the tested models or tasks. The text gives no absolute throughput numbers, such as tokens per second, only the relative improvement over EAGLE-3, so the real-world speed a deployment would see depends on how fast EAGLE-3 itself runs in that deployment's configuration.

“We introduce AdaptiveSpec, a training-free per-step speculative decoding method that adapts both decisions from internal signals already produced during decoding.”

— the paper