SIMPLE law predicts LLM inference energy with four parameters

The paper starts from a simple observation: generative AI is billed per token, yet tokens are not equally expensive. Each token enters the KV-cache and raises the cost of the tokens that follow, so energy grows non-linearly along a sequence even though its tokens look identical. The authors say no analytical model states what a token costs, and that the most accurate state-of-the-art (SOTA) model is a fit rather than a law: it absorbs architecture into fitted coefficients and needs fitting for each LLM.

To fix that, they derive closed-form expressions for the computational and memory-access complexity of LLM inference in terms of architectural hyperparameters, covering both multi-head attention (MHA) and grouped-query attention (GQA). They compare a family of analytical models of increasing granularity. From these they obtain the SIMPLE Inference Model to Predict LLMs Energy, or SIMPLE: a four-parameter law that separates prefill from decode, and splits decode into KV-cache cost and next-token computation.

Tested on traces of 13 dense LLMs from six families on NVIDIA H100 GPUs, SIMPLE reaches R2 = 0.969 and 11.97% MAPE (mean absolute percentage error) without per-LLM calibration. The paper puts that at 3.1-3.5x below the SOTA.

The authors argue the law explains as well as predicts. Prefill energy grows quadratically with the hidden dimension, so larger LLMs suffer most from long inputs. More aggressive GQA KV grouping reduces per-token energy by up to 3x relative to standard MHA.

On reasoning, increasing the token budget pushes output length beyond the analytically derived energy minimum and drives super-linear energy growth. Accuracy rises with the budget, but energy rises faster, so Intelligence per Joule peaks at an intermediate budget. It falls by up to 3.14x across a 4.0x reasoning-budget range.

On agentic serving, the paper reports that energy per request is 13.0x that of non-agentic requests. The authors attribute this to growth of both the prefill and KV-cache terms with input length. Prefix caching removes most of the prefill cost (-93%), but the KV-cache term is unaffected, so even a cached agentic request costs 4.8x more.

Key facts

SIMPLE is a four-parameter law that separates prefill from decode and splits decode into KV-cache cost and next-token computation, derived from closed-form complexity expressions covering MHA and GQA.
On traces of 13 dense LLMs from six families on NVIDIA H100 GPUs it reaches R2 = 0.969 and 11.97% MAPE without per-LLM calibration, which the paper says is 3.1-3.5x below the SOTA.
More aggressive GQA KV grouping cuts per-token energy by up to 3x relative to standard MHA; prefill energy grows quadratically with the hidden dimension.
Intelligence per Joule peaks at an intermediate reasoning budget and falls by up to 3.14x across a 4.0x budget range.
Agentic serving raises energy per request by 13.0x versus non-agentic requests; prefix caching cuts prefill cost by 93%, yet a cached agentic request still costs 4.8x more.

Why it matters

Token billing treats every token alike, but the paper argues the energy cost of a token depends on where it sits in a sequence, because each token enters the KV-cache and raises the cost of later ones. The authors say existing predictors are fits that need per-LLM fitting, and that no analytical model states what a token costs. SIMPLE is offered as a law built from architectural hyperparameters rather than a curve fitted model by model. Its by-products are concrete: a quadratic prefill dependence on hidden dimension, a GQA saving of up to 3x per token, a reasoning budget beyond which efficiency drops, and a large energy premium for agentic workloads.

Who it affects

The findings bear on anyone who serves or budgets for LLM inference. Operators of long-context workloads are told larger models suffer most from long inputs. Teams choosing between MHA and GQA designs get an energy figure for more aggressive KV grouping. Teams that set reasoning-token budgets are told where Intelligence per Joule peaks. Builders of agentic systems are shown that their requests cost 13.0x more energy per request than non-agentic ones, and that prefix caching only partly helps.

How to use it

The abstract gives no formula, parameter list, code or dataset, so it cannot be applied directly from this text. What a reader can take from it are the directional lessons: favour more aggressive GQA KV grouping where quality allows, expect long inputs to hurt larger models most, and treat the reasoning budget as a tuning knob with an intermediate optimum for Intelligence per Joule. For agentic serving, prefix caching removes most of the prefill cost (-93%) but leaves the KV-cache term, so the paper says to expect a cached agentic request to cost 4.8x a non-agentic one.

How solid is it

The paper reports R2 = 0.969 and 11.97% MAPE on traces of 13 dense LLMs from six families on NVIDIA H100 GPUs, with no per-LLM calibration, and says this is 3.1-3.5x below the SOTA. These are the authors' own results. No authors or institutions are named. The names of the 13 LLMs and six families are not given, nor is the SOTA model being compared against. It is not stated whether the 3.1-3.5x is a ratio of MAPE or of another error measure, beyond 'below the SOTA'.

Risks and caveats

Evidence is limited to dense LLMs on H100 GPUs; other hardware and MoE models are not covered. The baseline for the 13.0x and 4.8x agentic figures is described only as 'non-agentic ones', and the models, request lengths and workloads behind them are not given. The four parameters of the law and its formula are not stated. No dollar costs, absolute joule values or token counts are given, and no code, dataset release or publication venue is mentioned. The figures like 'up to 3x' and 'up to 3.14x' are maxima, not typical values.

“Generative AI is billed per token, yet tokens are not equally expensive.”

— Paper abstract