Baseten maps the efficient frontier of LLM inference serving

A blog post from inference platform Baseten borrows the economics term "efficient frontier" to organize how engineers should think about serving large language models. A frontier model, the article says, is one that offers the highest intelligence at a given cost or size; the same logic applies to serving infrastructure, where the usual tradeoff is between latency and throughput (which sets cost), and sometimes between quality and speed, or intelligence and speed via reasoning level.

The piece splits serving techniques into two categories. The first moves a deployment along the existing frontier, trading one factor for another. The second pushes the whole frontier outward, creating more total efficiency that can then be spent on lower latency, higher throughput, or both. The article frames its examples around running a model like GLM-5.3 or Kimi K3 for agentic coding, with KV cache reuse and KV-aware routing enabled, though this is a working assumption for the discussion rather than a benchmark result.

Among tradeoff techniques: batch sizing is the most direct lever. Small batches give excellent per-user latency but generate few tokens per GPU, so cost per token is high; increasing batch size worsens per-user latency but improves overall throughput and lowers cost. Parallelism strategy is the second lever. Because today's models run to hundreds of billions or trillions of parameters and must be split across multiple GPUs, the split method matters: Tensor Parallelism is recommended for latency-sensitive deployments despite its costly all-to-all communication, since that communication is fast over high-bandwidth NVLink; Expert Parallelism at a lower degree is often associated with better latency, while wide EP, including EP spread across a full rack of GPUs, generally supports higher throughput; Attention Data Parallelism replicates attention layers to boost throughput at the expense of per-request speed. Quantization, meanwhile, is described as pushing the frontier out rather than just trading along it: running a model at lower precision in weights, activations or KV cache improves both latency and throughput, and with microscaling formats such as MXFP4 and NVFP4 a large efficiency gain is possible with little to no quality loss, though this introduces its own jagged quality-versus-efficiency frontier.

Among frontier-pushing techniques: kernel optimization improves the low-level functions (like matrix multiplication) that make up a forward pass, reducing the resources needed per generated token; the article points to a separate writeup by Baseten intern Brian Li for more detail on kernel-level performance. Speculative decoding, which guesses upcoming tokens and validates the guesses, used to trade latency for throughput because speculation was expensive and only viable at small batch sizes; newer techniques such as EAGLE-3, DSpark and DFlash still compete with the main model for resources but perform well enough, especially on predictable code-generation output, to yield genuine efficiency gains from skipped forward passes rather than just faster tokens. Prefill/decode disaggregation, running the two inference phases on separate dedicated workers, lets each phase be optimized separately and lets the ratio of prefill to decode workers be tuned to the traffic's input and output lengths and cache hit rates. The article notes that these frontier-pushing techniques compound: doubling performance from better hardware together with doubling performance from better software would yield a four-times improvement in overall serving capacity, though this figure is offered as an illustrative example of compounding, not a measured result. The article closes by pointing to the author's free book, Inference Engineering, for detail on every technique covered.

Key facts

Baseten splits LLM serving techniques into two kinds: those that trade latency for throughput along an existing 'efficient frontier', and those that push the whole frontier outward, freeing capacity to spend on either.
Batch sizing is the clearest tradeoff lever: small batches give the best per-user latency but the highest cost per token; larger batches worsen per-user latency while raising overall throughput and lowering cost.
GPU parallelism choices matter differently for each goal: Tensor Parallelism favors latency, narrow Expert Parallelism often favors latency while wide EP (up to a full GPU rack) favors throughput, and Attention Data Parallelism trades per-request speed for system throughput.
Quantization (including microscaling formats MXFP4 and NVFP4) is presented as a true frontier-pusher, improving latency and throughput together with little to no quality loss in many cases.
Kernel optimization, newer speculative-decoding methods (EAGLE-3, DSpark, DFlash) and prefill/decode disaggregation are the techniques described as expanding overall serving capacity rather than just reallocating it, and the article says such gains compound.

Why it matters

Teams serving large language models constantly juggle latency, throughput and cost, and the two kinds of levers get confused in practice: a change that only reallocates existing capacity (like shrinking batch size) gets treated as if it created new capacity, and vice versa. Naming the distinction, tradeoff techniques versus frontier-pushing techniques, gives engineers a framework for knowing which knob to turn when a workload's requirements change, rather than tuning by trial and error. The framing is especially relevant for agentic coding workloads, the article's own example, where both fast per-step responses and high overall throughput matter at once.

Who it affects

The audience is inference and ML infrastructure engineers responsible for deploying and serving LLMs in production, particularly teams running large models like GLM-5.3 or Kimi K3 that must be split across multiple GPUs. It applies both to teams building high-throughput batch pipelines and to teams serving latency-sensitive interactive users with a high willingness to pay for speed, since the article treats these as opposite ends of the same tradeoff space.

How to use it

For latency-sensitive deployments, the article's guidance is to favor Tensor Parallelism and narrower Expert Parallelism, keep batch sizes small, and adopt quantization and newer speculative-decoding methods since these push the frontier rather than costing latency elsewhere. For throughput- and cost-sensitive batch workloads, it recommends wider Expert Parallelism (up to a full GPU rack), Attention Data Parallelism, and larger batch sizes. Separately, kernel optimization and prefill/decode disaggregation are framed as worth pursuing regardless of which side of the tradeoff a deployment favors, since they expand total capacity rather than reallocating it. The article stresses that exact cutoff points are jagged and unintuitive, and must be found through empirical sweeps rather than assumed from general principles.

How solid is it

This is a vendor engineering blog post from Baseten, an inference-serving company, rather than a research paper or an announcement with measured results. The article contains no benchmark numbers, percentages, dollar costs, or latency and throughput figures for any of the techniques it describes; its one numeric example, a four-times improvement from compounding a doubling in hardware performance with a doubling in software performance, is explicitly a hypothetical illustration rather than a reported measurement. The technical claims read as generalized engineering experience (for example, on Tensor Parallelism, Expert Parallelism, and quantization with MXFP4/NVFP4) rather than as findings tied to a specific case study or dataset.

Risks and caveats

Because Baseten sells LLM inference infrastructure, this piece functions partly as content marketing for the company's expertise and for a referenced free book, Inference Engineering, even where the technical points are reasonable. The article itself cautions that the efficient frontier is jagged rather than smooth, meaning small configuration changes can have outsized and counterintuitive effects, so general recommendations here may not transfer directly to a given deployment without empirical testing. No author byline, publication date, or named production deployment appears in the crawled text, so the guidance should be read as general engineering advice rather than as evidence from a specific verified rollout.

“A model is a "frontier model" if it offers the highest degree of intelligence at a given cost or size.”

— Baseten blog post, "The efficient frontier of LLM inference"