AWS rations CPU cycles as agentic AI strains server capacity

Earlier this year AWS leadership told its engineers to conserve CPU cycles at all costs, after the company reportedly saw wait times for CPU server capacity explode as AI workloads strained its cloud infrastructure. The shortage caught AWS off guard: the AI boom had driven demand for GPUs and then memory, while CPUs were mostly left out of the story because their limited parallelization makes them a poor fit for running LLM inference.

Agentic AI is changing that. Matt Kimball, vice president and principal datacenter analyst at Moor Insights & Strategy, says 2026 has brought a spike in CPU demand tied largely to agents. "It's one thing to have this agentic workload, and let's say, it spawns 100 agents. If I'm going to roll this out across my enterprise, those 100 become tens of thousands, hundreds of thousands, or millions of agents," he says, adding that agents spawn sub-agents, make API calls and talk to other agents through protocols such as Anthropic's model context protocol. While an LLM's own inference still runs mainly on a GPU, the tool calls it makes, opening files, hitting the internet, running code, are typically pushed to the CPU. Souvik Kundu, senior staff research scientist at Intel, explains that the CPU parses the output, figures out which tool to invoke, makes the API call or runs the code, collects the result and feeds it back. AMD's Madhu Rangarajan, vice president of compute and enterprise AI, says the company's own testing found seven of the eight stages in realistic agentic AI pipelines run entirely on the CPU.

Kundu co-authored a paper with researchers at the Georgia Institute of Technology showing that the CPU often sits idle while the GPU handles inference, and that the GPU sits idle in turn while the CPU handles tool calls. Their proposed scheduling optimizations can cut end-to-end agentic-workload latency by up to 1.8x under sustained load, but the target keeps moving: agentic systems generate and multiply work fast, and the article notes that OpenAI's model fired off as many as 300 actions an hour during what it describes as OpenAI's inadvertent hack of Hugging Face, with a single agent able to spawn sub-agents that make tool calls of their own. Safety and policy guardrails add further CPU load: some are rule-based checks on syntax and logs, others use small models under a billion parameters to judge task complexity or intent, and even though these could run on a GPU, Kundu says they usually stay on the CPU to minimize latency.

A second Georgia Tech paper, co-authored by PhD student Euijun Chung, adds a related bottleneck: tokenization, the step that turns text into token IDs before inference. Chung and his co-authors found that when a server has too few CPU cores, it falls behind dispatching work to the GPUs, which then stall waiting for instructions. Tokenization is sequential, branchy string manipulation rather than the massively parallel math of inference, and an agent's tool calls make it worse: as Chung puts it, if an ongoing sequence already runs 100,000 tokens and a tool returns 1,000 more, the tokenizer has to retokenize the whole sequence again, at every tool call. The paper found that increasing CPU core counts can cut time-to-first-token latency by roughly 1.5x to 7x at longer sequence lengths, though the tests were limited to smaller models, Alibaba's Qwen 3-30B and Meta's Llama 3.1-70B, by available hardware. Chung expects larger models to see less dramatic bottlenecks thanks to their higher GPU demand, but also expects agentic AI to push sequence lengths well past what was tested; he points to Claude-class models, which he says can easily reach 500,000 or even a million tokens, as a sign the problem will keep growing.

Market signals back up the analysts' read. Intel has sold out of server CPUs through at least the end of the year and has cut client CPU production to prioritize server chips, even as its new 18A process grows client sales. AMD has doubled its server CPU forecast. Arm and Qualcomm have both announced new CPUs aimed at agentic AI, and Nvidia has prioritized Vera, its Arm-based CPU that is part of the Vera Rubin platform. Kimball calls the demand surge an "absolute tell" that CPUs are now central to agentic AI systems, and warns the trend could bring the kind of broader shortages and price increases already seen with GPUs and memory. "You're already seeing a CPU crunch to some degree. When you look at the constraints in the market, it even trickles down into the consumer space," he says.

Key facts

AWS mandated engineers to conserve CPU cycles after wait times for CPU server capacity reportedly exploded as AI workloads strained its cloud infrastructure.
AMD's own testing found seven of the eight stages in realistic agentic AI pipelines run entirely on the CPU; Intel's Souvik Kundu co-authored a scheduling fix that cuts end-to-end agentic latency by up to 1.8x under sustained load.
Georgia Tech's Euijun Chung found tokenization must rerun on an entire growing sequence at every agentic tool call, and that more CPU cores cut time-to-first-token latency by roughly 1.5x to 7x at longer sequence lengths, in tests on Qwen 3-30B and Llama 3.1-70B.
Intel has sold out of server CPUs through year end and AMD has doubled its server CPU forecast, while Arm, Qualcomm and Nvidia (with its Vera chip) have all released or prioritized new CPUs for agentic workloads.
Analyst Matt Kimball says the CPU crunch is already trickling into the consumer market, with Intel cutting client CPU production in favor of server CPUs.

Why it matters

The AI industry's capacity crunch has moved beyond GPUs and memory to CPUs, chips that were mostly sidelined during the earlier boom because they are a poor fit for the massively parallel math of LLM inference. Agentic AI changes that: an agent's tool calls, code execution, file handling and coordination with other agents through protocols like Anthropic's model context protocol run on the CPU, and each agent can spawn sub-agents that multiply the load. Researchers at Intel, AMD and Georgia Tech independently describe the same pattern: while inference sits on the GPU, most of the surrounding work in an agentic pipeline runs on the CPU, and the two chips end up idling while waiting on each other.

Who it affects

Cloud providers like AWS, which is already rationing CPU cycles internally; enterprises rolling out agentic systems at scale, where a single workflow's 100 agents can become millions across an organization; and chipmakers Intel, AMD, Arm, Qualcomm and Nvidia, all shipping or announcing CPUs built for agentic workloads. The article also flags a knock-on effect for ordinary buyers: Intel has shifted production toward server CPUs, and Kimball says the crunch is already trickling into the consumer market.

How to use it

For anyone building or scaling agentic systems, the takeaway is that CPU provisioning needs planning alongside GPU allocation: tool calls, output parsing, safety checks and repeated tokenization of growing context windows all add CPU load that compounds as an agent spawns sub-agents. Guardrail checks, often small models under a billion parameters or rule-based scans of syntax and logs, are kept on the CPU for latency reasons even when a GPU could run them. Longer conversations make this worse: Chung notes that context windows on models like Claude can reach 500,000 or a million tokens, and every tool call forces the whole sequence to be retokenized, so systems with long-running agents should expect the CPU cost to keep climbing.

How solid is it

The claims come from named, on-the-record sources: Moor Insights & Strategy analyst Matt Kimball, Intel's Souvik Kundu and AMD's Madhu Rangarajan speaking to their employers' own testing, and Georgia Tech PhD student Euijun Chung describing peer research he co-authored. The two headline performance figures, the 1.8x latency cut and the 1.5x to 7x time-to-first-token improvement, come from two separate papers, though the article does not name the papers, their venues or publication dates. Chung's tokenization tests ran only on smaller models, Qwen 3-30B and Llama 3.1-70B, because of hardware limits, and Chung himself expects the pattern to differ at larger scale. AWS's mandate and its wait-time increase are described as reported rather than independently confirmed, and neither is quantified.

Risks and caveats

Several of the headline numbers are framed as maximums rather than typical results: the 1.8x latency reduction is described as "up to," and the 1.5x to 7x figure as "roughly," both under specific test conditions that may not generalize. OpenAI's inadvertent hack of Hugging Face is mentioned only in passing, with no date or further explanation beyond the 300-actions-an-hour figure, so it reads as an illustrative aside rather than a fully sourced incident. The piece gives no numbers for how much CPU prices have risen or how much AWS's wait times actually grew, and it does not say whether AWS's internal conservation mandate has had any measured effect.

“It's one thing to have this agentic workload, and let's say, it spawns 100 agents. If I'm going to roll this out across my enterprise, those 100 become tens of thousands, hundreds of thousands, or millions of agents.”

— Matt Kimball, vice president and principal datacenter analyst at Moor Insights & Strategy