Nvidia and Cerebras tout inference speeds that won't scale

At the Hot Chips conference in California this week, Nvidia announced its new Groq-3-based LPX racks had entered production. Early tests, Nvidia said, showed the systems running Gemma 4 31B at 3,400 tokens a second per user, four times Cerebras' rate. Cerebras answered a day later, touting nearly equivalent performance from its next-generation CS4 accelerators, which had been revealed the week before.

The Register argues that neither company is lying, but that both are quoting a number their customers will probably never see in production. The 3,400 tok/s figure, verified by benchmarking firm Artificial Analysis for both vendors, describes a single request running alone on the hardware. No inference-as-a-service operator in their right mind, the outlet writes, would run a production system that way.

To show why, the outlet works through the memory math for Gemma 4 31B, a 31 billion-parameter model it picked because both chipmakers' specs made an estimate possible and because a small model is the best case for both. Each Groq-3 LPU carries 500 MB of on-chip memory. At Nvidia's 8-bit precision, holding the model's weights alone takes about 31 GB, or roughly 64 LPUs. Serving a request also needs a KV cache built during prompt processing: per LMcache's calculator, a 100,000-token input at 8-bit precision consumes another 8 GB per sequence, equal to 16 more LPUs. A full LPX rack holds 256 LPUs, which by this arithmetic tops out at a batch size of about 12 requests of 100,000 tokens each before running out of memory.

Cerebras' CS4 rack, built from three WSE-3T accelerators with 44 GB of SRAM apiece for 132 GB total, has more headroom but lands on the same ceiling: a maximum batch size of about 12. The Register stresses this is a memory limit, not a compute limit; a rack can only serve more requests by adding hardware or shrinking the prompt. The outlet concedes that real chatbot traffic, made up of shorter, asynchronous, varied-length requests rather than fixed 100,000-token batches, could let a single rack serve well beyond 12 users, so its estimate is a simplified floor rather than a real production number.

The piece's broader argument is that SRAM-heavy chips like Groq's and Cerebras' work best as decode accelerators paired with GPUs or other high-throughput chips, not as standalone systems, because pairing largely removes the memory-capacity problem that limits them alone. That, The Register says, is why Nvidia spent $20B to license Groq's technology and poach its engineering team, and why Cerebras has partnered with both AWS and AMD. Combining GPUs with Groq- or Cerebras-style accelerators, the outlet argues, is what Nvidia, AMD and AWS are all betting on to make low-latency premium inference economical.

The Register also contrasts benchmarking methodology. Nvidia and Cerebras used Artificial Analysis' single-point test, while OpenAI used SemiAnalysis' InferenceX to show its Jalapeño accelerators across both bulk and premium inference, a benchmark the outlet says gives a fuller picture across the full performance range. No benchmark yet exists for combined GPU-plus-LPX or GPU-plus-CS4 systems, though the outlet expects both companies to publish one eventually; it also guesses, without confirming with either company, that Nvidia and Cerebras avoided InferenceX because their racks would look weaker on the bulk end of that curve.

Key facts

At Hot Chips this week, Nvidia said its new Groq-3-based LPX racks are in production and hit 3,400 tokens a second on Gemma 4 31B, four times Cerebras' rate; Cerebras answered a day later with near-equal numbers from its CS4 accelerators, unveiled the week before.
The Register's own math: an LPX rack has 256 Groq-3 LPUs at 500 MB each; running Gemma 4 31B at 8-bit precision needs about 31 GB (roughly 64 LPUs) for the model's weights plus 8 GB per 100,000-token input for the KV cache, capping a rack at a batch size of about 12.
A CS4 rack, with three WSE-3T accelerators totaling 132 GB of SRAM, hits the same batch-size-of-12 ceiling by a separate calculation.
The Register calls this a memory limit, not a compute limit: past that ceiling, a rack needs more hardware or a shorter prompt to serve more requests, though real chatbot traffic could push the practical number well above 12.
Nvidia spent $20B to license Groq's technology and poach its engineering team, and Cerebras has partnered with AWS and AMD; The Register argues the real fix is pairing GPUs with these SRAM-heavy chips as decode accelerators, benchmarked on SemiAnalysis' InferenceX, the tool OpenAI already uses for its Jalapeño chips, rather than Artificial Analysis' single-point test.

Why it matters

The piece is a case study in reading AI hardware benchmarks skeptically. Nvidia's and Cerebras' headline numbers, verified by Artificial Analysis, both come from a single request run alone on the hardware, and The Register's own memory math shows a full rack of either system can support only about 12 such requests before running out of memory, whatever the raw speed. That gap between a marketing number and a production number is why the piece argues the real fix is heterogeneous compute: pairing SRAM-heavy chips like Groq's and Cerebras' with GPUs as decode accelerators rather than running them standalone, which is already why Nvidia spent $20B to license Groq's technology and poach its engineering team, and why Cerebras partnered with AWS and AMD.

Who it affects

Directly: Nvidia and Cerebras, whose new LPX and CS4 racks are the subject of the comparison, and Groq, whose technology Nvidia licensed and whose engineering team it poached for $20B to build the LPX line. Benchmarking firms Artificial Analysis, which produced the disputed figures, and SemiAnalysis, whose InferenceX tool the piece holds up as the more honest alternative, are also named, along with OpenAI, which already uses InferenceX to benchmark its own Jalapeño accelerators. Indirectly: any inference-as-a-service operator or enterprise buyer weighing these chips against GPUs, plus rival SRAM-heavy chipmakers like SambaNova, whose SN-series parts fall in the same category as Groq's and Cerebras' hardware.

How to use it

The piece is analysis, not a product to adopt, but it offers a concrete filter for reading the next chip benchmark: ask where on the Pareto curve, from high-volume bulk throughput to low-latency single-request speed, a headline number sits, and whether it describes a realistic batch size or one request running alone. The Register's suggested proxy for a more honest comparison is SemiAnalysis' InferenceX, which plots performance across that full range rather than a single best case, and which OpenAI already uses to present its own accelerators.

How solid is it

The 3,400 tok/s-class figures for both Nvidia and Cerebras were produced by Artificial Analysis, a named benchmarking firm, and the article does not dispute their accuracy, only their relevance to production. The batch-size-of-12 ceiling, by contrast, is The Register's own back-of-envelope calculation from published memory specs, not a number reported by either chipmaker or any customer, and the outlet says real chatbot traffic, with shorter and asynchronous requests, could push a rack's practical capacity well past 12. No named individual, engineer, executive or analyst is quoted anywhere in the piece, and the crawled item carries no byline; no specific inference-as-a-service provider or customer is named as having tested or rejected either system.

Risks and caveats

The Register's own explanation for why Nvidia and Cerebras avoided the fuller InferenceX benchmark, that their racks would look weaker on the bulk-throughput end of the curve, is a guess the outlet does not claim to have confirmed with either company. Neither the price of Nvidia's LPX racks nor of Cerebras' CS4 systems is given, and no benchmark yet exists for the combined GPU-plus-LPX or GPU-plus-CS4 systems the piece argues will matter more; the outlet only says it expects both companies to publish such results once they are ready.

“In reality, these numbers are more like the top speed on a race car. It makes for great marketing but you probably aren't driving that fast on a regular basis, and if you did, you wouldn't get very far before your tank runs dry.”

— The Register