Glonce

AMD MI355X beats Nvidia B300 on cost per GPU running Kimi K3

infrastructureThe Glonce editorsHacker News🔥78Glonce rating78Impact70Novelty52Relevance98Surprise38Players63Credibility82

Wafer.ai, an AI infrastructure company, published a blog post benchmarking the open-source model Kimi K3 on AMD's MI355X GPUs against Nvidia's B300 and B200. Kimi K3 has 2.8 trillion parameters, which needs over 1.5TB of VRAM for the weights alone before any KV cache is added for a 1M-token context window; for comparison, GLM5.2 has 753B parameters and DeepSeek V4-Pro has 1.6T. That size rules out a single 8-GPU B200 node, since each B200 carries less VRAM than the newer chips. Both the MI355X and the B300 offer 288GB of VRAM per GPU, so either can hold Kimi K3 on one node. Wafer.ai priced the three chips at $2.50 per GPU-hour for the MI355X, $6.00 for the B300, and $4.25 for the B200, putting the MI355X at about 2.4 times cheaper per GPU than the B300 and about 1.7 times cheaper than the B200. On a benchmark of 1,024-token input and 400-token output, the MI355X reached 952 tokens per second per node in aggregate and 118 tokens per second in a single stream. The B300 still wins on raw aggregate throughput, by about 1.65 times, but at 2.4 times the price, so the MI355X comes out ahead on performance per dollar. Against Wafer.ai's own two-node, 16-GPU B200 deployment (498 tokens per second total, about 249 per node), the MI355X reached about 3.8 times the aggregate throughput per node and about 1.3 times the single-stream decode speed. Getting there took engineering work. Kimi K3 ships without its own draft tensors for speculative decoding, so Wafer.ai used an external draft model, RadixArk's Kimi-K3-DSpark. On AMD's ROCm software stack, the first real request crashed the inference scheduler with a NameError, because the sglang framework's ROCm build was missing a kernel for top-k probability renormalization that the CUDA build has. The fix required no custom kernel, only a small PyTorch function performing a sort, a mask and a rescale. Once patched, single-stream performance rose about 2.2 times, per-stream throughput at moderate load rose about 1.7 times, and peak aggregate throughput rose 18%, with the peak now landing at a concurrency of 64 requests instead of 24. A second problem hit prefill speed rather than decode: an identical 172,000-token cold prefill took about 51 seconds on the MI355X versus about 23 seconds on the B300. The cause was that Kimi K3, run at tensor-parallelism of 8, produces 12 attention heads per rank, while AMD's fast AITER MLA prefill kernel only supports 4, 8, or multiples of 16 heads, so the system fell back to a much slower generic Triton kernel. The fix was to zero-pad the head count from 12 to 16, run the fast kernel, then discard the padding from the output. That let the AITER kernel run at about 13,000 tokens per second in steady state, versus about 4,000 to 7,000 for the Triton fallback, cutting prefill time by roughly 2 to 3 times. Wafer.ai concludes that AMD's day-zero software support for Kimi K3 meant fewer framework bugs to work around than it saw with GLM5.2, and that state-of-the-art performance on AMD hardware is close.

Key facts

  • On a 1,024-token input, 400-token output benchmark, AMD's MI355X reached 952 tokens per second per node in aggregate and 118 tokens per second single-stream serving Kimi K3; Nvidia's B300 still won on raw aggregate throughput by about 1.65 times, but at 2.4 times the price, so the MI355X won on performance per dollar.
  • Kimi K3 has 2.8 trillion parameters, needing over 1.5TB of VRAM for weights alone before a 1M-token KV cache; both the MI355X and the B300 provide 288GB of VRAM per GPU, letting either fit the model on a single 8-GPU node, unlike the B200.
  • Wafer.ai priced the comparison at $2.50 per GPU-hour for the MI355X, $6.00 for the B300, and $4.25 for the B200.
  • Fixing a missing ROCm kernel for speculative decoding gained about 2.2 times single-stream throughput, about 1.7 times per-stream throughput at moderate load, and 18% more peak aggregate throughput.
  • The MI355X was slower than the B300 on cold prefill (about 51 seconds versus 23 seconds for an identical 172,000-token prompt) until Wafer.ai padded Kimi K3's 12 attention heads per rank to 16 so AMD's fast AITER MLA kernel would load, cutting prefill time by about 2 to 3 times.

Why it matters

Kimi K3's 2.8 trillion parameters make it too large for a single Nvidia B200 node, forcing a choice between splitting the model across two B200 nodes or using a GPU with more memory per chip. Wafer.ai's benchmark argues that AMD's MI355X, with the same 288GB of VRAM per GPU as Nvidia's B300, can serve such oversized open-weight models at a lower cost per unit of throughput, even though the B300 is still faster in absolute terms. That argument only holds because AMD shipped working day-zero support for Kimi K3, cutting the amount of custom engineering needed compared with Wafer.ai's earlier experience with GLM5.2.

Who it affects

The comparison is aimed at teams choosing hardware to serve very large open-source models such as Kimi K3, GLM5.2, and DeepSeek V4-Pro, including cloud GPU providers and companies running their own inference infrastructure. It also feeds the broader competitive question between AMD and Nvidia over inference workloads, since the result depends on AMD narrowing the software gap that has historically favored Nvidia's CUDA ecosystem.

How to use it

The post is a technical write-up, not a product or service to buy. Two of its fixes are directly reusable: patching sglang's ROCm build with a small PyTorch function for top-k probability renormalization to unblock speculative decoding, and zero-padding an attention head count from 12 to 16 so AMD's AITER MLA prefill kernel will load instead of falling back to a slower generic kernel. Wafer.ai's stated per-GPU-hour prices, $2.50 for the MI355X, $6.00 for the B300, and $4.25 for the B200, give a reference point for comparing serving costs, though the source does not say whether these are list prices, spot prices, or negotiated rates.

How solid is it

The benchmark comes from a single source, Wafer.ai, an AI infrastructure company with a commercial interest in demonstrating multi-vendor GPU efficiency; the article does not name an individual author, gives no publication date, and does not state whether AMD, Nvidia, or any outside party has reproduced or verified the numbers. The throughput figures are tied to one specific test configuration (1,024-token input, 400-token output, plus a 172,000-token cold-prefill test), so they describe that workload rather than a general claim about all inference workloads.

Risks and caveats

The cost comparison depends on GPU-hour prices whose basis (list, spot, or contract) is not disclosed, so the 2.4 times and 1.7 times cost advantages could shift under different pricing terms. The B300 still beats the MI355X on raw aggregate throughput and on cold-prefill speed for large contexts, so workloads sensitive to time-to-first-token may still favor Nvidia hardware despite the fixes described. The results have not been independently reproduced or reviewed.

“AMD continues to prove its case as the winner in performance per dollar.”

— Wafer.ai's blog post