Glonce

KernelGenBench shows LLM-generated kernels struggle across chips

researchThe Glonce editorsarXiv🔥70Glonce rating70Impact62Novelty70Relevance97Surprise50Players28Credibility80

Researchers have introduced KernelGenBench, a benchmark built to test how well large language models and AI agents can write Triton kernels, the low-level code that accelerator hardware runs to execute deep learning operations. Kernel development is normally a specialized, labor-intensive task, and the paper argues that despite rapid progress in LLM-based code generation, no benchmark had rigorously evaluated LLM-generated kernels across both diverse operator sources and different hardware platforms at the same time.

KernelGenBench is built from two complementary parts. KernelGenBench-MS (Multi-Source) evaluates 210 operators drawn from three sources beyond the standard PyTorch-centric tasks that prior work relied on. KernelGenBench-MC (Multi-Chip) measures performance portability using a 110-operator subset run across six heterogeneous hardware platforms.

The evaluation itself was large, consuming more than 15 billion tokens. Three findings stand out. First, agent-based methods, systems that iterate on and check their own output, consistently beat plain LLM sampling, though cuBLAS operators proved the hardest case for every method tested. Second, generation performance varies sharply by hardware: even recent kernel-specialized agents suffer severe cross-platform degradation, with AutoKernel's success rate dropping from 87% on NVIDIA hardware to 25% on Platform E. Third, autonomous kernel generation remains highly cost-intensive: specialized agent methods needed an average of 5.11 million tokens per successful operator, with AKO4all reaching 5.19 million, orders of magnitude above what simple LLM sampling approaches required.

Key facts

  • KernelGenBench combines two sub-benchmarks: Multi-Source (210 operators from three sources beyond standard PyTorch tasks) and Multi-Chip (a 110-operator subset run across six hardware platforms).
  • The full evaluation consumed more than 15 billion tokens.
  • Agent-based methods consistently outperform plain LLM sampling, but cuBLAS operators were the hardest case across all methods.
  • AutoKernel's success rate fell from 87% on NVIDIA to 25% on Platform E, showing severe cross-platform degradation even for kernel-specialized agents.
  • Specialized agent methods averaged 5.11 million tokens per successful operator, with AKO4all reaching 5.19 million, orders of magnitude above simple LLM sampling.

Why it matters

Kernel development is a bottleneck in deploying efficient AI accelerators, and interest in using LLMs and agents to automate it has grown quickly. KernelGenBench is the first benchmark to test this across both operator diversity and multiple hardware platforms at once, giving a more realistic picture than prior benchmarks that stuck to PyTorch-centric tasks on a single chip.

Who it affects

ML systems engineers and researchers building agentic kernel-generation tools, teams evaluating whether AI can cut down hardware-specific engineering work, and hardware vendors whose platforms are covered by the Multi-Chip portability tests.

How to use it

KernelGenBench is a benchmark rather than a product: using it means running candidate LLMs or agents against its two sub-benchmarks, the 210-operator Multi-Source set and the 110-operator Multi-Chip subset spanning six platforms, to compare kernel-generation methods on both success rate and hardware portability. The source text does not mention pricing or a specific public release channel for the benchmark code.

How solid is it

The claims rest on a large-scale evaluation that consumed over 15 billion tokens, spanning 210 operators from three non-PyTorch sources and six heterogeneous hardware platforms, a substantial testbed. The headline findings are backed by concrete, checkable numbers, AutoKernel's drop from 87% to 25% and the 5.11 million and 5.19 million token averages, which supports reproducibility, though the source text does not describe independent replication outside the authors' own runs.

Risks and caveats

The benchmark also exposes the limits of the technology it measures: cuBLAS operators remain the hardest case for every method tried, cross-platform generalization is weak even for specialized agents, and autonomous kernel generation is described as highly cost-intensive, orders of magnitude above plain LLM sampling. That suggests current agentic kernel generation is far from a drop-in replacement for hand-written kernels on new hardware.