KernelMosaic: open CUDA dataset lifts a small model's KernelBench L1 correctness from 4% to 88%
A new arXiv paper tackles a data problem: training language models to write efficient CUDA kernels needs supervision that covers many kinds of computational requirements and shows iterative code optimization. That supervision is hard to build, the authors say, because specifications come in different forms and good implementations emerge only through repeated, complicated testing and refinement.
Their answer is KernelMosaic, an open dataset that pairs diverse CUDA tasks with verified solutions and trajectories. Tasks are curated through human-guided review and diversity-aware selection. They are expressed as natural-language descriptions, reference programs, mathematical specifications, and combinations of these.
The solutions and trajectories come from an agentic framework. It guides a strong LLM through three stages: implementation, correctness repair, and performance optimization. The framework supplies reusable CUDA guidance and execution feedback along the way.
The headline result comes from supervised fine-tuning of a small open-source model on KernelMosaic. Correctness on KernelBench rises from 4%, 0% and 0% before fine-tuning to 88%, 82% and 16% after, on levels L1, L2 and L3 respectively. These are absolute correctness rates. Fine-tuning also increases the fraction of tasks with kernels faster than PyTorch eager at every level, though the abstract gives no figures for that.
Further training experiments, according to the authors, show that supervision from the curated trajectories substantially outperforms direct responses and promotes sustained testing and revision. They conclude that KernelMosaic enables a general-purpose model to achieve substantial gains in CUDA kernel generation.
Key facts
- KernelMosaic is an open dataset pairing diverse CUDA tasks with verified solutions and trajectories, with tasks curated through human-guided review and diversity-aware selection.
- Solutions and trajectories were built by an agentic framework that guides a strong LLM through implementation, correctness repair and performance optimization, using reusable CUDA guidance and execution feedback.
- Supervised fine-tuning of a small open-source model on KernelMosaic raises correctness from 4%, 0% and 0% to 88%, 82% and 16% on KernelBench L1, L2 and L3.
- The fraction of tasks with kernels faster than PyTorch eager also increases at every level.
- In further experiments, training on the curated trajectories substantially outperforms training on direct responses and promotes sustained testing and revision.
Why it matters
Writing fast, correct GPU kernels is a task where a model has to test, fail and revise, not just produce code in one shot. The paper's point is that the training data should show that process. KernelMosaic stores trajectories, not only final answers, and the authors report that this supervision substantially outperforms direct responses. The jump on KernelBench L1 (4% to 88%) and L2 (0% to 82%) is large for a small open-source model, and the dataset is described as open.
Who it affects
Researchers and engineers who train or fine-tune language models for CUDA and GPU code generation are the direct audience, since the work offers both a dataset and a method for building one. Teams that build coding agents may also care about the finding that trajectories promote sustained testing and revision.
How to use it
The paper describes KernelMosaic as an open dataset for supervised fine-tuning of models on CUDA kernel generation, and describes the agentic framework used to produce its solutions and trajectories. The abstract gives no license, download link or release date for the dataset, so check the paper page for where to get it.
How solid is it
This is an arXiv preprint, and the numbers come from the authors' own abstract. The results are stated on KernelBench at levels L1, L2 and L3, as absolute correctness rates before and after fine-tuning. The authors back their conclusion with further training experiments comparing curated trajectories against direct responses. The abstract does not state which metric defines correctness beyond the term itself.
Risks and caveats
Gains are uneven: L3 correctness reaches only 16%, far below L1 and L2. The small open-source model that was fine-tuned is not named, nor is its size, and the strong LLM that generated the solutions and trajectories is not named either. No dataset size is given. No comparison with other models or prior methods on KernelBench is given. The speed claim is qualitative: only that the fraction of tasks with kernels faster than PyTorch eager increases at every level, with no figures for how many tasks or how large the speedups are.
“These results demonstrate that KernelMosaic enables a general-purpose model to achieve substantial gains in CUDA kernel generation.”
— KernelMosaic paper abstract, arXiv 2611.14604