Glonce

PyTorch 2.15 brings FP8 and much faster MoE decoding to Mac GPUs

toolsThe Glonce editorsHugging Face🔥71Glonce rating71Impact60Novelty70Relevance100Surprise45Players60Credibility85
PyTorch 2.15 brings FP8 and much faster MoE decoding to Mac GPUs

The PyTorch team at Hugging Face has published a rundown of what the MPS backend (PyTorch's route to the Apple GPU) gains in PyTorch 2.15. The authors say their goal is to make the Mac a first-class place to run and train PyTorch models. At the start of 2026, they write, even small models on MPS often meant working around unsupported operations, performance cliffs and unexpected memory growth. Their method is to run real models from the Hugging Face Hub across language, image and video, find where MPS falls short, and fix it upstream, increasingly by replacing MPSGraph with dedicated Metal kernels.

FP8 arrives on MPS. PyTorch 2.15 adds torch.float8_e4m3fn tensors: one byte per value, half the storage of FP16 or BF16. They can be multiplied with F.scaled_mm, which supports both tensorwise and rowwise scaling, and the post includes a short example that quantizes a tensor and multiplies it with one scale per tensor.

The headline result is LLM decoding. During decode each new token goes through the model as a single row, so large matrix multiplications become matrix-vector products. For dense layers, MPS has had GEMV kernels for torch.mm since 2.14, but F.linear still went through MPSGraph. In 2.15 F.linear uses them directly for the common [1, 1, hidden_size] input, with eligible bias additions fused in. Across 96 BF16 layer shapes from real models that is a 1.23× geometric-mean speedup. The biggest gains are on small projections such as MoE gates; large layers and LM heads were already close to bandwidth-bound and stay near 1×.

For mixture-of-experts layers, 2.14 copied the group offsets to the CPU and looped over the groups in grouped matmul. 2.15 adds native Metal kernels that read the offsets on the GPU, using Metal Performance Primitives where available and a separate Metal path elsewhere, plus a dedicated kernel for the small groups typical of decoding. Together with fewer GPU-CPU syncs and faster top-k routing, MoE models decode 5-6.5× faster than in 2.14. On an M3 Ultra with torch.compile, Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s and LFM2.5-8B-A1B from 23.6 to 123.6 tokens/s. The post separately says the decoding kernel gives MoE models about 1.3× more tokens per second on the same setup. Softmax and log_softmax now run on dedicated Metal kernels: an FP32 softmax over a [1, 128256] row of logits drops from 57 µs to 7 µs, about 8× faster than the MPSGraph version.

Linear algebra expands. complex64 LU factorization, LU solve and triangular solve now run on MPS, which also enables operations built on them, such as complex determinants. Batches of FP32 matrices up to 4 × 4 are inverted in a single Metal kernel; QR uses blocked updates; triangular solves lean on matrix multiplication. Gated DeltaNet workloads get a fast path for batches of 16 × 16, 32 × 32 and 64 × 64 FP32 triangular systems with at least eight right-hand sides. SVD now handles very small singular values, extreme input magnitudes and basis completion with full_matrices=True.

The post calls the video result one of the most striking. The VAE decode in Wan-AI/Wan2.2-TI2V-5B-Diffusers makes 8,764 padding calls over a full 28-tile decode. With constant padding moved onto Metal, peak driver memory drops from about 270 GiB to 8 GiB. The same padding change fixes a bug in which the Qwen-Image-2.1 VAE encoder silently corrupted images on MPS in 2.14, breaking image-edit workflows (reported in ComfyUI). An encode-decode round trip now matches CPU at 29.95 dB PSNR, up from 10.06 dB. The post also lists faster paths for convolutions, dropout, interpolation backward and reductions.

Other changes: public MPS streams (torch.mps.Stream, the torch.mps.stream context manager, and Tensor.record_stream), more reliable event timing, and new MPS support for torch.frexp, torch.logspace, torch.take, Tensor.put_, torch.heaviside, torch.special.laguerre_polynomial_l and unsigned-integer comparisons (logspace does not yet support integer or boolean outputs with more than one element). The MPS math implementation of scaled dot-product attention now handles batch and head broadcasting correctly. Requirements tighten: MPS now requires macOS 15 or later, and PyTorch 2.15 requires Python 3.11 or later. The availability check now says why MPS is unavailable if the OS is too old or no Metal GPU is present.

The post closes with a preview of 2.16; some items have landed on main and are in nightlies, others are still in review. Decoding will commit the command buffer every 16 kernels so the GPU can start while the CPU is still encoding. Scaled dot-product attention is dropping MPSGraph completely, with every shape on Metal kernels. On an M5 Pro, grouped-query attention decoding (32 query heads, 8 KV heads, head dim 80, 4K context) drops from 465 µs to 65 µs, and a 4K × 4K prefill with head dim 40 goes from 42 ms to 6 ms. Among the further MPSGraph-to-Metal moves, MSE loss (up to 12× faster on small inputs), std_mean and var_mean (up to 4× faster) and bincount have landed, while batch norm forward and backward are in progress. Counting 4 million integers with bincount, when most values fall in one bin, goes from 6.8 seconds to half a millisecond. Bundled shaders will build for Metal 3.1 by default, shrinking libtorch_cpu.dylib by about 25%, from 384 MiB to 287 MiB.

Key facts

  • PyTorch 2.15 adds torch.float8_e4m3fn tensors on MPS (one byte per value, half of FP16 or BF16), multiplied through F.scaled_mm with tensorwise or rowwise scaling.
  • MoE models decode 5-6.5× faster than in 2.14 on an M3 Ultra with torch.compile: Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s and LFM2.5-8B-A1B from 23.6 to 123.6 tokens/s.
  • FP32 softmax over a [1, 128256] row of logits falls from 57 µs to 7 µs, about 8× faster than the MPSGraph version.
  • Peak driver memory for the Wan2.2 video VAE decode drops from about 270 GiB to 8 GiB after constant padding moved to Metal; the same change fixes corrupted Qwen-Image-2.1 edits.
  • MPS now requires macOS 15 or later and PyTorch 2.15 requires Python 3.11 or later; the 2.16 preview shows attention fully on Metal and a roughly 25% smaller libtorch_cpu.dylib.

Why it matters

The post argues that MPS was held back by unsupported operations, performance cliffs and memory growth, and 2.15 attacks all three on real Hugging Face Hub models. The biggest single change is that mixture-of-experts decoding no longer bounces group offsets through the CPU: on an M3 Ultra, Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s. The Wan2.2 video decode memory drop, from about 270 GiB to 8 GiB, turns a workload that overran the machine into one that fits. The wider direction is the replacement of MPSGraph with dedicated Metal kernels, which the 2.16 preview extends to attention.

Who it affects

People who run models on an Apple silicon Mac with transformers, diffusers or tools like ComfyUI. Local LLM users get faster decoding, and the authors say dense models also gain from faster linear layers and softmax. Video-generation users get the Wan2.2 memory cut. Image-editing users of Qwen-Image-2.1 get a fix for silent image corruption. Anyone on an older system is affected by the new floor: macOS 15 or later and Python 3.11 or later.

How to use it

You need an Apple silicon Mac with macOS 15 or later and Python 3.11 or later. The post shows how to try the release candidate in a separate environment: uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/test/cpu. For FP8, quantize a tensor to torch.float8_e4m3fn with a scale and call F.scaled_mm with the scale recipes set to TensorWise or RowWise. F.grouped_mm takes the tokens, per-expert weights and an offs tensor of group end positions. The new public stream API is torch.mps.Stream with the torch.mps.stream context manager. The 2.16 items can be tried in nightlies only where they have landed on main.

How solid is it

This is a first-party post from the PyTorch team at Hugging Face, with concrete benchmarks and code samples. The 5-6.5× headline rests, in the visible text, on two models (Qwen3.6-35B-A3B and LFM2.5-8B-A1B) on an M3 Ultra with torch.compile. The post also states a separate 1.3× tokens-per-second gain for the MoE decoding kernel, and the visible text does not say what that figure is measured against, since its chart is not included. The 1.23× linear-layer figure is a geometric mean across 96 BF16 shapes, not a maximum. The 2.15 install command is for a release candidate, and no release date is given for 2.15 or 2.16. The source text available here is cut off partway through the 2.16 preview, and the list of convolution and padding speedups is not in it.

Risks and caveats

2.15 is shown as a release candidate, so final behaviour could differ. The requirement jump to macOS 15 and Python 3.11 will exclude older setups. Large layers and LM heads stay near 1× on the new linear path because they were already close to bandwidth-bound, so dense models should not expect MoE-sized gains. The 2.16 material is explicitly provisional: some changes are on main, others are still in review, and details may change before release. logspace still lacks integer or boolean outputs with more than one element. The benchmark numbers come from the authors' own tests on an M3 Ultra (2.15) and an M5 Pro (2.16 preview), and results on other chips are not given.

“the goal is to make your Mac a first-class place to run and train PyTorch models”

— PyTorch team at Hugging Face