Glonce

Apple M3 Neural Engine bug throttles DRAM bandwidth, fix more than doubles Llama speed

infrastructureThe Glonce editorsHacker News🔥59Glonce rating59Impact65Novelty75Relevance80Surprise70Players40Credibility85

Eileen Yoon, profiling DRAM weight-streaming throughput on the Apple M3 Neural Engine for single-token decode, found that a kernel DMA transfer whose size is an exact multiple of 1 MiB per core collapses from the nominal 45-60 GB/s down to a fixed floor of 17-19 GB/s. The bug currently affects 7 of the 15 models in the ANEMLL project, an Apple Neural Engine inference backend. She first noticed it by accident: at output dimension N=4096, input dimension D=1536 ran nearly 3 times faster than D=2048, the default used by Llama 3.2. Sweeping D across its full range and repeating each measurement 40 times under constant thermal and load conditions, she found the collapse recurs at every integer multiple of D=2048, with an FFT of throughput against D showing a resonance at exactly that period; at D=2048 throughput measured 16.93 GB/s versus 44.5 GB/s at the neighboring D=2016, a drop of 27.57 GB/s, or 61.96 percent.

She ruled out two DRAM-level explanations before landing on a digital-logic one. Sweeping the number of active Neural Engine cores from 1 to 16 showed the throttle already present with a single core, so it is not core-to-core contention. Randomly scrambling the physical addresses the weights are fetched from, spread across a roughly 64 MiB IOVA arena, produced only a marginally higher median throughput (32.29 GB/s versus a 31.37 GB/s baseline), nowhere near enough to explain a drop of that size, so DRAM bank aliasing was not the cause either. Register-level hexdiffs across a small range of D values confirmed that only the DMA address and size fields changed, pointing at the transfer size itself.

The 1 MiB threshold works out to 16,384 (0x4000) 64-byte DMA lines, and 0x4000 is 2 to the 14th power, the same width as the 16 KiB virtual-memory page size Apple Silicon uses. Yoon's hypothesis is that a speculative prefetch ring inside the kernel DMA engine tracks its position with a 14-bit head or tail pointer that has no separate bit to distinguish a full lap from an empty ring; a transfer whose length is an exact multiple of 0x4000 lines makes the pointer arithmetic alias 'one full lap remaining' to 'empty,' so the prefetcher stops issuing lookahead requests for the entire transfer. The fetch still completes correctly, since this is not a correctness bug, but it forces the whole transfer onto a slow, unprefetched path. The pattern fit: throughput recovers to nominal within about 256 lines (one 16 KiB page) of a 1 MiB boundary, and the per-lap time curves for transfers spanning multiple 1 MiB laps scale linearly with the number of laps, with a fit quality of R-squared 0.96 to 0.99.

The fix needs no firmware change. Any kernel DMA task that would total exactly 1 MiB (or a multiple of it) of weight data per core is split into chunks that are not themselves multiples of 1 MiB, for example two 512 KiB transfers instead of one 1 MiB transfer. Splitting a single 0x4000-line task (17.25 GB/s) into two 0x2000-line tasks recovered 45.52 GB/s, a 2.66x speedup; four 0x1000-line tasks gave 44.83 GB/s, 2.60x. A control test split a transfer that was never a 1 MiB multiple in the first place and got no speedup at all, which isolates the gain to avoiding the 1 MiB boundary rather than to chunking itself. Across five tested transfer sizes from 1 MiB to 8 MiB, the chunked version raised throughput from a pinned 17.3-19.1 GB/s to 43.5-60.5 GB/s, speedups of 2.51x to 3.16x. Applied to real models, Llama 3.2 1B decode throughput rose from 10.0 to 24.3 tokens per second (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens per second (DRAM usage from 22.4 to 48.7 GB/s). The post, published August 10, 2026, also lists the other affected models and their split configurations: Llama 3.1 8B, DeepSeek and DeepHermes 8B, DeepHermes 3B, and a shard of Gemma 3 4B's lm_head.

Key facts

  • An RTL erratum in the Apple M3 Neural Engine throttles DRAM weight-streaming bandwidth from a nominal 45-60 GB/s to a fixed 17-19 GB/s whenever a per-core kernel DMA transfer is an exact multiple of 1 MiB, affecting 7 of the 15 models in the ANEMLL inference project.
  • The author traced the cause through a 40-run D-dimension sweep, a core-count sweep that pinned the throttle to the per-core level, and an address-scrambling test (31.37 GB/s baseline versus 32.29 GB/s scrambled) that ruled out DRAM bank aliasing as the explanation.
  • The likely root cause is a 14-bit prefetch-ring pointer in the kernel DMA engine that lacks an epoch bit, so a transfer of an exact multiple of 0x4000 (16,384) lines aliases 'one full lap remaining' to 'empty' and stalls its own speculative lookahead.
  • Splitting a 1 MiB-per-core transfer into two 0x2000-line tasks restored 45.52 GB/s versus 17.25 GB/s unsplit, a 2.66x speedup, while splitting a transfer that was never a 1 MiB multiple produced no speedup at all.
  • The fix raised Llama 3.2 1B decode throughput from 10.0 to 24.3 tokens/s (DRAM usage 24.7 to 60.0 GB/s) and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage 22.4 to 48.7 GB/s), with no Apple firmware change involved.

Why it matters

The bug is invisible at the API level: kernel DMA still completes every transfer correctly, so nothing crashes and no error surfaces, it just runs at roughly a third of the chip's rated DRAM bandwidth whenever a weight-streaming transfer happens to land on a round number. Because 1 MiB per core is a common transfer size for FP16 weight tensors, the erratum quietly caps real-world token throughput for Apple Neural Engine inference stacks that stream weights this way, and the fix that recovers most of the lost bandwidth is a few lines of compiler-level chunking rather than a firmware update from Apple.

Who it affects

Anyone running LLM inference through the ANEMLL project's Apple Neural Engine backend on an M3 chip. The post lists seven of ANEMLL's fifteen models as hitting the bug: Llama 3.2 1B, Llama 3.1 8B, DeepSeek and DeepHermes 8B (their q/o and gate/up/down projections), DeepHermes 3B, Qwen3-8B (q/o and gate/up/down), and a shard of Gemma 3 4B's lm_head. Each is affected because its projection dimensions multiply out to an exact 1 MiB, or a multiple of it, of weight data per core.

How to use it

The workaround needs no Apple firmware update: any kernel DMA task that would total exactly 1 MiB (or a multiple) of weights per core is split into chunks that are not themselves multiples of 1 MiB, for example two 512 KiB transfers instead of one 1 MiB transfer. Yoon applied this by expressing an MLP's three 1x1 convolutions as two partial reductions so the compiler emits two 0x2000-line KernelDMA tasks instead of fusing them back into one 0x4000-line task; the sub-millisecond overhead of dispatching an extra task is negligible against the 30-50 GB/s recovered. A second, untested option is padding the transfer to sit roughly 8 KiB (256 lines) away from the 1 MiB boundary, which the author notes is more invasive since it changes the computation graph.

How solid is it

The diagnosis rests on controlled measurement rather than a single anecdote: a D-dimension sweep repeated across 40 runs under constant thermal and load conditions, a core-count sweep that pinned the throttle to the per-core level, an address-scrambling test that ruled out DRAM bank aliasing, and a register hexdiff confirming only the DMA address and size fields changed across the D values tested. An FFT of throughput against D found a resonance at exactly the 2048 period, and the per-lap timing curves scale linearly with the number of 1 MiB laps at an R-squared of 0.96 to 0.99. A control experiment then isolated the fix from mere chunking: splitting a transfer that was never a 1 MiB multiple produced no speedup at all, so the recovered bandwidth is specifically attributable to avoiding the 1 MiB boundary rather than to the act of splitting itself.

Risks and caveats

The root cause is the author's own hypothesis, introduced as 'most likely,' not a diagnosis Apple has confirmed; Apple is not quoted or cited anywhere in the post as acknowledging the erratum. All measurements were taken on one machine, an M3 Air, and the post makes no claim about whether M1, M2 or M4 Neural Engines share the bug. The published remedy is a software workaround that avoids the trigger condition, not a firmware or silicon fix, so the underlying RTL defect itself remains unpatched.

“This is not an RTL correctness bug, as kernel DMA still completes the transfer correctly. But the requests around 2048 are being forced into a separate, credit-starved issue regime, choking throughput by an unreasonable 28-43 GB/s.”

— Eileen Yoon