Apple's MLX runs 816B-parameter Muse Spark Open on four Mac Studios

Apple's machine learning research site has published a guide to running large language models locally on Mac Studio with MLX, its open-source array framework for Apple silicon. The headline example is Muse Spark Open, a newly released open model that the post calls frontier-class, with 816 B parameters ("nearly a trillion"). Apple says that on a cluster of four Mac Studios the model generates over 36 tokens per second, which it describes as well above average human reading speed. The post says it uses the latest version of MLX, released the same day.

Apple benchmarked three models with the MLX LM package: Muse Spark Open (816 B, mixture of experts, run in 4-bit NVFP4), Mistral Medium 3.5 (128 B, dense, run in FP8) and Qwen 3.8 (27 B, dense, run in BF16). It compared two generations of Mac Studio, the recently released M5 Ultra and the earlier M3 Ultra, measuring time to first token (TTFT), generation throughput and peak power at 8,192 and 65,536 tokens of context. Apple chose those sizes because agentic coding tools often need large contexts. All numbers came from mlx_lm.benchmark.

On a single Mac Studio with up to 512GB of unified memory, Qwen 3.8 27B and Mistral Medium 3.5 128B fit comfortably, using up to 68GB and 158GB of RAM at the 65,536-token setup. TTFT is compute bound, and the M5 Ultra's GPU Neural Accelerators make it up to 4.6× faster than the M3 Ultra. Generation speed is memory-bandwidth bound: the M5 Ultra offers 1.2 TB/s against 819 GB/s for the M3 Ultra, and Apple reports 1.6× faster tokens per second. Peak power is about 200W for the M3 Ultra machine and 300W for the M5 Ultra machine, which the post compares to a fraction of a toaster's draw and says allows deployment in any home or office.

Muse Spark Open does not fit on one machine. It would need more than 1.6TB of RAM in its native BF16 precision for the parameters alone. In NVFP4 the footprint drops to about 493GB, and once memory for intermediate computation is added, it still cannot fit on a single Mac Studio.

The first fix is pipeline parallelization on two Macs: one holds the first half of the layers, the other the second half, and the first passes its output to the second. Apple calls this simple to implement. It relies on a low-latency RDMA driver over Thunderbolt 5 introduced in macOS 26.2, and on MLX v0.30.1 and later, which add jaccl communication collectives. The two machines are linked with two Thunderbolt cables to double link bandwidth, and jobs launch with the mlx.launch helper using the jaccl backend and the experimental MLX_METAL_FAST_SYNCH=1 flag for faster CPU-GPU synchronization. Decode speed is very similar to a single machine because layers still run sequentially. At 65,536 tokens, though, prefill is up to ~1.8× faster than on one node, because MLX splits the context into 8,192-token chunks and overlaps them across the two machines.

The second approach is tensor parallelization, which splits each layer's width across machines. It is harder to implement and involves many small communication operations per token and layer, so a very low-latency link such as Thunderbolt RDMA through jaccl is critical for any speedup. The machines form a Thunderbolt mesh, each pair joined by two cables. Apple reports that dense models scale better across multiple computers than the mixture-of-experts model. For Muse Spark Open, separate normalization of the experts' outputs adds prefill communication, so at 65,536 tokens two-machine tensor parallelization lags two-machine pipeline parallelization; the approach pays off with four M5 Ultras. Decode is unaffected and shows a significant speedup, comparable to the dense models. On four M5 Ultras at 65,536 tokens, two cables per link cut TTFT from about 50.5s (one cable) to 40.6s, the figure reported in Table 2.

The post ends with local agentic use. Launching mlx_lm.server, optionally prefixed with mlx.launch to spread it over several Macs, starts a local server on a port such as 8080. Tools like OpenCode can be pointed at it with a short config using an OpenAI-compatible provider. In the conclusion, Apple says it has spent the past year on MLX support for the M5 Neural Accelerators, RDMA over Thunderbolt across Macs, and efficient mixture-of-experts implementations.

Key facts

On four Mac Studios, Muse Spark Open (816 B parameters, mixture of experts, NVFP4) generates over 36 tokens per second, according to Apple's post.
The model needs more than 1.6TB of RAM in BF16 and about 493GB in NVFP4, so it cannot fit on one 512GB Mac Studio and must be split across machines.
M5 Ultra vs M3 Ultra: up to 4.6× faster time to first token, 1.6× faster generation, with memory bandwidth of 1.2 TB/s vs 819 GB/s and peak power of about 300W vs 200W.
Clustering relies on macOS 26.2's RDMA driver over Thunderbolt 5 and MLX v0.30.1 or later with jaccl; two cables per link cut Muse Spark Open TTFT on four M5 Ultras from about 50.5s to 40.6s at 65,536 tokens.
A local server started with mlx_lm.server can power agentic tools such as OpenCode.

Why it matters

Apple is showing that a model of nearly a trillion parameters can run on desktop machines in a private setting, with generation speed it calls well above human reading speed. The enabling pieces are an RDMA driver over Thunderbolt 5 in macOS 26.2 and MLX's jaccl collectives, which make small, frequent communication between Macs cheap enough for tensor parallelization to help. The post also gives a rule of thumb: prefill (TTFT) is compute bound and benefits from the M5 Ultra's Neural Accelerators, while generation is memory-bandwidth bound.

Who it affects

AI researchers and developers who want to experiment locally with new models and methods, which is the audience the post names. Developers who run agentic coding tools with large contexts are called out specifically. Owners of M3 Ultra or M5 Ultra Mac Studios are the ones with benchmark numbers to compare against.

How to use it

Install the latest MLX and MLX LM, run macOS 26.2 or later, and link the Macs with Thunderbolt 5 cables (the post uses two per pair, in a mesh). Launch with mlx.launch --hostfile hostfile.json --backend jaccl, add --env MLX_METAL_FAST_SYNCH=1, then run mlx_lm.benchmark or mlx_lm.server on the model. Add --pipeline for pipeline parallelization; leave it off for tensor parallelization. For agent use, start mlx_lm.server (for example on port 8080) and point OpenCode at http://127.0.0.1:8080/v1 through an OpenAI-compatible provider entry. The post gives no price for the Macs or for a four-machine cluster.

How solid is it

This is a first-party post from Apple's machine learning research site, and the numbers come from the mlx_lm.benchmark tool with the commands shown. Several figures are stated as maximums, such as the 4.6× TTFT gain and the ~1.8× prefill gain. The 36 tokens per second headline sentence does not give a context size, and the detailed TTFT and decode numbers sit in tables and figures that the text does not reproduce. The cluster results are on M5 Ultra machines in the tensor-parallel setup.

Risks and caveats

The benchmarks are Apple's own, on Apple hardware, and the text includes no comparison with GPU servers or cloud inference. Tensor parallelization depends on a very low-latency link; without Thunderbolt RDMA it does not yield a speedup. Scaling is uneven: dense models scale better than Muse Spark Open, and at 65,536 tokens two-machine tensor parallelization trails two-machine pipeline parallelization for that model. The model runs in 4-bit NVFP4, and the MLX_METAL_FAST_SYNCH flag is described as experimental. Peak power rises from about 200W on M3 Ultra to 300W on M5 Ultra.

“On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second”

— Apple ML Research, Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio