Multiverse Computing prunes LLMs using an Ising spin-glass optimization

Multiverse Computing has published a paper, "LLM Compression by Block Removal with Constrained Binary Optimization," that reframes a practical engineering problem in physics terms. Existing methods for deciding which transformer blocks to remove from a large language model score each block in isolation, using heuristics such as magnitude or "block influence," then cut the ones that look least important. Multiverse's team calls this a mean-field approximation: it ignores that whether removing block 20 hurts the model depends on whether block 19 or block 24 was also removed.

The paper's fix is to attach a binary variable to each transformer block, 0 to keep it and 1 to remove it, like a spin pointing down or up. The team then takes a second-order Taylor expansion of the model's loss to get an approximate Hessian matrix. The diagonal of that Hessian scores each block on its own; the off-diagonal entries capture the pairwise couplings between blocks that mean-field methods discard. The resulting task, find the M blocks whose removal minimizes the energy of that Hessian, is mathematically a constrained binary optimization problem and physically an Ising spin glass with a fixed number of "up" spins. The Hessian is computed once, from forward and backward passes on a small calibration set; after that, scoring any candidate configuration is a single cheap energy calculation rather than an actual benchmark run.

For configuration spaces small enough to check, the team brute-forces the search on a single GPU, up to tens of billions of configurations. The hardest tractable case, removing 8 of Llama-3.3-70B-Instruct's 80 blocks (about 29 billion configurations), took roughly two days. Beyond that, the same problem in its QUBO form is handed to classical, quantum, and quantum-inspired solvers built for Ising-type problems; an open-source tabu solver reliably found the lowest-energy states in seconds, even on the hardest cases the team could still verify against brute force. The authors say they do not need the true ground state, only a handful of good low-energy states, which is why lightweight solvers are enough.

Reading off the full low-energy spectrum, not just the ground state, turned out to matter. For Llama-3.1-8B-Instruct with 16 of 32 blocks removed, the 17th excited state is the first configuration to remove a block near the start of the model, and after light retraining it outperforms the ground state across several benchmarks. The team says this disproves the common assumption that the best pruning is a single consecutive chunk of middle-or-late blocks.

On results, across Llama-3.1-8B-Instruct, Qwen3-14B and Llama-3.3-70B-Instruct, the method, called CBO, matches or beats state-of-the-art block-removal baselines, with the advantage widening at deeper compression. Up to 24 of Llama-3.3-70B-Instruct's 80 blocks removed, CBO is roughly on par with the block-influence baseline; at 32/80 and especially 40/80 (50% depth, evaluated without retraining) it pulls decisively ahead, holding MMLU near 77 versus the mid-50s for the strongest baseline, an advantage of almost 23 percentage points, and beating that baseline on every benchmark tested at that setting. For Qwen3-14B with 12 of 40 blocks removed, CBO leads MMLU by about 10 points. The team also tested NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model interleaving Mamba2, attention and mixture-of-experts layers, without retraining; removing 2 to 3 MoE layers or 2 attention layers, CBO found configurations that beat block influence on the AIME25 and GPQA benchmarks. The team says redundancy in such hybrid models is real but unevenly distributed, with some expert layers far more disposable than others.

The code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization, and block removal is presented as one stage of Multiverse's broader compression stack, alongside quantization, low-rank/SVD compression, width pruning and distillation-based healing.

Key facts

Frames transformer block pruning as a constrained binary optimization problem mathematically equivalent to finding low-energy states of an Ising spin glass, capturing the pairwise couplings between blocks that prior "mean-field" scoring methods ignore.
At 50% compression of Llama-3.3-70B-Instruct (40 of 80 blocks removed, no retraining), the method holds MMLU near 77 versus the mid-50s for the strongest baseline, an advantage of almost 23 percentage points.
The coupling matrix (an approximate Hessian) is computed once from a small calibration set; after that, scoring a candidate configuration is a single cheap energy calculation, and an open-source tabu solver finds near-optimal configurations in seconds where brute force on the hardest tractable case took about two days for roughly 29 billion configurations.
The best pruning found is often not the lowest-energy "ground state": for Llama-3.1-8B-Instruct with 16 of 32 blocks removed, the 17th excited state, which cuts a block near the start of the model, outperforms the ground state after light retraining.
The method also generalizes to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid Mamba2/attention/MoE model, beating the block-influence baseline on AIME25 and GPQA; code is open-sourced on GitHub.

Why it matters

Most block-removal methods score each transformer block on its own and cut the weakest scorers, a mean-field shortcut that ignores how blocks interact: removing block 20 can hurt more or less depending on whether block 19 or 24 was also cut. Multiverse Computing's paper captures those interactions directly by turning block selection into a constrained binary optimization problem physically equivalent to an Ising spin glass, with an approximate Hessian supplying both each block's individual importance and its pairwise couplings with every other block. The energy of a candidate configuration is a cheap proxy for its benchmark quality, so a huge number of configurations can be ranked without running the model at all. The payoff shows up specifically in the deep-compression regime, where accounting for couplings matters most and the gap over prior methods is largest.

Who it affects

Anyone deploying large language models under memory, latency or cost constraints, since deeper, higher-quality compression means smaller models that keep more of the original's capability. It matters most where compression needs to go deep, such as removing half a model's blocks, a regime where the paper's method separates itself most from prior block-removal baselines. It is also directly relevant to Multiverse Computing's own business: block removal is presented as one stage in a larger compression pipeline the company already offers, alongside quantization, low-rank/SVD compression, width pruning and distillation-based healing.

How to use it

The full paper and code are public: the code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization, and the blog post points to the full paper on Hugging Face for the Taylor-expansion derivation, the QUBO mapping, solver benchmarks and complete results tables. Applying the method requires computing the Hessian once from forward and backward passes on a small calibration dataset; from there, smaller search spaces (up to tens of billions of configurations) can be brute-forced on a single GPU, while larger ones are handed to classical, quantum or quantum-inspired solvers built for Ising-type problems, with an open-source tabu solver cited as reliably finding near-optimal configurations in seconds. The source gives no pricing or licensing terms beyond the open-source code release.

How solid is it

The method is benchmarked across three dense models, Llama-3.1-8B-Instruct, Qwen3-14B and Llama-3.3-70B-Instruct, at multiple compression depths against state-of-the-art block-removal baselines, plus a fourth, heterogeneous model, NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, that interleaves Mamba2, attention and mixture-of-experts layers. Results are cross-checked in two ways: the low-energy configurations an open-source tabu solver finds are verified against brute-force search on cases small enough to enumerate exhaustively, and specific pruned configurations are evaluated on concrete benchmarks such as MMLU, AIME25 and GPQA rather than on the energy proxy alone. The code and paper are public, so the claims are independently checkable, though the results as published are the company's own and the source does not mention independent replication.

Risks and caveats

The results come from Multiverse Computing itself, describing its own method against baselines it selected, with no independent replication mentioned in the source. The headline 50%-compression result on Llama-3.3-70B-Instruct is reported without retraining; the paper's other retraining example, on Llama-3.1-8B-Instruct, involves only "light retraining," and it is not stated whether that step was applied at the deepest compression settings on the larger model. The reported gains are measured on specific benchmarks, chiefly MMLU plus AIME25 and GPQA for the hybrid model, and the source does not say how the method performs on tasks outside those benchmarks. Brute-force verification is only possible up to search spaces of roughly tens of billions of configurations, itself a two-day computation on a single GPU for the hardest case tested; beyond that scale, the method relies on heuristic solvers rather than a provable optimum. The source gives no author names, no publication or code-release date, and no cost figures for running the method.

“We don't actually need the ground state. What we need is a fast way to generate a handful of good low-energy states, and that is a far easier bar”

— Multiverse Computing, in the blog post