Multiverse Computing has reformulated the problem of pruning large language models (LLMs) as a physics optimization task, mapping block removal onto an Ising glass model. The approach, detailed in a new paper, treats each transformer block as a binary variable and calculates pairwise couplings between them using a second-order Taylor expansion. At 50% compression of Llama-3.3-70B-Instruct, the method gained almost 23 percentage points on the MMLU benchmark over the strongest competing block-removal technique, without any retraining.
Why block removal is a combinatorial problem
Deleting whole transformer blocks-also called depth pruning-is one of the cheapest ways to accelerate an LLM. The model literally gets shorter, producing predictable inference speedups that stack with quantization and low-rank compression. The difficulty lies in choosing which blocks to cut. The effect of removing one block depends on which others you also remove, making the choices interact in ways that simple ranking methods miss.
Most existing methods score each block independently using magnitude, sensitivity, or influence heuristics. In physics terms, these are mean-field approximations that treat each block's contribution as if it were independent of the others. "The trouble is that blocks are not independent, any more than spins in a real magnet are," the authors write. Removing block 20 hurts differently depending on whether block 19 or block 24 was also removed-an interaction that mean-field approaches discard.
Mapping pruning to an Ising Hamiltonian
The method attaches a binary variable to each block: 0 means keep it, 1 means remove it. A second-order Taylor expansion of the model's loss with respect to those variables produces an approximate Hessian matrix. The diagonal captures how much each block matters individually; the off-diagonal entries are the pairwise couplings between blocks.
This turns block selection into a constrained binary optimization problem: find the set of M blocks whose removal minimizes the energy, subject to removing exactly M of the N blocks. The energy of that spin system acts as a strong proxy for how well the pruned model will actually perform on benchmarks, meaning candidate configurations can be ranked cheaply without running the model. The same Hessian can be reused for many different compression targets since the couplings do not depend on M.
Solving with classical and quantum-inspired tools
For most models, the configuration space is large but still enumerable. Computing one energy is cheap enough that brute force on a single GPU can check tens of billions of spin configurations. The hardest tractable case-removing 8 of Llama-3.3-70B's 80 blocks, roughly 29 billion configurations-took about two days.
When enumeration becomes impossible, the problem's QUBO form can be handed to optimized classical and quantum-inspired solvers built for this class of Hamiltonian. An open-source tabu search solver reliably found the lowest-energy states in seconds, even on cases verified against brute force. The team notes that finding the true ground state is not required; generating a handful of good low-energy states is sufficient, which is a far easier bar for lightweight solvers.
Excited states often outperform the ground state
The energy proxy is strong but not perfect, so the single lowest-energy configuration is not always the best model. Once the Hamiltonian is set up, reading off the ground state and low-lying excited states is essentially free, yielding a spectrum of high-quality candidate prunings.
A concrete example emerged with Llama-3.1-8B-Instruct at 16 of 32 blocks removed. Most top states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state proposed removing a block near the beginning, and after light retraining that configuration outperformed the ground state across several benchmarks. "That directly disproves the common assumption that the best pruning is one consecutive chunk of middle-or-late blocks," the paper states.
Results across model scales
Across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct, the constrained binary optimization (CBO) method matched or exceeded state-of-the-art block-removal baselines, with the gap widening under aggressive compression. At 40 of 80 blocks removed from Llama-3.3-70B-Instruct, CBO held MMLU near 77 while the block influence baseline fell to 54.0. For Qwen3-14B at 12 of 40 blocks removed, CBO led MMLU by about 10 points.
The method also generalized to heterogeneous architectures. Applied to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model interleaving Mamba2, attention, and mixture-of-experts (MoE) layers, CBO found configurations that beat block influence on AIME25 and GPQA benchmarks. The formulation does not assume a homogeneous stack-a coupling is a coupling regardless of block type. The results confirmed that some expert layers are far more disposable than others, and the best configuration was again often an excited state rather than the ground state.
Why this matters for IT, development, and research teams
Block removal composes cleanly with the rest of a compression pipeline-quantization, low-rank compression, width pruning, and knowledge-distillation-based healing-so it slots into existing workflows rather than replacing them. The Hessian computation requires only forward and backward passes on a small calibration dataset, making it practical for teams that lack the resources to benchmark thousands of candidate prunings. For deeper understanding of LLM mechanics, the Generative AI Courses page covers foundational concepts relevant to model optimization. The code is open-sourced, and the full paper with derivation details, solver benchmarks, and complete results tables is available on Hugging Face.
Your membership also unlocks:
