像物理学家一样剪枝 LLM：把块移除变成伊辛优化问题

One of the cheapest ways to make a large language model faster is also one of the bluntest: delete whole transformer blocks. Because the model literally gets shorter, block removal (also called depth pruning) buys predictable inference speedups on top of the memory savings, and it stacks cleanly with quantization, low-rank compression, and other techniques. The hard part is deciding which blocks to cut. Remove the wrong ones and the model collapses; and the effect of removing any one block depends on which others you remove alongside it, so the choices interact. That makes it a combinatorial problem, not a ranking problem, and combinatorial problems with interacting binary variables are exactly what the physics of spin systems was built to describe.

让大模型变快最便宜的办法之一，也是最粗暴的办法之一：整块删掉 transformer 层。模型字面意义上变短了，块移除（block removal，也叫深度剪枝 depth pruning）在省显存之外还买来可预期的推理加速，而且与量化、低秩压缩等技术干净地叠加。难的是决定剪哪些块：剪错几块模型直接垮掉；而剪掉任何一块的影响，取决于你同时剪了哪些别的块——选择之间相互作用。这使它成为组合问题而非排序问题，而「带相互作用的二元变量的组合问题」，恰恰是自旋系统物理学天生要描述的东西。

Our latest paper, LLM Compression by Block Removal with Constrained Binary Optimization, takes that correspondence literally. We reformulate block selection as a constrained binary optimization (CBO) problem that maps directly onto an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of "up" spins. The energy of that spin system turns out to be a strong, cheap proxy for how well the pruned model will actually score on benchmarks, which means we can rank a huge number of candidate configurations without benchmarking any of them, and hand the hard instances to the same classical and quantum-inspired solvers we use elsewhere at Multiverse. The payoff in the deep-compression regime is large: at 50% compression of Llama-3.3-70B-Instruct, we gain almost 23 percentage points on MMLU over the best competing block-removal method.

我们最新的论文《LLM Compression by Block Removal with Constrained Binary Optimization》把这种对应关系当了真：把块选择重新表述为一个受约束二元优化（CBO）问题，直接映射到伊辛玻璃（Ising glass）——一种全连接相互作用、「向上」自旋数目固定的无序自旋系统。这个自旋系统的能量，被证明是「剪枝后模型基准得分」的强而便宜的代理指标：我们可以给海量候选配置排序而一个基准都不用跑，把困难实例交给我们 Multiverse 在其他场合也在用的经典与量子启发求解器。深压缩区间的回报巨大：Llama-3.3-70B-Instruct 压缩 50% 时，MMLU 比最好的竞争性块移除方法高出近 23 个百分点。 译注 译注译注：Multiverse Computing 是论文团队所在公司，压缩产品线为 CompactifAI——文中「我们在其他场合也在用的求解器」，即指该公司为经典/量子启发优化自建的 solver 栈。

Most existing block-removal methods score each block on its own, then remove the ones that look least important, using magnitude, sensitivity, or "block influence" heuristics. In physics terms these are mean-field methods: they treat each block as if its contribution were independent of the others, the way mean-field theory replaces a spin's neighbors with a single averaged field. A related shortcut is to only ever remove a single consecutive run of blocks, which keeps the problem small but throws away most of the search space.

现有的块移除方法大多给每块单独打分，再用幅值、敏感度或「块影响力」之类的启发式，删掉看起来最不重要的那些。用物理的话说，这些都是平均场（mean-field）方法：把每块当作贡献彼此独立来处理，就像平均场理论把一个自旋的邻居换成单一平均场。另一个相关捷径是只删连续的一段块——问题规模小了，但搜索空间也扔掉了大半。

The trouble is that blocks are not independent, any more than spins in a real magnet are. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24, an interaction, or coupling, between the two decisions. As models get deeper and more heterogeneous, ignoring those couplings leaves quality on the table, especially when you want to remove a lot of blocks at once. What you really want is to search over combinations of blocks while accounting for how they interact, but the number of combinations grows exponentially, so brute force looks hopeless. This is precisely the regime, exponentially large configuration spaces with pairwise couplings, where the tools of statistical physics earn their keep.

麻烦在于，块之间并不独立，正如真磁铁里的自旋也不独立。剪掉第 20 块伤不伤模型，取决于你是否也剪了第 19 或第 24 块——这是两个决策之间的相互作用（interaction），即耦合（coupling）。模型越深越异构，无视这些耦合浪费的质量越多，尤其当你想一次剪掉大量块时。你真正想要的是在块的组合空间里搜索、同时把相互作用算进去——但组合数指数增长，暴力枚举看似无望。而这恰恰是统计物理的工具挣饭吃的地盘：指数大的配置空间加两两耦合。

难点 难点提示全文唯一需要线代的地方。Hessian=「损失对每个变量多敏感、敏感度之间怎么互相牵动」的矩阵——对角线是自己，非对角是别人。 We attach a binary variable to each transformer block: 0 means keep it, 1 means remove it, just like a spin that can point down or up. Then we do a second-order Taylor expansion of the model's loss with respect to those variables, which produces an (approximate) Hessian matrix. The diagonal of that Hessian is how much each block matters on its own; the off-diagonal entries are exactly the pairwise couplings between blocks, the many-body physics that mean-field methods throw away.

给每个 transformer 块挂一个二元变量：0 保留、1 移除，就像一个可上可下的自旋。然后对模型损失做关于这些变量的二阶泰勒展开，得到一个（近似的）Hessian 矩阵：对角线是每块自身的贡献，非对角元正是块与块之间的两两耦合——被平均场方法扔掉的那部分多体物理。

That reformulation turns "which blocks should I remove?" into a clean optimization: find the set of M blocks whose removal minimizes the energy xᵀH⁰x, subject to removing exactly M of the N blocks. Mathematically this is a constrained binary optimization problem; physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization (the fixed number of removed blocks plays the role of a fixed total spin). The key property we establish is that this energy is a strong proxy for downstream quality: low-energy states of the spin system correspond to high-performing pruned models. Minimizing energy and maximizing benchmark score become the same search.

这一重述把「我该剪哪些块」变成一个干净的优化问题：在 N 块里恰好移除 M 块的约束下，找到让能量 xᵀH⁰x 最小的那组块。数学上这是受约束二元优化；物理上这是一块伊辛玻璃——全连接、磁化强度守恒的自旋系统（固定移除块数扮演固定总自旋的角色）。我们确立的关键性质是：这个能量是下游质量的强代理——自旋系统的低能态对应高性能的剪枝模型。最小化能量与最大化基准分数成了同一场搜索。

Block selection becomes a constrained binary optimization problem, equivalent to finding low-energy states of an Ising glass; each solution says which M of N blocks to delete. Right: the coupling variable α we insert into each block's residual path to build the Hessian. Source: paper Figure 1.

块选择成为受约束二元优化问题，等价于寻找伊辛玻璃的低能态；每个解指明 N 块中删哪 M 块。右图：我们插入各块残差路径、用于构造 Hessian 的耦合变量 α。来源：论文图 1。

The reason this is practical is cost. The Hessian, i.e. the full set of couplings, is computed just once, from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation, no need to run the actual model, let alone benchmark it. And because the couplings don't depend on the compression target, the same Hessian can be reused to solve for many different values of M.

这套方法 practical 的原因是成本：Hessian（即全部耦合）只需算一次——在小校准集上做几次前向反向传播。之后评估任何候选配置都只是一次便宜的能量计算，不用跑真实模型，更不用说跑基准。而且耦合不依赖压缩目标，同一个 Hessian 可以复用去解各种不同的 M。

For most models the configuration space is large but still checkable. Because computing one energy is so cheap, we brute-force it on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds; the hardest tractable case here, removing 8 of Llama-3.3-70B's 80 blocks (about 29 billion configurations), took roughly two days.

对多数模型，配置空间巨大但仍在可查范围内。因为算一次能量太便宜了，我们直接在单张 GPU 上暴力枚举，最多查验了数百亿个自旋配置。几百万个只要几秒；这里最难的可解案例——从 Llama-3.3-70B 的 80 块里移除 8 块（约 290 亿种配置）——花了大约两天。

难点 难点提示QUBO 就是把「恰好删 M 块」的约束塞进惩罚项后的等价二次优化形式——物理界已为这类哈密顿量造好一堆求解器，这里是借东风。 Beyond that the exact approach breaks down, and this is where casting the problem as an Ising glass pays off a second time. In its equivalent QUBO form (the constraint absorbed into a penalty term), the exact same task can be handed to the highly optimized classical, quantum, and quantum-inspired solvers built for this class of Hamiltonian, the machinery of quantum annealing, QAOA, tabu search, and specialized branch-and-bound. We find that an open-source tabu solver reliably reaches the lowest-energy states in seconds, even on the hardest cases we can verify against brute force.

超过这个规模精确方法就失效了——而这正是「表述成伊辛玻璃」第二次兑现的地方。写成等价的 QUBO 形式（约束吸收进惩罚项）后，同一个任务可以交给为这类哈密顿量打造的高度优化的经典、量子与量子启发求解器：量子退火、QAOA、禁忌搜索、专门的分支定界。我们发现一个开源禁忌求解器在几秒内就能稳定到达最低能态，即便在我们能用暴力枚举验证的最难案例上也如此。

难点 难点提示与优化界直觉相反：不求精确基态，只要快速产出一小把好的低能态——门槛低几个量级，轻量求解器因此够用。 There's a subtle but important point here, and it runs against the usual grain of optimization. Normally a CBO or annealing solver is judged by whether it finds the true ground state. We don't actually need the ground state. What we need is a fast way to generate a handful of good low-energy states, and that is a far easier bar, which is why lightweight solvers work so well for us and why we can afford to run several of them.

这里有个微妙但重要的点，而且与优化界 usual 的评判标准相反：CBO 或退火求解器通常以「是否找到真正基态」论英雄。但我们并不需要基态——我们要的是快速产出一小把好的低能态，这个门槛低得多。这就是为什么轻量求解器对我们如此好用、为什么我们跑得起好几个。

The energy is a strong proxy for quality, but not a perfect one, so the single lowest-energy state isn't always the best model. This turns out to be a feature, not a bug: once the Hamiltonian is set up, reading off the ground state and the low-lying excited states is essentially free, giving a spectrum of high-quality candidate prunings to try rather than one fragile answer. Exploring excited states, not just the ground state, is itself an area of active physics research, and it maps neatly onto what practitioners actually need here.

能量是质量的强代理，但不是完美代理，所以单一最低能态并不总是最好的模型。这反而是特性不是缺陷：哈密顿量一旦建好，读出基态和低激发态几乎是免费的——给你的是一谱系高质量候选剪枝方案，而不是一个脆弱的答案。探索激发态（而不只是基态）本身就是物理学的研究前沿，而它恰好对上了从业者的真实需要。

难点 难点提示全文最反直觉的结果，值得放慢：能量只是质量的代理而非完美代理，只盯着基态会漏掉好方案。 A concrete example: for Llama-3.1-8B-Instruct at 16/32 blocks removed, most of the top states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state is the first to propose removing a block near the beginning of the model, and after light retraining that configuration outperforms the ground state across several benchmarks. That directly disproves the common assumption that the best pruning is one consecutive chunk of middle-or-late blocks, and it shows why respecting the full many-body structure of the problem pays off.

一个具体例子：Llama-3.1-8B-Instruct 移除 16/32 块时，多数头部低能态都按先前工作的预期剪模型尾部；但第 17 激发态首次提出剪掉靠近模型开头的一块——轻量重训后，这个配置在多个基准上反超基态。这直接证伪了「最佳剪枝是中后段连续一整段」的通行假设，也说明了尊重问题完整多体结构的价值。

Left: which blocks each of the 20 lowest-energy states removes (red = removed). Right: the 17th excited state, which removes an early block, beats the ground state on several benchmarks after retraining. The best model is an excited state, not the ground state. Source: paper Figure 2.

左：20 个最低能态各自剪了哪些块（红=剪除）。右：第 17 激发态剪掉一块早期层，重训后在多个基准上胜过基态。最好的模型是个激发态，不是基态。来源：论文图 2。

Across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct, our method (CBO) is on par with or better than state-of-the-art block-removal baselines, and the gap widens as compression gets more aggressive.

在 Llama-3.1-8B-Instruct、Qwen3-14B、Llama-3.3-70B-Instruct 三款模型上，我们的方法（CBO）持平或优于最先进的块移除基线，且压缩越激进差距越大。

The clearest win is deep compression of Llama-3.3-70B-Instruct, evaluated without retraining. Up to 24 of 80 blocks removed, CBO is roughly on par with block influence. But at 32/80 and 40/80, it pulls decisively ahead, with an almost 23-point MMLU advantage at the deepest setting, where it beats the baseline on every benchmark we tested. For Qwen3-14B at 12/40 removed, CBO leads MMLU by about 10 points. At lighter compression the methods are comparable, which is expected: the couplings matter most when you're cutting deep.

最漂亮的胜利是 Llama-3.3-70B-Instruct 的深压缩（不重训评估）：剪到 24/80 块为止 CBO 与 block influence 大致持平；但到 32/80 和 40/80 它果断拉开——最深档 MMLU 领先近 23 个百分点，且在我们测的每个基准上都赢过基线。Qwen3-14B 剪 12/40 块时 CBO 的 MMLU 领先约 10 个百分点。轻压缩下各方法相当，这在预期内：耦合只在剪得深时才最要命。

At 40/80 (50% depth), CBO holds MMLU near 77 while the strongest baseline falls to the mid-50s. Source: paper Table 2.

在 40/80（50% 深度）处，CBO 把 MMLU 守在 77 附近，而最强基线跌到 55 上下。来源：论文表 2。

Block removal gets much harder on modern heterogeneous architectures, where different block types are interleaved, and the Ising formulation doesn't care: a coupling is a coupling regardless of what kind of block sits at each site. To stress-test that, we applied the method to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model that interleaves Mamba2, attention, and mixture-of-experts (MoE) layers in a non-uniform pattern, without any retraining.

在现代异构架构上块移除难得多——不同类型的块交错排布——而伊辛表述毫不在乎：不管每个格点上坐的是什么块，耦合就是耦合。为压力测试，我们把方法用在 NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 上——一个以非均匀模式交错排布 Mamba2、注意力与混合专家（MoE）层的混合模型——全程零重训。

Nothing about our formulation assumes a homogeneous stack, so it transfers directly. Removing 2–3 MoE layers or 2 attention layers, CBO finds configurations that beat block influence on AIME25 and GPQA. The results also confirm that redundancy in these hybrid models is real but unevenly distributed: some expert layers are far more disposable than others, and the method's ability to search the coupled configuration space is what locates the good cuts. Even here, the pattern from the dense models holds, the best configuration is often an excited state rather than the ground state.

我们的表述不假设同构堆叠，所以直接迁移：剪 2–3 个 MoE 层或 2 个注意力层，CBO 找到的配置在 AIME25 与 GPQA 上胜过 block influence。结果还证实：混合模型的冗余真实存在但分布不均——有些专家层远比其他的可弃，而方法在耦合配置空间里搜索的能力，正是找准下刀处的关键。即便在这里，稠密模型的规律依旧成立：最好的配置常常是激发态而非基态。

Reframing a messy machine-learning problem as an Ising Hamiltonian, then solving it with the classical and quantum-inspired optimization machinery built for physics, is squarely in Multiverse's wheelhouse, it's the same instinct that runs through our compression stack. And block removal composes with the rest of that stack, quantization, low-rank/SVD compression, width pruning, and knowledge-distillation-based healing, so it slots into a larger pipeline rather than competing with it.

把一团乱的机器学习问题重述成伊辛哈密顿量、再用为物理学打造的经典与量子启发优化机器去解，正是 Multiverse 的本行——我们的压缩技术栈里贯穿的就是同一个直觉。块移除与栈里其余部分干净叠加：量化、低秩/SVD 压缩、宽度剪枝、基于知识蒸馏的愈合——它嵌进更大的流水线，而不是与谁竞争。

Want the full technical details, including the Taylor-expansion derivation, the QUBO mapping, the solver benchmarks, the calibration-dataset ablations, and the complete results tables? Read the full paper on Hugging Face, or get in touch with our team to talk about applying this to your own models. The code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization.

想要完整技术细节——泰勒展开推导、QUBO 映射、求解器基准、校准集消融与完整结果表？在 Hugging Face 读全文，或联系我们的团队聊聊怎么用到你自己的模型上。代码已开源：github.com/CompactifAI/Block_removal_through_constrained_binary_optimization。