Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. These scores are often treated as precise measurements of relative model capability and increasingly inform decisions about which models to deploy. However, we've found that infrastructure configuration alone can produce differences that exceed those margins. In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).
SWE-bench、Terminal-Bench 这类 Agent 编程基准常用来对比前沿模型的软件工程能力——排行榜头部名次之间往往只差几个百分点。这些分数常被当作模型相对能力的精确度量，并日益影响「部署哪个模型」的决策。但我们发现：光是基础设施配置，就能制造超过这些名次间隔的分差。内部实验中，Terminal-Bench 2.0 上资源配置最松与最严的两套环境，成功率相差 6 个百分点（p < 0.01）。
Static benchmarks score a model's output directly—the runtime environment doesn't factor into the result. Agentic coding evals are different: models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns. The runtime is no longer a passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test.
静态基准直接给模型输出打分——运行时环境不进入结果。Agent 编程评测不是这样：模型拿到的是一个完整环境，在其中写程序、跑测试、装依赖、多轮迭代。运行时不再是被动的容器，而是解题过程的组成部分。两个资源配额和时间限制不同的 Agent，考的压根不是同一张卷子。
Eval developers have begun accounting for this. Terminal-Bench 2.0, for instance, specifies recommended CPU and RAM on a per-task basis in their latest 2.0 release. However, specifying resources isn't the same as enforcing them consistently. Moreover, we discovered that enforcement methodology can change what the benchmark ends up actually measuring.
评测开发者已经开始正视这一点。比如 Terminal-Bench 2.0 在最新发布里按任务粒度规定了推荐的 CPU 和内存。但「规定资源」不等于「一致地强制执行」。更进一步的发现是：强制执行的方式本身，会改变这个基准实际测到的东西。
How we got here · 问题的由来
We run Terminal-Bench 2.0 on a Google Kubernetes Engine cluster. While calibrating the setup, we noticed our scores didn't match the benchmark's official leaderboard, and infra error rates were surprisingly high: as many as 6% of tasks were failing because of pod errors, most of which were unrelated to the model's ability to solve the tasks.
我们在 Google Kubernetes Engine 集群上跑 Terminal-Bench 2.0。校准环境时注意到：我们的分数对不上官方排行榜，而且基础设施错误率高得反常——多达 6% 的任务因 pod 错误而失败，其中大多数与模型解题能力无关。
The discrepancy in scores came down to enforcement. Our Kubernetes implementation treated the per-task resource specs as both a floor and a hard ceiling: each container was guaranteed the specified resources but killed the moment it exceeded them. Container runtimes enforce resources via two separate parameters: a guaranteed allocation—the resources reserved up front—and a hard limit at which the container is killed. When these are set to the same value, there's zero headroom for transient spikes: a momentary memory fluctuation can OOM-kill a container that would otherwise have succeeded. To account for this, Terminal-Bench's leaderboard uses a different sandboxing provider, whose implementation is more lenient, allowing temporary overallocation without terminating the container in order to favor infrastructural stability.
分数差异的根源在强制执行方式。我们的 Kubernetes 实现把按任务的资源规格同时当作下限和硬上限：每个容器保证拿到规定资源，但一超过就被杀掉。容器运行时用两个独立参数执行资源限制：一个是保障配额（guaranteed allocation，预先预留的资源），一个是容器被杀的硬上限（hard limit）。两者设成同一个值时，对瞬时尖峰就是零余量：一次短暂的内存波动就能把本可成功的容器 OOM 杀掉。为此，Terminal-Bench 官方排行榜用的是另一家沙箱提供商，其实现宽松得多——允许临时超额而不终止容器，以照顾基础设施稳定性。
This finding raised a larger question: how much does resource configuration impact evaluation scores?
由此引出一个更大的问题：资源配置到底能把评测分数影响多少？
To quantify the effect of the scaffold, we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped. Everything else stayed constant: same Claude model, same harness, same task set.
为量化 scaffold（脚手架）的这种效应，我们在六档资源配置下跑了 Terminal-Bench 2.0——从严格执行按任务规格（1x，下限即上限），到完全不限量。其余一切不变：同一个 Claude 模型、同一个 harness、同一套任务。
In our experiments, success rates increased with resource headroom. This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped. The drop between strict enforcement to 3x headroom (5.8% to 2.1%) was significant at p < 0.001. With more headroom, fewer containers get killed for exceeding their allocation.
实验中，成功率随资源余量单调上升。主因是基础设施错误率逐档单调下降：从严格执行的 5.8% 降到不限量时的 0.5%；从严格执行到 3 倍余量这一段（5.8% 降到 2.1%）显著性达 p < 0.001。余量越足，因超额被杀的容器越少。
From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40). Most of the tasks that were crashing at 1x would have failed regardless—which is something that we observed in the data. The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution.
从 1x 到 3x，成功分数只在噪声区间内波动（p=0.40）。1x 下崩溃的大多数任务本来就做不出来——这是数据里能看到的：Agent 探索一番、撞上资源墙、被抢占，但它从一开始就不在通往正确解的路径上。
Starting around 3x, however, this trend changes: success rates climb faster than infra errors decline.
然而从 3x 附近开始，趋势变了：成功率的爬升快于基础设施错误率的下降。
Between 3x to uncapped, infra errors drop an additional 1.6 percentage points, while success jumps almost 4 percentage points. The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. At uncapped resources, the total lift over 1x is +6 percentage points (p < 0.01). At the margins, tasks like rstan-to-pystan and compile-compcert significantly improve their success rates when getting memory headroom.
从 3x 到不限量，基础设施错误率只再降 1.6 个百分点，成功率却跳升近 4 个百分点。多出来的资源让 Agent 得以尝试那些只在宽裕配额下才行得通的路径：拉取大型依赖、派生昂贵的子进程、跑吃内存的测试套件。不限量时，相对 1x 的总提升是 +6 个百分点（p < 0.01）。具体到任务层面，rstan-to-pystan、compile-compcert 这类任务在拿到内存余量后成功率显著改善。
How this affects measurement · 这如何影响测量
Up to roughly 3x Terminal-Bench specs, the additional resources fix infrastructure reliability problems, namely transient resource spikes. The sandboxing provider used by the Terminal-Bench maintainers is implicitly doing this behind the scenes; the eval gets more stable without getting easier.
到大约 3 倍 Terminal-Bench 规格为止，追加的资源修的是基础设施可靠性问题，也就是瞬时资源尖峰。Terminal-Bench 维护方用的沙箱提供商在幕后默默做的正是这件事：评测变稳了，但没有变简单。
Above the 3x mark, however, additional resources start actively helping the agent solve problems it couldn't solve before, which shows that limits can actually change what the eval measures. Tight limits inadvertently reward very efficient strategies, while generous limits are more forgiving and reward agents that can better exploit all available resources.
但过了 3x 这条线，追加的资源就开始主动帮 Agent 解出以前解不出的问题——这说明资源限制真的能改变评测测的东西。紧限制无意中奖励了高效策略；宽限制更宽容，奖励的是更会榨取全部可用资源的 Agent。
An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.
快速写出精简高效代码的 Agent，在紧约束下表现好；用重型工具暴力破解的 Agent，在宽约束下表现好。两者都是正当的测试对象，但若不注明资源配置就把它们压进一个分数，差异——以及真实世界的泛化性——就无从解读。
On bn-fit-modify, a Terminal-Bench task requiring Bayesian network fitting, some models' first move is to install the standard Python data science stack: pandas, networkx, scikit-learn, and all their toolchain. Under generous limits, this works. Under tight ones, the pod runs out of memory during installation, before the agent writes a single line of solution code. A leaner strategy exists (implementing the math from scratch using only the standard library), and some models do default to it. Others don't. Different models have different default approaches, and the resource configuration determines which of those approaches happen to succeed. We replicated the core finding across different Anthropic models. The direction of the effect was consistent, while the magnitude varied. The same trends seem to hold on models other than Claude, but we haven't rigorously tested them.
在 bn-fit-modify（一个要求做贝叶斯网络拟合的 Terminal-Bench 任务）上，有些模型的第一反应是装标准 Python 数据科学全家桶：pandas、networkx、scikit-learn 及其整条工具链。宽限制下，这样能成；紧限制下，pod 在安装阶段就内存耗尽——Agent 还没写下一行解题代码。更精简的策略是存在的（只用标准库从零实现数学部分），有些模型默认就走这条路，有些不走。不同模型有不同的默认打法，而资源配置决定了哪些打法恰好能活下来。我们在多代 Anthropic 模型上复现了核心结论：效应方向一致、幅度有差异。非 Claude 模型上趋势似乎同样成立，但没有严格测过。
We also tested whether this pattern holds on evals outside Terminal-Bench by running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each. The same effect held, though the magnitude was smaller: Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x. SWE-bench tasks are less resource-intensive, so a smaller effect is expected, but it shows resource allocation isn't neutral there either.
我们还在 Terminal-Bench 之外做了交叉实验验证这个模式是否成立——在 SWE-bench 上，对 227 道题、每题 10 个样本，把可用总内存从基线拉到 5 倍。效应同样成立，幅度更小：分数仍随内存单调上升，但 5x 只比 1x 高 1.54 个百分点。SWE-bench 任务本身不太吃资源，效应小在意料之中，但这说明资源分配在它那里同样不是中性的。
Other sources of variance · 其他的方差来源
Resource allocation isn't the only hidden variable. In certain configurations, time limits too start playing a role.
资源分配不是唯一的隐藏变量。某些配置下，时间限制也会开始起作用。
In principle, every element of the evaluation setup can influence the final score, from the cluster health to the hardware specs, from the concurrency level to even egress bandwidth. Agentic evals are end-to-end system tests by construction, and any component of that system can act as a confounder. We have observed anecdotally, for instance, that pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents. We have not formally quantified this effect, but it illustrates a larger point: the boundary between "model capability" and "infrastructure behavior" is blurrier than a single benchmark score suggests. A model provider can shield its eval infrastructure from this by dedicating hardware, but external evaluators can't easily do the same.
原则上，评测设置的每一个环节都能影响最终分数——集群健康状况、硬件规格、并发水平、乃至出口带宽。Agent 评测天生就是端到端的系统测试，系统里的任何组件都可能成为混淆变量。比如我们非正式地观察到，通过率随一天内的时段波动，很可能因为 API 延迟随流量模式与事故起伏。这个效应没有正式量化，但它说明了一件更大的事：「模型能力」与「基础设施行为」的边界，比单个基准分数所暗示的模糊得多。模型厂商可以用独占硬件把自己的评测基础设施屏蔽在这些干扰之外，外部评测者却很难做到。
Public benchmarks are typically meant to measure pure model capabilities, but in practice they risk conflating them with infrastructure quirks. Sometimes this may be desirable, as it enables end-to-end testing of the entire stack, but more often it's not. For coding evals meant to be shared publicly, running at multiple times and on multiple days would help average out the noise.
公开基准本意是测纯模型能力，实践中却容易把它和基础设施的怪癖混在一起。有时这是可取的——能对整个技术栈做端到端测试——但多数时候不是。对要公开分享的编程评测，在多个时段、多个日期各跑几轮，有助于把噪声平均掉。
What we recommend · 我们的建议
The ideal scenario is to run each eval under the exact same hardware conditions—both the scaffold running the eval and the inference stack—as it would ensure perfect reproducibility across the board. However, this may not always be practical.
理想情形是让每次评测跑在完全相同的硬件条件下——跑评测的 scaffold 和推理栈都一样——这能保证全方位的完美可复现。但这不总现实。
Given how container runtimes actually enforce resources—via a guaranteed allocation and a separate hard kill threshold—we recommend that evals specify both parameters per task, not a single pinned value. A single exact spec sets the guaranteed allocation equal to the kill threshold, leaving zero margin: the transient memory spikes we documented at 1x are enough to destabilize the eval. Separating the two parameters lets you give containers enough breathing room to avoid spurious OOM kills, while still enforcing a hard ceiling that prevents score inflation.
鉴于容器运行时实际是靠「保障配额 + 独立的硬杀阈值」两个参数执行资源的，我们建议评测按任务同时规定这两个参数，而不是单一钉死的值。单一规格会让保障配额等于击杀阈值、零冗余：我们在 1x 档记录到的瞬时内存尖峰就足以让评测失稳。把两个参数分开，既给容器留出喘息空间、避免冤枉的 OOM 击杀，又保住防止分数虚高的硬上限。
The band between them should be calibrated so that scores at the floor and ceiling fall within noise of each other. For instance, in Terminal-Bench 2.0, a 3x ceiling over the per-task specs cut infra error rates by roughly two-thirds (5.8% to 2.1%, p < 0.001) while keeping the score lift modest and well within noise (p = 0.40). That's a reasonable tradeoff: the infrastructure confounder is largely neutralized without removing meaningful resource pressure. The exact multiplier will vary by benchmark and task distribution, and should thus be reported, but the empirical calibration principle is general.
两个参数之间的带宽应校准到：下限和上限下跑出的分数落在彼此的噪声区间内。比如 Terminal-Bench 2.0 里，按任务规格 3 倍的上限把基础设施错误率砍了约三分之二（5.8% 到 2.1%，p < 0.001），而分数提升温和、稳稳落在噪声内（p = 0.40）。这是个合理的取舍：基础设施混淆因子基本被中和，又有意义的资源压力并未消失。具体倍数因基准和任务分布而异、因此应随结果一并报告，但「用实验校准」这个原则是通用的。
Why we care · 我们为什么在意
These findings have practical consequences beyond eval infrastructure. Benchmark scores are increasingly used as decision-making inputs, but this increased attention (and reliance) hasn't always come with corresponding rigor in how they're run or reported. As things stand today, a 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both. Without published (or standardized) setup configurations, it's hard to tell from the outside unless interested parties go the extra mile to reproduce objective results under identical conditions.
这些发现对评测基础设施之外也有实际影响。基准分数日益成为决策输入，但随之而来的关注（和依赖）并没有总是伴随相应的运行与报告严谨性。按现状，排行榜上 2 个百分点的领先，可能反映真实的能力差，也可能反映其中一次评测跑在更强的硬件上、甚至跑在更走运的时段，或者两者兼有。没有公开（或标准化）的环境配置，外人很难分辨——除非较真的人费尽力气在完全相同的条件下复现出客观结果。
For labs like Anthropic, the implication is that resource configuration for agentic evals should be treated as a first-class experimental variable, documented and controlled with the same rigor as prompt format or sampling temperature. For benchmark maintainers, publishing recommended resource specs (as Terminal-Bench 2.0 does) can go a long way, while specifying enforcement methodology would close the gap we identified. And for anyone consuming benchmark results, the core takeaway is that small score differences on agentic evals carry more uncertainty than the precision of the reported numbers suggests—especially as some confounders are simply too hard to control for.
对 Anthropic 这样的实验室，含义是：Agent 评测的资源配置应被当作一等的实验变量，以与 prompt 格式或采样温度同等的严谨度去记录和控制。对基准维护者，公布推荐资源规格（Terminal-Bench 2.0 已经在做）能走很远，而写明强制执行方式则能补上我们发现的缺口。对一切基准结果的消费者，核心结论是：Agent 评测上的小分数差异，其不确定性比报告数字的精度所暗示的更大——尤其某些混淆因子根本无法完全控制。
Until resource methodology is standardized, our data suggests that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched. The observed spread across the moderate range of resource configurations in Terminal-Bench is just below 2 percentage points. Naive binomial confidence intervals already span 1-2 percentage points; the infrastructure confounders we document here stack on top of that, not within it. At the extremes of the allocation range, the spread reaches 6.
在资源配置方法论标准化之前，我们的数据建议：排行榜上小于 3 个百分点的差距，在评测配置被记录并对齐之前都值得怀疑。Terminal-Bench 中等资源配置区间内观测到的分数离散度略低于 2 个百分点；朴素的二项分布置信区间本来就有 1-2 个百分点宽，我们此处记录的基础设施混淆因子是叠在其上、不是含在其中；而到配额区间的两端，离散度达到 6。
A few-point lead might signal a real capability gap—or it might just be a bigger VM.
几个百分点的领先，可能意味着真实的能力差距——也可能只是台更大的 VM。
Written by Gian Segato. Special thanks to Nicholas Carlini, Jeremy Hadfield, Mike Merrill, and Alex Shaw for their contributions. This work reflects the collective efforts of several teams working on evaluations for coding agents. Interested candidates who would like to contribute are welcome to apply at anthropic.com/careers.
作者：Gian Segato。特别感谢 Nicholas Carlini、Jeremy Hadfield、Mike Merrill、Alex Shaw 的贡献。本文是多个编程 Agent 评测团队集体工作的成果。有意贡献者欢迎投递 anthropic.com/careers。