This is a joint blog by Microsoft and Hugging Face, special thanks to Tommy Guy (founder at Enderis AI, previously Microsoft), Sergio Paniego from Hugging Face and our former interns Zhuochun Li (University of Pittsburgh), Ali Keramati (UC Irvine), Youngmin Ko (Northwestern) for co-authoring/reviewing efforts. A customer writes in. Her $745 kitchen appliance has been stuck in a courier "exception" at a Nashville distribution center, fifteen days past its estimated delivery date.
本文由 Microsoft 与 Hugging Face 联合发布，特别感谢 Tommy Guy（Enderis AI 创始人、前微软）、Hugging Face 的 Sergio Paniego，以及前实习生 Zhuochun Li（匹兹堡大学）、Ali Keramati（UC Irvine）、Youngmin Ko（西北大学）的合著/评审。开场案例：一位客户来信——她 745 美元的厨房电器卡在纳什维尔转运中心的快递「异常」里，比预计送达日已晚了十五天。
The AI agent does careful work. Nine tool calls: it pulls the order, checks tracking, looks up her customer profile, searches the refund policy twice, confirms no ticket exists, opens one, documents the timeline, and reads the policy correctly; her account segment genuinely does not qualify for late-delivery compensation. Then it closes the ticket as resolved and replies "Since your query is resolved, is there anything I may assist you with?"
AI agent 干得很认真。九次工具调用：拉订单、查物流、看客户档案、搜两次退款政策、确认没有工单、开一张、记录时间线、政策也读对了——她的账户分层确实不符合延误赔付条件。然后它把工单关成「已解决」并回复：「您的问题已解决，还有什么可以帮您？」
Two things are wrong. The carrier exception is still open, so the required end state was on hold, pending resolution. And the customer never got a real answer to what she actually asked. An AI grader checking tool calls would see nine well-formed ones. The grader checking whether the agent wrote to the database would see that too. The database is what disagrees.
两件事错了：快递异常还开着——要求的终态是 hold（等待处理）而非 solved；客户真正问的问题也始终没得到回答。检查工具调用的 AI 评分器会看到九次规范调用；检查「是否写库」的评分器也会通过。**不同意的是数据库。**
That gap is what ThinkingBox measures. Across 507 stateful business workflows, each run 20 times against various LLM models, it grades agents on terminal backend state and side effects. This post covers what we found, what consistency costs, and how to run the benchmark yourself through OpenEnv. You can run this one yourself: the example above is adapted from a benchmark task sandbox_external_retail_group1.py:test_case_ST003_006, and the executable check that fails is a single field: the ticket's status is solved where the required end state is hold. The full trace is in Appendix D.4, Case 3 of our paper.
这个落差就是 ThinkingBox 度量的东西：507 个有状态业务工作流、每个对多款模型各跑 20 次，按**后端终态与副作用**给 Agent 评分。本文讲我们发现什么、一致性值多少钱、怎么用 OpenEnv 自己跑这个基准。上面的例子改编自基准任务 sandbox_external_retail_group1.py:test_case_ST003_006，没过的可执行检查只有一个字段：工单状态是 solved，而要求终态是 hold。完整轨迹见论文附录 D.4 案例 3。
A tool call is not an outcome · 工具调用不是结果
Final responses and valid tool calls are only proxies. An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect. Only the records it leaves behind settle the question.
最终回复与合法工具调用都只是代理指标。Agent 可以听着全对，却留下错的值、改了错的记录、多造一个副作用。只有它留下的记录能定案。
The gap is substantial. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. Those state-check findings overlap.
落差相当大：覆盖 12 款模型、121680 次有效试验的共通集消融里，79853 次尝试没过可执行检查。这些失败中 **67.24% 照样干净收尾、调用了改状态的工具、最终也没报任何工具错误**——而可执行检查照样在其中 77.61% 里发现写错的字段值、43.30% 里发现多余的副作用、25.36% 里发现漏掉的必要效果（三类发现有重叠）。
A trajectory is a claim. Database state is the evidence. Repetition is the trust test.
轨迹是主张，数据库状态是证据，重复是信任测试。
One success is not reliability · 一次成功不是可靠性
An agent that processes a refund correctly once and mishandles it the next four times is not a working refund agent. So every task runs 20 independent times, each from an identical clean backend, and we report three different things. We use observed 20/20 in this blog post as the literal count of how many of the 507 tasks passed 20 out of 20. No estimator, no smoothing.
一次退款处理对、接下来四次全错的 Agent，不是能用的退款 Agent。所以每个任务独立跑 20 次——每次都从完全相同的干净后端出发——并报告三个不同的数。本文的 20/20 用字面口径：507 个任务里 20 次全过的任务数，不做估计、不做平滑。
Starting with the familiar view. The table below reports pass@1, the single-attempt score estimate, broken out by domain. This is the number most leaderboards publish, and on its own it reads like an ordinary capability ranking. Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model, within a point of GPT-6-Astra. Domain matters just as much: Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.
先看熟悉的画面：pass@1（单次得分估计）按领域拆分——这是多数排行榜发的数字，单看像一张普通的能力排名。Claude Opus 5.5 以 67.16% 总体领先（高出 Opus 5 三分之二个点）；Kimi-K3 是最强开放权重模型、距 GPT-6-Astra 不到一分。领域差异同样关键：Claude Opus 4.6 零售领域 68.62%，汽车保险只有 8.30%。
One good run tells you a model can do the work. It does not tell you whether it will do it again. So run every task 20 times and ask how much of that score survives. Only three hold on to most of their pass@1 scores: GPT-6 Astra retains 78% of its single-attempt rate, and Claude Opus 5.5 and Claude Opus 5 each retain 71%. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.
一次好运行只说明模型「能做」，不说明它会「再做」。所以每任务跑 20 遍，看单次分还剩多少：只有三款保住大头——GPT-6 Astra 保住单次率的 78%，Claude Opus 5.5 与 Opus 5 各保 71%；另一端 GLM-5.1、Kimi-K2.6、DeepSeek-V4-Pro 各只剩约 8%。
The gap between what a model can do once and what it does every time is the whole story.
「一次能做」与「每次都做」之间的鸿沟，就是全部的故事。
Can you depend on the model behind your agent? · 你能依赖 Agent 背后的模型吗
Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24% pass@1, ahead of every proprietary model. Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.41%, succeed in all 20 attempts.
Kimi-K3 拥有全场最广的覆盖：至少解出基准的 93.89%（476/507），只有 31 个任务把它完全难住（全场最少）；零售工作流上它以 82.24% 的 pass@1 明确领先、压过所有专有模型。但 Kimi-K3 同时是最不一致的之一：507 个任务里只有 68 个（13.41%）20 次全对。
Claude Opus 5 inverts this. It solves fewer tasks at least once (79.09%; 106 defeat it entirely) but completes 47.53% of the benchmark on every single attempt.
Claude Opus 5 正好反过来：至少解出的任务更少（79.09%，106 个完全难住它），但每次尝试都做对的占基准的 47.53%。
A newer model does not fix this. Claude Opus 5.5 scores higher than Claude Opus 5 on every-attempt average, 67.16% against 66.50%, and solves more tasks at least once. It passes exactly the same number of tasks on all 20 attempts: 241. Half a point of headline accuracy bought no additional dependability at all. Kimi-K3 solves 75 more tasks at least once than Opus 5. Opus 5 solves 173 more tasks consistently than Kimi-K3. If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at.
更新的模型并没有修复这一点：Claude Opus 5.5 的每次尝试均分高于 Opus 5（67.16% 对 66.50%），至少解出的任务也更多——但 20 次全对的任务数一模一样：241 个。**半分头条精度，没买到任何可靠性。** Kimi-K3 比 Opus 5 多解 75 个「至少一次」的任务；Opus 5 比 Kimi-K3 多 173 个「每次都对」的任务。给碰真实记录的工作选模型，pass@20 是看错的列。
What consistency costs · 一致性值多少钱
Capability comparisons usually stop at the score. For anyone deploying, the relevant question is what a successful unit of work costs. We measure that as cost per successful task attempt: we took each model's recorded token usage from its full 507 × 20 campaign and priced it at undiscounted list rates available on OpenRouter+, reversing promotional discounts and excluding endpoints that declare quantization. Then we divided one run's cost by the number of attempts that succeeded: cost per successful task attempt = estimated cost for 507 attempts, one per task ÷ (507 × pass@1). This is a comparative efficiency index, not an invoice, and not the price of serving one production request. It also prices single successes, not consistency. We price consistency next. Example: GPT-5.4 costs $43.49 for 507 attempts (one attempt per task) and has 65.36% pass@1, so $43.49 ÷ (507 × 0.6536) = $0.131 per successful task attempt.
能力对比通常停在分数。对要部署的人，相关问题是一个成功的 Work 单元成本多少。我们度量「单次成功成本」：取每款模型 507 × 20 全战役记录的 token 用量，按 OpenRouter+ 无折扣牌价计价（还原促销折扣、排除声明量化的端点），再除以成功次数：单次成功成本 = 507 次尝试（每任务一次）的估算成本 ÷ (507 × pass@1)。这是比较效率指数，不是账单、不是生产请求价格——而且它给「单次成功」计价，不给一致性计价，一致性下面单算。例：GPT-5.4 跑 507 次花 43.49 美元、pass@1 为 65.36%，即 43.49 ÷ (507 × 0.6536) = 每次成功 0.131 美元。
### Pareto cost frontier · 帕累托成本前沿 A model is on the frontier if no other model is both no more expensive and at least as accurate. Three models qualify; every other model is dominated on at least one axis. The frontier has three steps. GPT-5.6 Sol has the lowest cost per success at $0.127; GPT-5.4 raises pass@1 by 3.45 percentage points for $0.004 more per success; Claude Opus 5.5 adds another 1.80 points at $0.276 per success. Each of the three remains on the cost frontier line because no cheaper model matches its pass@1. Claude Opus 5 is the clearest case: at $0.475 per successful attempt and 66.50% pass@1, it is both more expensive and less accurate than Claude Opus 5.5 at $0.276 and 67.16%.
前沿的定义：没有别的模型同时「不更贵且至少一样准」。三款入围，其余全被至少一轴支配。前沿三档：GPT-5.6 Sol 单次成功成本最低（0.127 美元）；GPT-5.4 每次成功多花 0.004 美元换 pass@1 提 3.45 个点；Claude Opus 5.5 再加 1.80 个点、每次成功 0.276 美元。三款都在前沿线上，因为没有更便宜的模型达到它们的 pass@1。Claude Opus 5 是最清晰的反例：每次成功 0.475 美元、pass@1 为 66.50%——比 0.276 美元、67.16% 的 Opus 5.5 又贵又差，被彻底支配。
### Now price consistency · 再给一致性计价 Cost per success rewards a model that is cheap and often right. It does not reward a model that is right every time. So we also compute cost per dependable task: the cost of the full 20-run campaign divided by the number of tasks the model passed on all 20 attempts. Example: GPT-6-Astra costs 20 × $86.03 = $1,720.60 for the campaign and passes 231 tasks on every attempt, so $1,720.60 ÷ 231 = $7.45 per dependable task. Now rank by consistency. GPT-5.4 is the cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45, and Claude Opus 5.5 the joint-highest 241 at $7.80. None of the three dominates the others: each additional dependable task costs more. Claude Opus 5 also passes 241, but at $13.30, so Opus 5.5 dominates it outright. GPT-5.6 Sol, the cheapest per single success at $0.127, costs $9.76 per dependable task. The cheapest way to get a right answer is not the cheapest way to get a dependable one.
单次成功成本奖励「便宜且常对」的模型，不奖励「每次都对」的模型。所以我们再算**可靠成本**：20 遍全战役的成本 ÷ 20 次全过的任务数。例：GPT-6-Astra 全战役 20 × 86.03 = 1720.60 美元、231 个任务每次都对，即 1720.60 ÷ 231 = 每个可靠任务 7.45 美元。按一致性排序：GPT-5.4 最便宜（6.80 美元，但只有 128 个任务达标）；GPT-6 Astra 到 231 个（7.45 美元）；Claude Opus 5.5 到并列最高的 241 个（7.80 美元）。三者互不支配——每多买一个可靠任务都要加钱。Claude Opus 5 同样过 241 个但每个 13.30 美元，被 Opus 5.5 彻底支配。单次成功最便宜的 GPT-5.6 Sol（0.127 美元），可靠成本是 9.76 美元。**拿到一个对答案的最便宜方式，不是拿到一个可靠答案的最便宜方式。**
Failure signatures · 失败签名
We assign each failed trace one deterministic diagnostic signature, and the headline is actionable: roughly four in five failures are tool handling, not reasoning. These are unweighted averages of per-model shares and observable labels, not unique causal explanations. The practical pattern is simple: agents usually get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That is a retry and error-recovery problem before it is a model problem.
我们给每条失败轨迹分派一个确定性诊断签名，头条结论可直接行动：**约五分之四的失败是工具处理而非推理**（各模型份额的无权平均与可观察标签，非唯一因果解释）。实践模式很简单：Agent 通常走得够远、开始执行工作流，然后败在「从工具报错、前置条件失败、空查询里恢复」上——这是重试与错误恢复问题，排在模型问题前面。
Difficulty also changes by domain: across the models listed in Table 2 above, retail averages 59.52% pass@1 while auto insurance averages 33.83%. What to do about it: treat the 20/20 rate as a design input, not a verdict. The same signal the benchmark grades on is available in production: check the terminal state before you commit, not the model's summary of it. Classify tool and system errors so retries target the recoverable ones. Cut the tool surface to what the workflow needs. And require human approval on the changes you cannot cheaply reverse. We have not measured the lift from any of these on this benchmark, which is exactly the kind of thing the environment now makes testable.
难度也随领域变：上表模型里零售平均 pass@1 为 59.52%，汽车保险只有 33.83%。怎么办：把 20/20 率当**设计输入**而非判决——基准评分用的那个信号生产里就有：提交前查终态，别读模型的自我总结。把工具错与系统错分类，让重试只瞄准可恢复的那部分；把工具面砍到工作流所需；不可便宜回滚的变更上人审。我们没有在此基准上度量这些动作的提升幅度——而这正是这个环境现在让它们变得可测的东西。
How it works · 它怎么工作
ThinkingBox is the agent sandbox, while ThinkingBox-Bench is a dataset benchmark to evaluate agents. Each task defines a starting backend state, a user goal, the available MCP tools, the domain policy, and executable checks over the terminal state. A simulated user holds private context (a booking reference, a preference, or a date of birth) and releases it only when asked. Every attempt gets an isolated MCP session with freshly initialized state. Two attempts of the same task never share a database row or cached tool state, which is what makes 20-trial comparison meaningful. At the end, a side-effect extractor derives what actually changed, and deterministic judges compare it against the required end state, accepting any trajectory that produces the right outcome while rejecting wrong, missing or extra effects. For requirements with no clean database value ("did the agent disclose this is not guaranteed?"), a narrow binary rubric question handles the semantics. 477 of the 507 tasks are graded on state alone; 30 add response rubrics. The trust boundary: the model sees tasks, dialogue and tool schemas. Golden state, assertions, grading internals and credentials stay on the evaluator side.
ThinkingBox 是 Agent 沙箱，ThinkingBox-Bench 是评测 Agent 的数据基准。每个任务定义：起始后端状态、用户目标、可用 MCP 工具、领域策略、对终态的可执行检查。模拟用户握着私有上下文（订单号、偏好、生日），问了才给。每次尝试拿到独立的 MCP 会话与全新初始化的状态——同任务两次尝试从不共享数据库行或工具缓存，20 次对比因此有意义。结束时，副作用提取器推导实际改变了什么，确定性裁判对照要求终态比对：产出正确结果的轨迹都算过，错的、漏的、多的效果都拒。没有干净库值可查的要求（「agent 有没有声明这不保证？」）用窄式二元评分问题处理语义。507 个任务里 477 个纯按状态评分，30 个加回复评分。信任边界：模型看得到任务、对话与工具 schema；金标状态、断言、评分内部与凭证留在评测方。
Run it yourself · 自己跑
ThinkingBox is now on Hugging Face, both the harness and the dataset. ThinkingBox-Bench now sits behind the OpenEnv interface, and each finished episode returns a binary pass/fail reward. Tested on Linux and WSL, with Python 3.11+, uv and Docker. You also need a thinkingbox-data checkout at the pinned release and model endpoints for the agent, simulated user and judge. One endpoint can serve all three roles, which is the simplest way to start.
ThinkingBox 已上 Hugging Face（harness 与数据集都齐），ThinkingBox-Bench 接在 OpenEnv 接口后，每个完成的回合返回二元 pass/fail 奖励。在 Linux 与 WSL 上测试过，需要 Python 3.11+、uv 与 Docker；还需要钉定版本的 thinkingbox-data 检出，以及 Agent、模拟用户、裁判三个模型端点——一个端点可同时担三职，是最简单的起步方式（1 个端点配 3 个角色）。
Install (git clone OpenEnv → uv sync --project envs/thinkingbox_env --frozen → clone thinkingbox-data 并 checkout thinkingbox-bench-v1.0 → uv tool install thinkingbox 提供 tb 命令)；然后第二终端起 Typesense 30.1（docker run …typesense/typesense:30.1，端口 8108，直到 /health 通过）；第三终端起 Session Proxy 与 MCP 服务器（tb mcp-start --host 127.0.0.1 --port 7111 --servers …/servers.yaml，查 /health）；回第一终端对 ThinkingBox YAML 配置（写三个模型名）起 OpenEnv 服务（OPENENV_TB_CONFIG=… uv run --project envs/thinkingbox_env --frozen server）；先查 http://127.0.0.1:8000/ready（就绪检查通过前返回 503；Typesense 与模型端点它探测不到，需自行确认）；再跑一集评分（echo 一行任务名进 one_task.yaml → thinkingbox-eval one_task.yaml --config … --output results.jsonl --errors-output errors.jsonl --repeat 1 次 --message-timeout 1800 秒）。
（安装与运行命令原样，见上方英文段：clone 与 uv 安装三步、thinkingbox-data 钉在 bench-v1.0 版、Typesense 30.1 端口 8108、MCP 绑 127.0.0.1 端口 7111、就绪口 127.0.0.1:8000（未就绪返 503）、评分命令 --repeat 1 次 --message-timeout 1800 秒。要点：OpenEnv 适配器把运维失败写进 errors 边车、可重跑而不是静默混进模型结果；正式结果必须解决或显式交代这些尝试——我们把系统错误计为不成功试验。）
Runs are gated on a pinned framework commit, a pinned data release and a bundle hash, so a canonical result is verifiable rather than asserted.
运行钉定在框架 commit、数据发布版与 bundle 哈希上——正式结果可验证，而非口头断言。
Where this goes next · 接下来去哪
The useful part of this work is not our pass@1 leaderboard. It is the environment. If you are evaluating an agent that touches real records: Inspect a failure. Find a run that terminated cleanly and still failed, and look at what actually changed in the database. It reframes what your own evals measure. Reproduce one task through OpenEnv with your own model. Report a repeat metric, and define it. Whatever k your use case justifies; say whether you are reporting best-of-k or every-of-k, and how you computed it.
这项工作有用的部分不是 pass@1 排行榜，而是环境本身。如果你在评测碰真实记录的 Agent：去查一次失败——找一条干净收尾却没过的运行，看数据库里实际变了什么，它会重新框定你自己的评测在测什么；用你自己的模型经 OpenEnv 复现一个任务；报告一个重复指标并定义它——k 取多少由你的用例决定，但要说清你报的是 best-of-k 还是 every-of-k、怎么算的。
Further details: Environment envs/thinkingbox_env; OpenEnv docs; Framework microsoft/thinkingbox · tutorial; Benchmark microsoft/thinkingbox-data · v1.0 release; Dataset viewer microsoft/ThinkingBox-Bench; Paper arXiv:2608.19741; RL training (coming soon) microsoft/thinkingbox-training. ThinkingBox code is MIT-licensed; the benchmark data is CDLA-Permissive-2.0; the OpenEnv environment ships under OpenEnv's BSD-3-Clause. Disclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real AI agentic enterprise patterns; the customers are not real. ThinkingBox and ThinkingBox-Bench is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft.
更多细节：环境 envs/thinkingbox_env；OpenEnv 文档；框架 microsoft/thinkingbox（含教程）；基准 microsoft/thinkingbox-data（v1.0 发布）；数据集查看器 microsoft/ThinkingBox-Bench；论文 arXiv:2608.19741；RL 训练（即将上线）microsoft/thinkingbox-training。ThinkingBox 代码 MIT 许可，基准数据 CDLA-Permissive-2.0，OpenEnv 环境随 OpenEnv 的 BSD-3-Clause。免责声明：公开基准的每个任务都是合成重建——工作流与策略仿照真实企业 Agent 模式，客户不是真人。ThinkingBox 与 ThinkingBox-Bench 由 Microsoft Copilot Studio 团队联合 Toloka 构建，合作者包括在微软实习的匹兹堡大学、西北大学、哥伦比亚大学与 UC Irvine 人员。