你的 Agent 这次做对了，下次还能吗？

Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task.

你的 Agent 在排练时好好的，正式演示时却走了另一条路，同一个任务挂掉了。

That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper.

在台上出丑还算小事。在生产环境里，这是一个可靠性问题：一个曾经成功过的工作流，可能在用户下次发起同样请求时失败。对于对账一笔金融交易、核查合同条款这类关键任务，这足以致命。

Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap.

大多数基准（benchmark）把这种波动藏进了平均值里。在 AppWorld 上，一个用 GPT-4.1 的 ReAct Agent 五次重复运行的平均成功率为 77.4%；但五个回合全部成功的任务只有 53.0%——一致性差距（consistency gap）达 24.4 个百分点。

Most benchmarks report the first number. We built a way to measure the second — and improve it.

大多数基准只报第一个数。我们造了一个办法来度量第二个数——并把它改善。

In an earlier post, we introduced ALTK-Evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call the Consistency Analyzer, that targets this gap directly.

在更早的一篇文章里，我们介绍过 ALTK-Evolve——一个把 Agent 自己的历史轨迹（trajectory）自动蒸馏成可复用指南、再在推理时注入回去的系统。它能实测提升任务成功率，但那些结果同样只回答了「平均情形」这半个问题。本文介绍一致性指南（consistency guidelines）——altk-evolve 里的一种新指南类型，构建在一个我们称为 Consistency Analyzer 的诊断工具之上，直接瞄准这个差距。

Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap. On hard tasks it reaches 30 points.

准确率掩盖了一个可靠性问题。一个平均成功率 77.4% 的 ReAct Agent（GPT-4.1，AppWorld test_normal），五次重复全部做对的 task 只有 53.0%——一致性差距 24.4 个百分点；在最难的任务档上这个差距能到 30 个百分点。

We built a diagnostic for exactly this. The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points — steps where the model was one token-sample away from doing something different. It needs one trace and no ground truth — it resamples each decision point in that trace with a single call requesting k completions (k=5 by default), rather than re-running the task end-to-end.

我们为此造了一个诊断工具。Consistency Analyzer 对 Agent 自己录下的轨迹做重采样（resampling），找出容易翻烙饼的决策点——那些模型距离「做出不同动作」只差一次 token 采样的步骤。它只需要一条轨迹、不需要标准答案——对轨迹里的每个决策点发一次请求 k 个补全的调用（默认 k=5）做重采样，而不是把任务从头到尾重跑一遍。

Turning that diagnosis into guidelines halves the gap — from 24.4pp to 12.0pp (same-task Pass⁵ +16.0pp, similar-task +13.0pp), without costing anything in average accuracy.

把诊断结果转成指南后，这个差距砍半——从 24.4 个百分点降到 12.0 个百分点（同任务 Pass⁵ +16.0 个百分点，相似任务 +13.0 个百分点），且平均准确率分毫未损。

Full methodology and evaluations are in the technical report on arXiv.

完整方法论与评测见 arXiv 技术报告。

Standard agent evaluation reports Mean@k: run a benchmark k times, average the pass rate. Often k=3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice.

标准的 Agent 评估报的是 Mean@k：把一个基准跑 k 次取通过率的平均。通常 k=3，有时干脆只跑 1 次。这是每个排行榜上的数字，也是实践中「准确率 77%」的真实含义。

Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k: the fraction of tasks where the agent succeeds on all k runs.

Mean@k 回答的是「这个 Agent 平均有多好」。它回答不了真实用户关心的那个问题：我原样再问一遍，它还好使吗？回答这个需要 Pass^k：k 次运行全部成功的任务占比。

难点 难点提示同字母反义词：Pass@k=至少一次成功（乐观，能验证能重试的场景用）；Pass^k=每次都必须成功（悲观，用户重复请求的体感）。恒有 Pass^k ≤ Mean@k ≤ Pass@k。 Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always.

注意：Pass^k 不是 Pass@k。熟悉的 Pass@k 是乐观口径——问的是 k 次里至少成功一次没有，这在你能验证、能重试的场景是对的问法。Pass^k 是它的悲观镜像：每一次都必须成功。同样的字母，相反的问题。永远有 Pass^k ≤ Mean@k ≤ Pass@k。

A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0%. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap.

一个 GPT-4.1 驱动的 ReAct Agent 的 Mean@5 是 77.4%——确实不弱。但 Pass^5 只有 53.0%。基准里近四分之一的任务属于「有时能解有时不能」，而任务本身在两次运行之间没有任何变化。我们把这个差距——Mean@k 减 Pass^k——叫作一致性差距（consistency gap）。

This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.

这不是一个「换更大模型就能修」的能力问题。它是一条正交的轴：一个 Agent 可以既能力强、又不稳定。

Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip.

LLM Agent 每做一次决策——调哪个 API、传什么参数、要不要重试——这个决策都来自一个「下一个 token」的概率分布。关键在这个分布的形状。尖锐的分布把大部分概率质量压在一个 token 上：第二名远远落后，于是一次又一次跑出同样的选择。平坦的分布把差不多的质量摊在几个近平并列的 token 上，谁赢接近掷硬币。

难点 难点提示全文核心机制：分布形状决定多大噪声能改变结果。尖锐分布抗噪；平坦分布一推就翻；轨迹是几十个决策连乘，小概率翻面被放大成 24 个百分点的差距。 The shape decides how much noise it takes to change the outcome. Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from.

分布的形状决定了「多大的噪声才能改变结果」。尖锐分布很抗噪——GPU 浮点非结合性、请求批处理这些平台侧效应只会让数字轻微挪动，远不足以掀翻一个明显的赢家。平坦分布恰恰怕这种轻推：近平并列的次序在小扰动下就可能重排。而一条轨迹串着几十个决策，每一步小小的翻面概率连乘起来，就是「某次运行走样」的大概率——24 个百分点的差距就是这么来的。

难点 难点提示容易滑过去的一段：贪心解码和固定种子只管「分布怎么变成 token」，管不了分布本身——托管端点概率会漂，temperature=0 不保险。 This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow.

这也解释了为什么这个问题不挑你的解码设置。贪心解码和固定种子管的都是「分布怎么变成 token」——它们对分布本身无能为力。在托管端点上，概率每次运行都会轻微漂移，于是同一个 prompt、同一个模型、temperature 为零，依然可能今天把近平并列解向这边、明天解向那边。

Our setup: the ReAct agent runs at temperature 0.0, so none of the variance above is ordinary sampling.

我们的实验设置：ReAct Agent 跑在 temperature 0.0 上——所以上面这些波动没有一分来自普通意义上的采样。

Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know?

于是问题变成一次搜索：给定轨迹里，哪些步骤是平坦的？知道之后又该怎么办？

Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve's existing machinery — with a new source signal driving what gets written.

一致性指南出自一条两阶段流水线，插在 ALTK-Evolve 现有机制上——只是用一个新的源信号来驱动「写什么」。

1. Detect — the Consistency Analyzer. Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one additional model call per decision step, done once offline — issued with the sampling parameter set to draw k completions at once (k=5 by default) — replayed against the already-recorded context, not new tool calls, not new environment interactions, and not a second end-to-end rollout of the task. This yields a consistency score per decision step that is written into a scorecard to pinpoint exactly which decisions are at risk of flipping on the next run. Detection is fully black-box — no logits, no model internals, no instrumentation beyond the trace you already have.

第 1 步检测——Consistency Analyzer。给它一条录好的轨迹，它对每个决策步骤做受控重采样，度量模型输出在该点到底有多不稳定。具体说，就是每个决策步骤额外发一次模型调用、一次性离线完成——采样参数设成一次抽 k 个补全（默认 k=5）——重放的是已录下的上下文：不产生新的工具调用、不产生新的环境交互、也不把任务端到端再跑一遍。这给每个决策步骤产出一个一致性分数，写进一张记分卡，精确指出哪些决策在下一次运行中有翻面风险。检测完全黑盒——不看 logits、不碰模型内部、除了你手上已有的轨迹之外零侵入。

2. Generate — targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example, generated by GPT-4.1 from a trajectory of the AppWorld task "How many activities are done in my bucket list as per my SimpleNote note?":

第 2 步生成——定向指南。每个被标记的步骤都会变成一条标准 ALTK-Evolve 格式的一致性指南候选，直接插进现有的存储与检索管道。下面是一个真实例子，由 GPT-4.1 从 AppWorld 任务「按我的 SimpleNote 笔记算，我的愿望清单完成了几项？」的轨迹中生成：

[Guideline 1] When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line.

【指南 1】统计笔记内容里的复选框样式标记时，用行锚定的正则匹配而不是朴素的子串计数——笔记标题常在图例行里重复出现标记符号。

[Guideline 2] Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.

【指南 2】对笔记类查询，拿到搜索结果后务必检查是否有多条匹配、确认目标笔记正确，再继续往下走。

Nothing here is task-specific trivia. String-counting bugs and unverified search results are decision points that show up with high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time.

这里没有一条是某个任务的专属琐事。字符串计数错误、未验证的搜索结果，是很多 AppWorld 任务里都高不确定出现的决策点。这正是要点所在：分析器瞄准的是不稳定，而不是失败——所以它能抓住那些这次碰巧做对、下次很容易做错的步骤。

Watch the 2-minute demo — five parallel runs of the agent split 3-2 on this task because of agent uncertainty about the counting strategy, then run again after these guidelines in context: all five agree.

看这个 2 分钟演示：同一任务的五路并行运行因为 Agent 在计数策略上的犹豫分成 3 比 2；把上面这些指南放进上下文再跑：五路全部一致。

We evaluated on AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs.

我们在 AppWorld test_normal（168 个任务）上评估，Agent 为 GPT-4.1 上的 ReAct 实现：每个任务从单条基线轨迹生成一致性指南，然后在 5 次全新运行上检验。

Mean@5 (%), aggregate — same scale as Pass^5 above.

Mean@5（%），聚合口径——与上文的 Pass^5 同尺度。

The consistency gap is cut roughly in half. Aggregate Pass^5 rises 53.0% → 69.0% while Mean@5 rises 77.4% → 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp. Nearly a third of previously-inconsistent tasks become tasks the agent passes on every single run.

一致性差距大约砍半。聚合 Pass^5 从 53.0% 升到 69.0%，Mean@5 从 77.4% 升到 81.0%——「看起来行」和「靠得住」之间的差距从 24.4 个百分点收窄到 12.0 个百分点。此前不稳定任务中近三分之一变成了次次通过的任务。

The middle and hard tiers gain most. Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative) — effectively tied in relative terms, with Medium ahead absolutely. Easy gains +12.2pp, having had the least room. This is consistency guidelines doing what they're designed to do: finding and stabilizing the specific decision points where an agent's own uncertainty was leaking into the outcome.

中档和难档提升最大：中档 +22.9 个百分点（相对 +44%），难档 +14.3 个百分点（相对 +45%）——相对提升基本打平，中档绝对值领先；简单档 +12.2 个百分点，本就空间最小。这是一致性指南在做它设计上该做的事：找到并钉稳那些「Agent 自身的不确定性正在渗入结果」的具体决策点。

Mean@5 never drops. Preserving average accuracy was a hard requirement, not a nice-to-have: a system that boosts Pass^5 by trading away Mean@5 would just be shifting unreliability around, not fixing it. Mean accuracy holds or improves at every difficulty level.

Mean@5 从未下降。保住平均准确率是硬性要求而不是加分项：一个靠牺牲 Mean@5 换 Pass^5 的系统，只是把不可靠挪了个地方，并没有修好它。平均准确率在每个难度档上都持平或提升。

The guidelines generalize — they aren't patching one trajectory. Applied to a different but related task in the same AppWorld scenario — another variant of the scenario the guidelines were mined from — consistency guidelines still lift Pass^5 by +13.0pp, only 3 points below the same-task number. A guideline derived from one run isn't just patching that run; it's capturing something that transfers.

指南是可泛化的——不是在补一条轨迹的丁。把一致性指南用到同一 AppWorld 场景下另一个相关任务（即挖掘指南的那个场景的另一个变体）上，Pass^5 依然提升 +13.0 个百分点，只比同任务口径低 3 个百分点。从一次运行里提炼出的指南，补的不只是那次运行——它抓住的是能迁移的东西。

The sharper evidence comes from a weaker model, gpt-oss-120b. Same-task Pass^5 rose +6.0pp from a much lower baseline (10.1% → 16.1%) — and, interestingly, the similar-task generalization number (+8.7 pp) actually exceeded the same-task gain, suggesting the guidelines were capturing genuinely reusable failure patterns rather than memorizing one trajectory's specifics.

更锋利的证据来自更弱的模型 gpt-oss-120b：同任务 Pass^5 在低得多的基线上提升 +6.0 个百分点（10.1% → 16.1%），而且有意思的是，相似任务的泛化数字（+8.7 个百分点）反超了同任务增益——说明指南抓到的是真正可复用的失败模式，而不是背下了某条轨迹的细节。

Report Pass^k next to Mean@k. Averages can't distinguish a reliable agent from a lucky one; even k=3 will surface a gap you didn't know you had.

把 Pass^k 与 Mean@k 并排报告。平均值分不出「可靠的 Agent」和「走运的 Agent」；哪怕 k=3，也能照出一个你自己都不知道的差距。

Expect the gap to widen with difficulty. Your hardest tier is where a single averaged number is most misleading.

预期差距随难度放大。你最难的那一档，正是单一平均数最骗人的地方。

Don't reach for a bigger model first. Consistency is orthogonal to capability. A stronger model raises Mean@k; it doesn't necessarily reduce the consistency gap.

别急着换更大的模型。一致性与能力是正交的：更强的模型抬高 Mean@k，未必收窄一致性差距。

Diagnosis needs no grader and no live replay. One extra LLM call per decision step (sampling k=5 completions by default) is enough — no ground truth, no re-running the task against the environment. That's what makes it usable on production traffic, where you often can't replay a task end-to-end even once.

诊断不需要评分器、也不需要在线重放。每个决策步骤一次额外的 LLM 调用（默认一次采 k=5 个补全）就够——不要标准答案、不用对着环境把任务重跑。这正是它能用在生产流量上的原因：生产上你往往连一次端到端重放都做不到。

Try the ALTK-Evolve toolkit — the open-source repo now includes the Consistency Analyzer and consistency-guideline generation used in these experiments — or read the technical report on arXiv for the complete methodology.

试试 ALTK-Evolve 工具包——开源仓库现已包含本实验使用的 Consistency Analyzer 与一致性指南生成——或阅读 arXiv 上的技术报告看完整方法论。

If accuracy numbers you can't reproduce on your own tasks sound familiar, we'd like to hear about it — concrete examples of flip-prone behavior in your own agents are exactly the kind of feedback that shapes what we build next. Open an issue or a discussion.

如果「准确率数字在自己任务上复现不出来」这件事你听着耳熟，我们想听听——你自己 Agent 里容易翻面的具体案例，正是塑造我们下一步做什么的那种反馈。欢迎开 issue 或讨论帖。

Mean@k. Run a task k times, report the average pass rate — what most benchmarks call "accuracy."

Mean@k：一个任务跑 k 次，报平均通过率——大多数基准口中的「准确率」。

Pass^k. The fraction of tasks where the agent succeeds on all k independent runs. Always ≤ Mean@k. What a user experiences if they run the same query twice.

Pass^k：k 次独立运行全部成功的任务占比。永远 ≤ Mean@k。也就是用户把同一查询跑两遍时实际体验到的东西。

Pass@k: At least one of k runs succeeds — the optimistic counterpart, common in code-generation papers.

Pass@k：k 次里至少一次成功——乐观口径的对应物，代码生成论文里常见。

Consistency gap. Mean@k − Pass^k, in percentage points.

一致性差距：Mean@k − Pass^k，单位为百分点。

ALTK-Evolve open source repo — github.com/AgentToolkit/altk-evolve

ALTK-Evolve 开源仓库：github.com/AgentToolkit/altk-evolve

Technical report — arXiv

技术报告：arXiv

Consistency guideline demo — 2 min video

一致性指南演示：2 分钟视频