LangSmith LLM Gateway：Agent 的运行时控制

Agents in production need runtime controls over model calls to avoid undesirable outcomes like cost overruns and outages. Spend caps, rate limits, model fallbacks, and sensitive data protections are all important controls, but without the right tooling, engineers have to write these controls manually within each agent for a given model provider. That's why we built LangSmith LLM Gateway. It's the central governance layer that sits between your agents and the models they call, designed specifically for the controls that agent engineering teams need in production. LLM Gateway gives teams one place to enforce runtime controls across agents, models, and providers, helping them consistently govern model usage while avoiding vendor lock-in.

生产环境中的 agent 需要对模型调用的运行时控制，以避免成本超支与宕机这类坏结局。支出上限、速率限制、模型回退、敏感数据保护都是重要控制——但缺趁手工具时，工程师只能在每个 agent 里、按特定模型 provider 手写这些控制。这就是我们造 LangSmith LLM Gateway 的原因：它是坐在你的 agent 与它们调用的模型之间的中央治理层，专为 agent 工程团队在生产中需要的控制而设计。它给团队一个统一的地方跨 agent、跨模型、跨 provider 执行运行时控制——一致地治理模型用量，同时避免供应商锁定。LangSmith LLM Gateway 现已进入公开测试。

In production, the model layer can quickly become the source of costly mistakes and customer-facing outages. Here's what this looks like in practice: your model provider has suffered an outage, and now your customer service agent is returning errors, leading to customer frustration and lost revenue. Or your coding agent gets stuck in a retry loop overnight, and by morning it's made 10,000 LLM calls and you have a four-figure invoice.

生产中，模型层很快会变成昂贵事故与面向客户的宕机之源。实际长这样：你的模型 provider 宕机了，客服 agent 开始报错——客户沮丧、收入流失；或者你的 coding agent 夜里陷入重试循环，到早上已经跑了 10,000 次 LLM 调用，你收到一张四位数的账单。

In these cases, it's not enough to understand what happened after the fact. You need to enforce policies before violations happen. But if controls are embedded in each agent's code, every new agent and model provider adds another implementation path to maintain. A model gateway lets teams define policies once, then enforce them consistently across models and agents.

这些场景里，事后搞清楚发生了什么并不够——你需要在违规发生**之前**执行策略。但若控制嵌在每个 agent 的代码里，每个新 agent、每个新模型 provider 都意味着又一条要维护的实现路径。模型网关让团队把策略定义一次，然后跨模型跨 agent 一致地执行。

LLM Gateway provides the controls that teams need first when agents go into production: cost controls, rate limits, model fallbacks, and sensitive data handling.

LLM Gateway 提供团队把 agent 推上生产时最先需要的那批控制：成本控制、速率限制、模型回退与敏感数据处理。

Avoid cost overruns: LLM Gateway lets you monitor and control LLM spend for internal and external users before it becomes a problem. You can set time-bound caps and track spend at four levels: Organization level — for company-wide controls. Workspace level — to keep teams within spend limits. API key level — useful for controlling spend on a specific project. User level — for individual spend controls. When a cap is hit, the agent receives a 402 response with a clear error.

**防成本超支**：在问题发生之前监控并控制内部与外部用户的 LLM 支出。可设有时限的上限，并在四个层级跟踪支出：组织级（公司范围控制）、工作区级（把团队圈在预算内）、API key 级（控制特定项目的支出）、用户级（个人支出控制）。触顶时，agent 收到一个带明确错误的 402 响应。

Limit runaway traffic: Apply rate limits at the organization, workspace, user, and API key levels to prevent accidental abuse or traffic spikes from overwhelming provider limits and degrading reliability for other agents or teams.

**限制失控流量**：在组织、工作区、用户与 API key 四个层级施加速率限制，防止意外滥用或流量尖峰打爆 provider 限额、拖累其他 agent 或团队的可靠性。

Per-customer policies for spend and rate limits: For multi-tenant environments and resellers, LLM Gateway gives you customer-level control without customer-specific API keys. Use a custom request header to identify each end customer or team, then enforce separate spend caps and rate limits while routing all LLM calls through one API key. You can update limits and check spend across customers from one place, instead of building and maintaining that logic yourself.

**按客户的支出与速率策略**：面向多租户环境与转售商，无需按客户发 API key 即可获得客户级控制——用自定义请求头标识每个终端客户或团队，全部 LLM 调用经同一个 API key 路由，同时执行各自独立的支出上限与速率限制。更新限额、查看各客户支出都在一处完成，不必自建自维护这套逻辑。

Improve agent reliability: Define fallback rules across models and hosts, then enforce them for all applications. When a provider is down, rate-limited, or unavailable, agents can route to another model without custom fallback logic in each application.

**提升 agent 可靠性**：跨模型跨主机定义回退规则，然后对所有应用统一执行——provider 宕机、被限流或不可用时，agent 可以路由到另一个模型，各应用不必各写一套回退逻辑。

Reduce sensitive data exposure: Detect, redact, and replace PII and secrets in requests before they reach model providers. LLM Gateway also redacts sensitive data from the traces it records in LangSmith. Currently, this feature is available only to Enterprise plan users.

**减少敏感数据暴露**：在请求到达模型 provider 之前检测、脱敏并替换 PII 与密钥；LangSmith 记录的轨迹里同样做脱敏。当前该能力仅对企业版开放。

Trace LLM Gateway events in LangSmith: Gateway events are logged as metadata in LangSmith traces, so you can monitor how controls behave with production traffic. When a request is blocked by a spend limit, rate limit, or other control, you can open the trace to inspect what happened in context and decide whether anything needs to change.

**在 LangSmith 追踪网关事件**：网关事件作为元数据写入 LangSmith 轨迹，你可以观察各项控制在真实流量下的行为。当请求被支出限制、速率限制或其他控制拦截，打开轨迹即可在上下文里检查现场、判断是否需要调整。

LLM Gateway provides a single interface for routing calls between your agents and model providers. It supports popular providers like OpenAI, Anthropic, and Fireworks and custom models with OpenAI- or Anthropic-compatible endpoints. LangSmith Gateway is BYOK-first. You can bring your own provider keys and use LLM Gateway for routing, governance, and policy controls without adding extra instrumentation to every LLM call. For teams that want simpler procurement or a faster way to start building agents, Gateway also supports hosted inference for open models powered by Fireworks, available directly through LangSmith using Gateway Credits. We believe open weight models are important because they give teams more control over AI systems. They give teams flexibility to optimize for cost, quality, latency, privacy, and deployment needs, and to tune models around the specifics of their business. Gateway is one way we're making open models easier to adopt in production.

LLM Gateway 提供单一接口，在 agent 与模型 provider 之间路由调用：支持 OpenAI、Anthropic、Fireworks 等主流 provider，以及带 OpenAI 或 Anthropic 兼容端点的自定义模型。**BYOK 优先**（Bring Your Own Key）：自带 provider 密钥，用网关做路由、治理与策略控制，不必给每次 LLM 调用加额外埋点。想要更简采购或更快起步的团队，网关也支持由 Fireworks 驱动的开模型托管推理（经 LangSmith 用 Gateway Credits 直接使用）。我们相信开放权重模型重要，因为它给团队对 AI 系统更多掌控：按成本、质量、延迟、隐私与部署需求灵活优化，还能围绕业务细节调优模型——网关是我们让开模型更容易进生产的途径之一。

Teams are already using LLM Gateway to put hard limits and shared controls around production agents. Some common patterns include public-facing agents that need spend caps, and multi-customer agents that need centralized billing, usage tracking, and per-customer limits.

团队已经在用 LLM Gateway 给生产 agent 套上硬限制与共享控制。常见模式：需要支出上限的面向公众的 agent；需要集中计费、用量跟踪与按客户限额的多客户 agent。

You can call LLM Gateway directly via API or use it with coding harnesses like Claude Code, Codex, and Deep Agents Code (dcode). To get started, point your agents at the LangSmith Gateway endpoint, authenticate with your LangSmith API key, and add your provider API keys to workspace secrets. Then, configure your controls for fallbacks, rate limits, spend limits, and sensitive data handling. Update your base_url, and keep the remainder of your code as is.

可以直接经 API 调用 LLM Gateway，也可配合 Claude Code、Codex、Deep Agents Code（dcode）这类 coding harness 使用。起步：把 agent 指向 LangSmith Gateway 端点、用 LangSmith API key 认证、把 provider 密钥加进 workspace secrets；然后配置回退、速率限制、支出限制与敏感数据处理。更新 base_url，其余代码原样保留。

We started LLM Gateway with the controls we've seen teams need most. Next, we're expanding that foundation with more ways to govern model calls in production. Broader guardrails — more coverage for the risks that come with agents calling external models and interacting with production systems. CI/CD controls — tools for deploying prompts and context, running A/B tests, managing blue-green and shadow deployments, and defining controls with Infrastructure-as-Code.

网关从团队最需要的控制起步，下一步在生产模型调用治理上继续扩展：**更广的护栏**——覆盖 agent 调用外部模型、与生产系统交互的更多风险；**CI/CD 控制**——部署提示词与上下文的工具、跑 A/B 测试、管理蓝绿与影子部署、用基础设施即代码定义控制。

LLM Gateway is available now in public beta for Plus and Enterprise plans. Get started by logging into LangSmith or signing up, then selecting "LLM Gateway" in the sidebar. Enterprise plan users can request access to Data Protection controls here.

LLM Gateway 现已在公开测试中向 Plus 与企业版开放。登录 LangSmith（或注册）后在侧栏选「LLM Gateway」即可开始；企业版用户可在此申请数据保护控制的访问权。 译注 译注译注：原文发布于 2026-07-30，10-23 才被探测捕捉（LangChain feed 顺序变化，此前该源最新条目长期指向别文——与此前记录过的 feed link 指旧文 bug 同源）。内容与 10-05 Simon《一切服务都需要默认硬预算上限》观点直接互文：网关正是该主张的产品化落地。