On the AI Infrastructure team at Ai2, we're responsible for providing the institute's GPU compute capacity, specifically targeting large, distributed training workloads. We think about this task as a pyramid of four metrics that build on each other.
在 Ai2 的 AI 基础设施团队，我们负责为整个机构提供 GPU 算力，专攻大规模分布式训练负载。我们把这件事想成一座由四个指标层层垒起的金字塔。
The foundation is availability: how often the hardware is healthy and ready for work. Above this is occupancy: the fraction of available time assigned to a specific workload. Next is impact: how often the most valuable workloads are chosen to receive resources. The capstone of the pyramid is utilization: the fraction of GPU capacity used over the lifetime of a workload.
塔基是可用性（availability）：硬件有多大比例时间是健康待命的。往上是占用（occupancy）：可用时间中分给具体负载的比例。再往上是影响力（impact）：最有价值的负载有多常被选中获得资源。塔尖是利用率（utilization）：一个负载的生命周期里 GPU 产能真正被用掉的比例。
This post is about improving the impact of our scheduling decisions. We recently replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. As a result, we shifted the debate about how much GPU time each research project deserves from a case-by-case operational task to a transparent administrative budgeting process.
这篇文章讲的就是如何改进调度决策的影响力这一层。我们最近用一套新系统替换了基于优先级的调度器——它包含 GPU 时间预算、层级公平共享分配与时间切片契约三个部件。结果是：每个研究项目该得多少 GPU 时间的争论，从逐案处理运维事务，变成了一个透明的行政预算流程。
Overcommitting · 超额认购
At Ai2, we manage thousands of NVIDIA H100, B200, and B300 GPUs arranged in clusters ranging in size from 88 to 1024 GPUs. These clusters are built for large-scale distributed training of AI models, and they serve a group of about 150 internal researchers whose work covers a diverse set of AI domains, including the full model flow of LLM and VLM training, robotics reinforcement learning (RL) simulation, and post-training for scientific agentic use cases.
Ai2 管理着数千块 NVIDIA H100、B200 与 B300 GPU，组成从 88 到 1024 卡不等的多座集群。这些集群为 AI 模型的大规模分布式训练而建，服务约 150 位内部研究员，工作覆盖广泛的 AI 领域：LLM 与 VLM 训练的全流程、机器人强化学习（RL）仿真、面向科学 Agent 用例的后训练。
Like many labs, we have demand for GPU time that far exceeds supply. Based on submitted workloads, at any moment in time we have outstanding requests for 2-3x more GPUs than are available. One way to think about this is that every available GPU hour on our cluster has 2-3 different research workloads competing for it.
和许多实验室一样，我们的 GPU 时间需求远超供给。按提交的负载算，任意时刻未满足的 GPU 请求都是现有数量的 2-3 倍。可以这样理解：集群上每一个可用的 GPU 小时，都有 2-3 个不同的研究负载在争抢。
Historically, we used a priority-based scheduler, and we allowed workloads to opt out of preemptability. Each team had a limit on concurrent GPUs that could be used by workloads which were protected from preemption. Preemptible workloads could exceed that limit on idle GPUs. This strategy produced predictable pathologies. For example, we observed instances of GPU "squatting" where users would park no-op workloads they could connect to when the need arose. This occurred because researchers found they could not launch debugging workloads with low enough latency to tackle problems in real time. We also observed priority inflation, where eventually 100% of scheduled workloads used HIGH priority. This meant that lower priority levels were starved of GPU time altogether. Since preemptability was optional, we also found that our on-call engineers spent a majority of their ticket response time negotiating the organized shutdown of non-preemptable workloads running on hosts with known maintenance problems.
历史上我们用基于优先级的调度器，并允许负载选择退出可抢占（opt out of preemptability）。每个团队有一个「受抢占保护的并发 GPU 数」上限；可抢占负载则可以在空闲 GPU 上超出这个上限。这套策略产生了可以预见的病态。比如我们观察到 GPU「占坑」（squatting）：用户把空转负载停在卡上备着，需要时连上去。起因是研究员发现 debug 负载的启动延迟做不到足够低，无法实时处理问题。我们还观察到优先级通胀（priority inflation）：最终 100% 被调度的负载都标着 HIGH 优先级——低优先级档被彻底饿死。又因为可抢占是可选的，on-call 工程师的工单响应时间大部分花在一件事上：跟运行在已知待维修机器上的不可抢占负载的主人谈判，商量有序关停。
Tragedy of the commons · 公地悲剧
When these problems emerged, we were slow to identify their root causes. Our initial attempts to ensure the most important work received GPU time were focused on tighter control of how priorities were set, and, ultimately, working around the priority-based scheduler by explicitly assigning GPU monopolies to important projects. While we didn't recognize it at first, we had built a perfect laboratory for observing the "tragedy of the commons." Individuals were competing over a scarce, shared resource and, by seeking to maximize individual outcomes, achieving a non-optimal global result and abusing the underlying resource.
这些问题出现时，我们没能很快定位根因。最初确保重要工作拿到 GPU 时间的尝试，集中在更紧地管控优先级怎么设，最终演变成绕开优先级调度器、直接给重要项目划分 GPU 垄断。虽然当时没意识到，我们其实搭建了一个观察「公地悲剧」的完美实验室：个体在一个稀缺共享资源上竞争，各自追求个体最优，得到的是全局次优的结果，并且在过度损耗这个资源。
We were far from the first to observe this kind of interaction. Resource allocation is a fascinating research domain that mixes algorithm development, economics, and system management. A central problem is that users often know the value of their own jobs better than the organization does, but they may have incentives to hide that value or hold on to resources even when doing so hurts total performance. For example, in their 2011 paper introducing Dominant Resource Fairness, Ghodsi et al. recount an anecdote in which a search company provided dedicated machines to jobs only if their users could guarantee high utilization. They soon discovered "users would sprinkle their code with infinite loops to artificially inflate utilization levels." The hardware changes, but the fundamental problems that make resource allocation complex persist.
我们远不是第一批观察到这类互动的人。资源分配是个迷人的研究领域，混合了算法开发、经济学与系统管理。它的中心难题是：用户往往比组织更清楚自己任务的价值，却有动机隐瞒这个价值、或者占住资源不放——即使这样损害总性能。例如 Ghodsi 等人 2011 年提出主导资源公平（Dominant Resource Fairness，DRF）的论文里讲过一件轶事：一家搜索公司只把专用机器给能保证高利用率的任务，很快发现「用户会往代码里撒无限循环，人为把利用率刷上去」。硬件在变，但让资源分配变复杂的那些根本问题一直都在。
Budgets not schedules · 要预算，不要排程
The classic solution to a tragedy of the commons is to privatize the shared resource—owners are incentivized to maximize the value of their property. When we assigned teams monopolies over sets of GPUs, we were already doing a version of this, but it was too coarse. It caused GPUs to sit idle due to the seasonality of research. Teams are ready to run experiments and training at different times, so assigning a monopoly would ensure that there would be times when no jobs were ready to execute, with another team left waiting for capacity.
公地悲剧的经典解法是把共享资源私有化——所有者自然有动力把自己财产的价值最大化。我们给团队划分 GPU 垄断时，其实已经在做一个粗糙版本，但它太粗了：研究的季节性会让 GPU 闲着。各团队跑实验和训练的节奏不同步，划垄断意味着必然出现「有卡的团队没有就绪的任务在跑，而另一个团队在等卡」的时段。
We were manually solving a knapsack problem, trying to fit dynamically changing research needs into a static schedule. We wanted the ownership incentive, but we also wanted to maintain full occupancy of the GPUs.
我们等于在手工解一道背包问题（knapsack problem）：把动态变化的研究需求塞进一张静态时间表。我们想要所有权带来的激励，但也想让 GPU 保持满占用。
We decided to iterate on the ownership model. Instead of issuing teams GPUs, we chose to allocate a portion of GPU time. Predicting demand into the future would require knowing the result of novel science experiments, so it cannot be forecast with precision. Priority across research efforts, however, is a question of strategy, and it can be more easily debated and decided in advance. Instead of trying to solve the scheduling puzzle, we enabled leadership to think like investors. Before the workloads exist, decide how to fund each research effort with GPU time based on their judgment of its likely impact. The scheduler could then use that information when prioritizing arriving workloads.
我们决定迭代所有权模型：不发 GPU 给团队，改发一段 GPU 时间。预测未来的需求等于要预知全新科学实验的结果，做不到精准；但各研究方向之间的优先级是个战略问题，可以提前辩论、提前定。于是我们不再试图解调度谜题，而是让管理层像投资人一样思考：在负载还不存在的时候，就基于对潜在影响力的判断，决定给每个研究方向投多少 GPU 时间。调度器随后用这份信息给到达的负载排优先级。
With this in mind, we devised a hierarchical system where managers could proportionally allocate GPU time to the projects and researchers they were responsible for. As the diagram below illustrates, this translates program strategy directly into a guaranteed share of GPU time. Project A1 knows it has a 35% claim on total capacity, regardless of how many other projects are queuing up elsewhere.
沿这个思路，我们设计了一套层级系统：各级管理者按比例把 GPU 时间分给自己负责的项目与研究员。如下图所示，这把项目战略直接翻译成有保证的 GPU 时间份额——项目 A1 知道自己握有总产能 35% 的申请权，不管别处排着多少项目。
Parenthetical values represent the total cluster capacity assigned to a leaf project.
（图中）括号里的数值表示分配给叶子项目的总集群产能占比。
In this system, every request for GPU time must be funded by a budget, or it is not protected from preemption. In the old system, HIGH priority carried no cost and non-preemptibility allowed a team to fill their concurrent GPU limit indefinitely, so everyone used them. Now, nothing is free, so any trick to get GPU time draws from the benefiting user's allocation. A squatting workload is spending team budget on nothing. Our strategy is to make gaming the scheduler more expensive than honestly engaging in the debate for a larger budget. We are constantly iterating on this budget review process, but the key requirements are that there are frequent opportunities for researchers to advocate for the time they need, and the decisions are made by managers with the most context on the tradeoffs in question. This means allocation decisions within a research project are made by a lead researcher, within a research program by a principal investigator, and across programs by a lead program manager, or by the CEO.
在这套系统里，每一笔 GPU 时间申请都必须有预算出资，否则不受抢占保护。旧系统里 HIGH 优先级不花钱、不可抢占性能让团队无限占满并发上限，所以人人都用。现在没有什么是免费的：任何骗取 GPU 时间的花招，花的都是受益者自己的额度——占坑的负载就是在拿团队预算买空气。我们的策略就是让博弈调度器比诚实参与「争更大预算」的辩论更贵。预算评审流程本身还在持续迭代，但关键要求有两条：研究员有频繁的机会为自己需要的时间陈情；决策由最了解相关取舍的管理者来做。具体说：一个研究项目内部的分配由 lead researcher 定，一个研究计划内由项目负责人（principal investigator）定，跨计划由首席项目计划经理定，或者由 CEO 定。
Fair-share · 公平共享
Paired with this GPU time budgeting tool, we built a hierarchical fair-share scheduler to manage actual occupancy of allocations throughout the program tree. The algorithm here is not new—hierarchical fair-share over a time window is part of a lineage that goes back to the Hadoop Fair Scheduler in 2009, and the same approach is in active use today in SLURM's Fair Tree and YARN's Fair Scheduler. What's new for us are the inputs: the tree mirrors the research program structure, and the weights are budgets set by managers rather than static quotas.
与 GPU 时间预算工具配套，我们造了一个层级公平共享（hierarchical fair-share）调度器，管理各预算在整个项目树上的实际占用。算法本身不新——时间窗口上的层级公平共享，谱系可以追到 2009 年的 Hadoop Fair Scheduler，今天 SLURM 的 Fair Tree 与 YARN 的 Fair Scheduler 仍在用同一思路。对我们而言新的是输入：这棵树镜像研究组织的结构，权重是管理者定的预算，而不是静态配额。
The scheduler tracks occupancy over a sliding lookback window (we default to 7 days) and sorts workloads from under-utilized allocations above those from over-utilized allocations. This way, over a week-long time range, we can expect every group to receive their allocated GPU time as long as they are actively submitting workloads with sufficient demand.
调度器在一个滑动的回看窗口（lookback window，我们默认 7 天）内跟踪占用情况，把「用得低于份额」的预算下的负载排在「用超了份额」的之前。这样在一周的时间尺度上，只要团队持续提交足够需求的负载，每个组都能拿到自己分到的 GPU 时间。
"The new scheduler makes it feel like we have an extra 30% compute. In the old scheduler, if we had moments when we didn't need our full slot limit, that compute was basically lost. Now with the new scheduler, if that happens, we can later burst beyond our allocation limit and still see our jobs scheduled quickly and without preemption, essentially letting us reclaim that compute. Our workloads are often bursty, so this gave us a significant amount of compute back." — Chris Clark
「新调度器让我们感觉凭空多了 30% 算力。旧调度器下，如果我们有段时间用不满自己的槽位上限，那些算力基本就浪费了；现在如果发生这种情况，我们之后可以爆发式超出分配上限，任务照样快速排上、不被抢占——等于把那些算力赚了回来。我们的负载经常是爆发式的，所以这为我们找回了一大块算力。」——Chris Clark
The scheduler distinguishes two kinds of occupancy. Allocated occupancy is time during which a workload is charged to a budget. This draws from the workload owner's allocations, which affects the fair-share budget calculation, and these workloads are protected from preemption during their minimum runtime window. Unallocated occupancy is not charged to any budget, is unprotected from the outset, and may be preempted by any allocated request. This allows us to keep the GPUs fully occupied even when allocations don't properly match demand and prevents teams from ever declining free GPU cycles.
调度器区分两种占用。已分配占用（allocated occupancy）：负载的时间记在某个预算头上——它消耗负载所有者的额度、影响公平共享的计算，并且在最短运行时窗口内受抢占保护。未分配占用（unallocated occupancy）：不记任何预算、从头就不受保护、任何已分配请求来了都可以抢占它。这让我们在预算与需求错配时也能保持 GPU 满载，也让团队永远不会拒绝白送的 GPU 周期。
The scheduling contract · 调度契约
An additional feature of distributed training that makes fair resource allocation difficult is that workloads can run for a very long time. Training jobs regularly run for hours, days, and sometimes even weeks. Once scheduled, a workload could remain on its assigned GPUs for a week or more, providing no opportunity for others to receive their budgeted time. This is the system property that made GPU squatting possible. It's also what forced on-call engineers to negotiate with long-running job owners to address ongoing maintenance issues.
分布式训练还有一个让公平分配变难的特性：负载可以跑非常久。训练任务动辄数小时、数天、有时数周。一旦排上，一个负载可以在分到的 GPU 上赖一周以上，别人预算内的时间永远轮不到——这正是让 GPU 占坑成为可能的系统属性，也是逼着 on-call 工程师去跟长任务主人谈判解决维修问题的同一个原因。
To address these problems, we introduced a "scheduling contract." In exchange for access to the cluster, a workload must declare its minimum runtime, or the shortest amount of occupancy required to make meaningful progress. During this time, a workload is protected from preemption. This gives the researcher a guarantee of progress, while giving the scheduler the right to rebalance once that progress is banked, automatically re-queueing resumable workloads. Alternatively, a user can set minimum runtime to zero, which indicates that the GPU time should be unallocated. These workloads are always subject to preemption, but they're also free in the sense that they are not charged to any budget.
为解决这些问题，我们引入了「调度契约」（scheduling contract）：作为上集群的交换，负载必须声明自己的最短运行时（minimum runtime）——做出有意义进展所需的最短占用时间。在这段时间内负载受抢占保护。这给了研究员一个进度保证，同时给了调度器「进度落袋后重新平衡」的权利——可恢复的负载会被自动重新排队。另一种选择是把最短运行时设为零，表示这段时间按未分配（unallocated）计：这类负载永远可被抢占，但也免费——不记任何预算。
The workload lifecycle follows this pattern: - The workload is submitted with a minimum runtime and indicates whether or not it is resumable. - The workload is scheduled according to the fair-share algorithm, weighted by a ratio of actual occupancy to allocated time in the lookback window. - The workload runs for its minimum runtime, which is charged to its allocations. - The workload may continue running as long as the associated allocations continue to prioritize it over others. This time is also charged to its allocations. - It may be preempted and requeued, which returns to step 2. - The workload completes, releasing its claim on any resources.
负载生命周期遵循这个模式：负载带着最短运行时提交，并声明自己是否可恢复；按公平共享算法调度——权重是在回看窗口内实际占用与分配时间的比值；跑满最短运行时，这段时间记入它的预算；只要所属预算继续把它排得比别人高，它就可以继续跑，这段时间同样记预算；它可能被抢占并重新排队，回到第 2 步；负载完成，释放全部资源占用。
Together, these agreements add time-slicing to our scheduler. Running workloads may be removed and requeued automatically, allowing fair-share to converge and disincentivizing squatting. They also let unhealthy hosts drain their workloads as they reach their minimum runtimes, so repair activities can be fully automated. This last point was more important than we realized when planning this work. It reduced repairs requiring a human-in-the-loop by 74%, which was a massive savings in on-call toil.
这些约定合在一起，给调度器加上了时间切片（time-slicing）。运行中的负载可以被自动取下重排，公平共享因此能够收敛，占坑失去意义。不健康的主机也能等负载跑到最短运行时后自动排空，维修活动得以全自动化。最后这一点在规划时我们低估了它的重要性：需要人在环路介入的维修减少了 74%，on-call 苦役省了一大块。
Simulations · 仿真
We know that scheduling policy changes can have unintended consequences. The zero-sum nature of the problem means that giving time to one researcher means taking away from another. Users who lose this exchange tend to look for new workarounds. Before rolling out the budget-based system, we wanted a fast way to predict where those longer wait times might arise and to test configuration knobs like the length of the lookback window or the maximum value to allow for minimum runtime (we chose 8 hours).
我们知道调度策略的改动可能有非预期后果。这个问题的零和性质意味着给一个研究员时间就是从另一个人那里拿走——在这场交换里吃亏的用户会去找新的绕行办法。在上线预算制系统之前，我们想要一种快速方法来预测哪里会出现更长等待，并测试配置旋钮——比如回看窗口多长、最短运行时允许的最大值（我们选了 8 小时）。
We built a small simulation environment that takes a set of workloads and their submission schedule as input and allows the scheduler to make preemption and GPU assignment decisions. With the knowledge of each workload's requested number of GPUs and total runtime, the simulator could jump ahead to schedulable moments and provide analysis of queue wait times, preemption events, and distribution of GPU time across projects for many simulated days in a few seconds. We ran the simulator against both historical submission data and constructed scenarios we wanted to better understand.
我们造了一个小型仿真环境：输入一组负载及其提交时间表，让调度器在里面做抢占与 GPU 分配决策。由于知道每个负载要的 GPU 数和总运行时长，仿真器可以直接跳到下一个可调度时刻，几秒钟跑完很多天的模拟，给出排队等待、抢占事件、各项目 GPU 时间分布的分析。我们既跑了历史提交数据，也跑了为验证特定猜想而构造的场景。
One hypothesis we wanted to test involved "debug workloads." These jobs require a small number of GPUs and a minimum runtime of 15 minutes or less, which is enough for the user to see whether a job launches successfully or crashes early due to a bug or misconfiguration. We wanted to know whether these jobs would see a shorter queue wait time than larger training workloads, which often need many GPUs and hours of runtime to make meaningful progress. Intuitively, these smaller jobs should rise to the top of the queue, since a small job can fit more places than a large one. But the precise queue latency was important. A short wait of a minute or two would unlock a new development practice, but a ten-minute wait becomes infeasible.
我们想验证的一个假设关于「debug 负载」：这类任务只要少量 GPU、最短运行时 15 分钟以内——足够用户看清任务是成功启动了，还是因为 bug 或配置错误早早崩掉。我们想知道这些任务排队会不会比大训练负载短——后者往往要很多卡、数小时运行才算有进展。直觉上小任务应该浮到队首，因为小任务塞得进更多空隙；但精确的排队延迟很关键：等一两分钟能解锁新的开发实践，等十分钟就不可行了。
Our simulations required hand-built test case data, because our historical record did not contain a high enough volume of these debug-like workloads. Our results supported the hypothesis, showing p90 debug workload wait times fall from about 6 hours to only 5 minutes.
仿真需要手工构造测试数据，因为历史记录里这类 debug 型负载量不够。结果支持了假设：debug 负载的 p90 等待时间从约 6 小时降到只有 5 分钟。
Smaller-scale simulator visualization of the baseline (left) and the new "allocations" scheduler (right). Each row is one GPU; each bar is a job, colored by parent workload with one color hue per team; hatching marks time where a job is interruptible, and a red edge marks a preemption. In the baseline, long urgent jobs are never interrupted, and fewer preemptions occur on lower-priority-level work. The new scheduler has a larger mix of colors on each GPU, illustrating occupancy rotation across teams.
小规模仿真可视化——左为基线，右为新的「预算分配」调度器。每行一块 GPU，每个色条是一个任务，颜色按所属团队区分；斜纹标记任务可被打断的时段，红边标记一次抢占。基线里，长的紧急任务从不受打扰，低优先级任务被抢占得更少（因为根本排不上）；新调度器里每块 GPU 上颜色更混杂，展示占用在各团队间轮转。
Results · 结果
With simulation results in hand, we began a cluster-by-cluster rollout at the end of July. The results we care about are whether the workloads we chose to fund received their time, whether the new system maintained full occupancy, and whether researchers could reason about the scheduler to make informed decisions.
拿着仿真结果，我们在 7 月底开始逐集群上线。我们关心的结果有三件事：我们选定出资的负载是否拿到了时间；新系统是否保住了满占用；研究员是否能理解调度器的行为、据此做明智决策。
Since rollout, we have observed users and teams consistently receiving their allocated GPU time. We count the time owed to a team as its allocation capped hour by hour at its actual demand. Over the 30-day test period, teams were delivered 98% of the GPU hours they were owed, and 13 of 15 team allocations received 95% or more with the worst case receiving 90%. Occupancy on the cluster held steady at 98% before and after the change, with demand exceeding capacity by 2-3x in both periods. 18% of delivered GPU time was unallocated, which is how we maintained high occupancy during periods when funded use cases were not ready to run.
上线以来，用户与团队持续拿到分配内的 GPU 时间。我们把团队应得时间按小时记为「其分配额度与实际需求的较小者」。30 天测试期内，团队拿到了应得 GPU 小时的 98%；15 个团队分配中 13 个拿到 95% 以上，最差的一个也拿到 90%。集群占用在变更前后都稳定在 98%，两个时期需求都超容量 2-3 倍。已交付的 GPU 时间里 18% 是未分配的——这正是出资用例还没就绪的那段时间里我们保住高占用的方式。
Our simulator results proved to be directionally accurate with real outcomes overperforming our predictions. Debug workload p90 queue wait time fell from 2 hours to 30 seconds under the new scheduler, against a simulated prediction of 6 hours to 5 minutes from hand-crafted test scenarios. It's worth noting that the smaller sample size of debug workloads in the baseline meant there was higher variance in those measurements. Queue latency in general improved as a side effect of time-slicing: on our largest H100 cluster, median queue wait time fell from 5 minutes to 24 seconds, and p90 wait time fell by about a third (from 2.8 hours to 1.8 hours).
仿真结果方向上准确，真实成绩还好于预测：新调度器下 debug 负载 p90 排队时间从 2 小时降到 30 秒，而手工场景的仿真预测是从 6 小时降到 5 分钟。值得一提的是，基线里 debug 负载样本量较小，那些测量值方差偏高。整体排队延迟也作为时间切片的副作用改善了：最大的 H100 集群上，中位排队等待从 5 分钟降到 24 秒，p90 等待降了约三分之一（从 2.8 小时到 1.8 小时）。
Compared against the three problems we set out to solve: - Squatting: Short debug workloads start in under a minute, reducing the value of squatting. The cost of this behavior charges the squatter's budget, which prevents them from receiving time when they really need it. - Priority inflation: We still allow workloads to declare priority, but it only impacts sorting within a team. Managers are incentivized to monitor priority across the group to optimize the use of their budgets. - On-call toil: Unhealthy hosts drain automatically as workloads reach minimum runtime. Repairs requiring a human-in-the-loop fell by 74%.
对照出发时要解决的三个问题：占坑——短 debug 负载一分钟内启动，占坑的价值大减；这种行为还烧占坑者自己的预算，让他们真正需要时反而拿不到时间。优先级通胀——我们仍允许负载声明优先级，但它只影响团队内部的排序；管理者因此有动力盯着组内优先级的使用，把预算花在刀刃上。on-call 苦役——不健康主机在负载到达最短运行时后自动排空，需要人在环路介入的维修降了 74%。
Challenges · 挑战
The learning curve was steeper than we had assumed. We rolled out the change incrementally, so in the early days researchers experienced varied behavior depending on which cluster they targeted. Furthermore, our interfaces retained some old terminology (like workload priority) whose meanings had changed. Documentation alone did not resolve the confusion. What did work was conducting live explanatory sessions, providing a forum for researchers to ask questions and for the engineering team to provide deeper descriptions of both how and why the scheduler made its prioritization decisions using real examples.
学习曲线比我们预想的陡。变更是渐进上线的，早期研究员在不同集群上遭遇的行为不一样；加上界面里留了一些旧术语（比如「负载优先级」），含义已经变了。光靠文档没有解决困惑。真正起作用的是开现场讲解会：给研究员一个提问的场子，工程团队用真实例子把调度器「怎么排」和「为什么这么排」讲透。
This was a key moment because it marked a pivot from an early period of frustration and folk theories to the current mode, where research groups communicate more often and more broadly about the GPU needs of their experiments. Researchers are now engaging in the budgeting discussion with clearer knowledge of the tradeoffs being made to accommodate any new request.
这是个关键节点：它标志着从早期的挫败与民间猜想，转向现在的模式——各研究组更频繁、更公开地沟通实验的 GPU 需求。研究员现在带着对取舍的清晰认知参与预算讨论：知道为容纳任何新申请，牺牲的是什么。
In addition to in-person sessions, we introduced new visualizations post-launch to give users a better sense of how closely their assigned GPU time is tracking their expected allocations and directly exposing the metric used for sorting the workload queue. This provided a simple place to look when a workload was preempted to understand why. These visualizations also helped budget owners, who could see how GPU time was being used across various projects under their management.
现场会之外，我们在上线后引入了新的可视化：让用户看清自己实拿的 GPU 时间与预期分配跟得有多紧，并直接暴露用于负载排队排序的那个指标。负载被抢占时，这里有处可查、能看懂为什么。这些可视化对预算持有者也有用——能看见自己管辖下各项目的 GPU 时间是怎么用的。
Example visualization of allocation usage over time.
分配额度使用情况随时间变化的可视化示例。
Not every use case improved. In addition to distributed training, our researchers launch interactive sessions where they perform data analysis and test training code while they write it. In the old system, a researcher could hold such a session for up to a week. With time-slicing, they were subject to the 8-hour cap on protected runtime, after which a session becomes preemptible if it exceeds its allocation. We did not appreciate the extent to which researchers were dependent on the volatile state of these sessions. Being preempted meant waiting to secure a new session and also rebuilding their state by hand. After surveying researchers to understand the breadth of this problem, we created two new roadmap projects. We are investing in a CPU-only cluster next to our on-prem storage for dev sessions focused on data prep tasks. This will preserve our training cluster capacity for workloads that truly need it. Additionally, we plan to build restorable sessions for these CPU-only workloads. This will allow us to continue to preempt workloads at the end of their minimum runtime for maintenance or time-slicing, while also being able to restore the session elsewhere without the researcher rebuilding it. We can retain the operational and scheduling benefits of this new system while also improving the user experience.
不是所有用例都变好了。除了分布式训练，研究员还会开交互式会话——边写代码边做数据分析、试训练代码。旧系统里这样的会话可以挂一周；时间切片之下，它们受 8 小时保护运行时上限约束，超时后再超出分配就变成可抢占。我们低估了研究员对这些会话里易失状态的依赖程度：被抢占意味着既要在排队里等新会话，又要手工重建状态。调研摸清问题范围后，我们立了两个新路线图项目：一是投资一个紧挨本地存储的纯 CPU 集群，专门跑以数据准备为主的开发会话——把训练集群产能留给真正需要它的负载；二是为这些 CPU 负载建设可恢复会话——让我们能继续在最短运行时结束时为了维修或时间切片抢占负载，同时能把会话在别处恢复，不用研究员手工重建。这样既保住新系统的运维与调度收益，又把用户体验补上。
We remain on the lookout for emerging issues. One potential problem we're investigating is capacity fragmentation, which may result in queue wait times increasing for the largest workloads. Our intuition is that minimum runtime protection is being applied to the types of jobs that used to rely on preemptible mechanisms to exceed their team concurrent GPU limit. Previously, those jobs could be interrupted at any time, potentially wasting that time, but also making it easier to schedule large jobs. Now the scheduler may have fewer opportunities to interrupt many jobs at once to place a large pending workload. We're currently using our simulator tools to reproduce this problem while also measuring the ground truth in production.
我们保持对新问题的警惕。正在调查的一个潜在问题是容量碎片化，它可能让最大型负载的排队时间变长。我们的直觉是：最短运行时保护如今被用在了过去靠「可抢占机制超出团队并发上限」的那类任务上——以前它们随时可被打断（那段时间可能浪费），但也因此更容易调度大任务；现在调度器能「一次打断一批任务、腾出整块地方放一个大负载」的机会变少了。我们正在用仿真工具复现这个问题，同时在生产环境测量真实情况。
The future · 未来
As we look beyond the scheduling work described here, we are aiming at the capstone of the pyramid: utilization. We need to ensure that bootstrapping, checkpointing, and the training applications themselves are all done as efficiently as possible, maximizing the value of the scheduled time each workload receives.
展望这篇讲的调度工作之外，我们的下一个目标是金字塔的塔尖：利用率。要确保启动引导、检查点保存与训练应用本身都尽可能高效，把每个负载分到的调度时间的价值榨到最大。
If you'd like to tackle challenges like these in close collaboration with researchers, we encourage you to explore open engineering roles at Ai2.
如果你想与研究员近距离协作、解决这类挑战，欢迎了解 Ai2 的工程岗位（原文有链接）。