Today we're releasing Olmo-core 3, a significant upgrade to our framework for developing large language models featuring a redesigned open mixture-of-experts (MoE) training system.
今天我们发布 Olmo-core 3——我们大模型开发框架的一次重大升级，核心是重新设计的开放混合专家（MoE）训练系统。
Olmo-core 3 is designed to scale MoE training into the trillion-parameter range while preserving computational efficiency. It's one of the core systems behind the next generation of Olmo, and part of our ongoing commitment to open up the tools and training infrastructure behind each new model.
Olmo-core 3 的设计目标：把 MoE 训练扩到万亿参数量级同时保住计算效率。它是下一代 Olmo 背后的核心系统之一，也是我们持续承诺的一部分——把每个新模型背后的工具与训练基建一并开放。
Training large AI models takes a lot of compute, driving up costs and energy use and putting advanced model development out of reach for many academic researchers and smaller labs. MoE models offer a more efficient approach—they can contain many more learned components, or parameters, without requiring every input to use all of them. But the full model still has to be stored across GPU memory and updated during training, and directing inputs to the right experts – the specialized components within an MoE – across a cluster creates its own communication and coordination costs. As MoEs grow, those costs can erode much of the computational advantage of using only part of the model for each input.
训练大模型要花大量算力，推高成本与能耗，把前沿模型开发挡在许多学术研究者和小实验室门外。MoE 提供了更高效的路线——容纳多得多的学习组件（参数），而不要求每个输入都用上全部。但完整模型仍要摊在 GPU 显存里、训练时仍要全部更新；而把输入送往正确的专家（MoE 里的专职组件）又要跨集群付出通信与协调成本。MoE 越大，这些成本越能侵蚀「每个输入只用一小部分模型」带来的计算优势。
Olmo-core 3 is built to close that gap. In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.
Olmo-core 3 就为堵住这个缺口而造。一项基准里，我们把专家池从 8 扩到 128、每 token 仍只选 4 个专家（token 是语言模型处理文本的小单元），每 token 激活参数大致钉在约 3.2B——总参数容量从 4.6B 涨到 47B，训练吞吐掉得还不到 5%。
The same infrastructure has been benchmarked at over one trillion total parameters.
同一套基建已在超过一万亿总参数的规模上完成基准测试。
Building a training stack around how MoEs actually work · 围绕 MoE 的真实工作方式搭训练栈
Olmo-core has evolved with each generation of Olmo. Our work on sparse models goes back to OlmoE, which used an MoE architecture with 64 routed experts. Olmo 3, by contrast, used a dense architecture, meaning nearly all of the model was active for every token and its training stack was built around that design. Olmo-core 3 extends the framework with a training system designed for much larger MoE models.
Olmo-core 随每一代 Olmo 演进。我们的稀疏模型工作追溯到 OlmoE（64 个路由专家的 MoE 架构）；Olmo 3 反而是稠密架构——几乎整个模型对每个 token 都激活，训练栈也围绕稠密设计。Olmo-core 3 为框架扩展出面向大得多的 MoE 模型的训练系统。
Our earlier MoE implementation in Olmo-core used fully sharded data parallelism (FSDP), configured to gather and reshard model weights for each small batch of training data. Olmo-core 3 switches to a system based on distributed data parallelism (DDP). It keeps experts resident on GPUs and routes the relevant data to them, avoiding that repeated weight gathering.
我们早期在 Olmo-core 里的 MoE 实现用的是全分片数据并行（FSDP）——每小批训练数据都要把模型权重聚合再切分。Olmo-core 3 切换到基于分布式数据并行（DDP）的系统：专家常驻 GPU、把相关数据路由过去，省掉反复的权重聚合。
NVIDIA's Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation. In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput.
NVIDIA 的 Megatron-Core 是训练大型 MoE 的成熟选项。Olmo-core 3 给 Olmo 背后的框架带来了集成的 MoE 训练栈，经重新设计后吞吐超过我们早先的 FSDP 实现：八张 NVIDIA B300 的初步测试里，47B 参数 MoE 每 GPU 每秒处理 52000 token，旧实现只有 19400——约 2.7 倍吞吐。
Scaling and optimizing MoE training · 扩展与优化 MoE 训练
Olmo-core 3 combines several techniques for distributing large MoEs across GPU clusters with optimizations that make routing and computation more efficient. Three techniques determine how the model and its training state are split across hardware: Expert parallelism spreads the experts across GPUs, so each GPU stores only part of the full expert pool. Pipeline parallelism splits the model's layers – the successive stages that transform an input – across groups of GPUs, reducing how much of the model each GPU needs to keep in memory. A distributed optimizer spreads the optimizer state – the additional data used to calculate and apply updates during training – across GPUs instead of storing a full copy on every GPU. Together, these techniques allow an MoE to scale without requiring every GPU to keep the entire model and its training state in memory.
Olmo-core 3 把「跨 GPU 集群摊开大型 MoE」的若干技术与「让路由和计算更高效」的优化组合起来。决定模型与训练状态怎么切到硬件的是三并行：**专家并行**把专家摊到各 GPU（每卡只存专家池的一部分）；**流水线并行**把模型的层（逐级变换输入的阶段）切给 GPU 组（降低单卡要驻存的模型量）；**分布式优化器**把优化器状态（训练中计算并应用更新所需的额外数据）摊到各 GPU，而不是每卡存一份全量。三者合力，MoE 扩容不再要求每张 GPU 装下整个模型及其训练状态。
Olmo-core 3 also reduces the cost of routing data to the right experts and running their computations. Rowwise expert parallelism places routed data directly into expert input buffers, minimizing the extra work needed to rearrange it. GPU-resident routing keeps routing metadata on the GPUs, so the CPU can queue work without waiting for that information to be copied back. And grouped GEMM combines many small expert computations so GPUs can execute them more efficiently.
Olmo-core 3 还降低了「把数据路由到对的专家并完成计算」的成本：**行级专家并行**把路由数据直接放进专家的输入缓冲，省掉重排的额外功；**GPU 常驻路由**让路由元数据留在 GPU 上，CPU 排活儿不必等它拷回来；**grouped GEMM** 把许多小的专家计算合并，GPU 执行更高效。
Finally, Olmo-core 3 supports MXFP8, a lower-precision number format that represents some values with fewer bits. This can reduce computation and the amount of data moved between GPUs, as long as those savings outweigh the cost of converting between number formats. We measured MXFP8's effect on end-to-end training throughput in a controlled benchmark on four NVIDIA B300 GPUs, with work distributed uniformly across experts. With MXFP8 enabled across the parts of the system where it helped most, training throughput was about 21% higher than with BF16, the higher-precision format we used as our baseline, while peak active memory fell from 103 GiB to 95 GiB. Most of the gain came from feed-forward computation and moving data between experts rather than attention alone.
最后，Olmo-core 3 支持 MXFP8——一种用更少位表示部分数值的低精度格式，能省计算与跨 GPU 搬运量，前提是这些节省盖过格式转换的成本。我们在四张 B300 的受控基准上测了 MXFP8 对端到端训练吞吐的影响（负载均匀分布到专家）：在收益最大的部分启用 MXFP8 后，吞吐比 BF16 基线高约 21%，峰值激活显存从 103 GiB 降到 95 GiB。收益大头来自前馈计算与专家间数据搬运，而非注意力本身。
These techniques and optimizations have to work together. Speeding up one part of training can create costs elsewhere; faster computation may require more data movement, while moving fewer bits may not help if converting the data takes too long. Olmo-core 3 is built around those trade-offs across the full training process, giving us – and researchers using the open stack – control over how the pieces fit together. Explore our interactive walkthrough to see how data, expert, and pipeline parallelism work together to scale MoE training—from a single GPU to many.
这些技术与优化必须协同：加速训练的一处可能在别处产生代价——计算变快可能要搬更多数据，位搬得少了若转换太慢也白搭。Olmo-core 3 就是围绕全训练流程的这些权衡构建的，让我们（以及用这套开放栈的研究者）掌控各部件怎么拼。交互式 walkthrough 演示了数据/专家/流水线三种并行怎么协同把 MoE 训练从单卡扩到多卡。
Scaling into the trillion-parameter range · 扩进万亿参数区间
We've benchmarked Olmo-core 3 across a range of configurations on NVIDIA B300 GPUs, including a 1.2-trillion-parameter model with 58.36 billion parameters active per token across 512 GPUs. Its highest observed throughput was 858 TFLOP/s/GPU—a measure of useful model computation per second on each GPU. These tests used random routing to measure system performance, rather than the quality of a trained model.
我们在 B300 上对多组配置做了基准，包括一个 1.2 万亿参数、每 token 激活 58.36 亿参数、横跨 512 张 GPU 的模型，最高观测吞吐 858 TFLOP/s/GPU（每卡每秒有效模型计算的度量）。这些测试用随机路由测的是系统性能，不是训练出的模型质量。
We've also experimented with DeepEP v2, an alternative way of handling communication between experts across GPUs, reaching a configuration with 2.38 trillion total parameters. This was a short-capacity test rather than a full training run, so it demonstrates the scale Olmo-core 3 can reach rather than sustained training performance.
我们还试了 DeepEP v2——另一种处理专家间跨 GPU 通信的方式——摸到了 2.38 万亿总参数的配置。这是短时容量测试而非完整训练，展示的是 Olmo-core 3 能到的规模，不是持续训练性能。
At these scales, systems performance is only part of the picture. Our technical report also documents experiments that informed how we train MoEs and measure their performance. For example: A score intended to encourage balanced routing could improve even as the actual workload became less balanced. We call this failure token gerrymandering. Lowering experts' learning rates – the size of their training updates – because they process fewer tokens did not improve results in the model family we tested. GPU calculations took different amounts of time when the values being processed changed, even with the same matrix dimensions. Performance comparisons therefore need matching input values as well as matching shapes. Overlapping communication and computation on separate GPU streams did not always make training faster. In some tests, it slowed end-to-end execution—a reminder that more overlap does not necessarily mean higher throughput.
到这个规模，系统性能只是画面的一半。技术报告还记录了指导我们怎么训 MoE、怎么测性能的一批实验，例如：**token gerrymandering（选区划分式作弊）**——一个本意鼓励路由均衡的分数，可以在真实负载越来越不均衡的同时照样上涨；因为专家见的 token 少就压低其学习率（训练更新的步幅），在我们测的模型家族里没有改善结果；GPU 计算耗时随被处理数值的内容而变——即使矩阵形状完全相同，所以性能对比要配数值不配形状；在独立 GPU 流上重叠通信与计算并不总是更快，某些测试里反而拖慢端到端——重叠多不等于吞吐高。
The report explains these findings alongside the approaches we tested and chose not to adopt.
报告把这些发现与「我们测过但选择不采用」的路线一并讲清。
Built for the next generation of Olmo, open for everyone · 为下一代 Olmo 而造，向所有人开放
Olmo-core 3 is the foundation for what we're building next. Our next-generation Olmo will use an MoE architecture, and we're aiming for it to be our most capable Olmo yet, trained on our largest dataset and with our longest context window. The new stack lets us scale beyond our previous MoE work while giving us more flexibility to adapt training as models and hardware evolve. And it's fully open—researchers and developers can use Olmo-core 3 to train their own MoEs, adapt it to different hardware, and experiment with routing, parallelism, and other parts of the system. That's part of how we think about open model development—model weights are more useful when the infrastructure and training decisions behind them are open too. For a deeper look at the systems design, experiments, ablations, and approaches we tested along the way, read our technical report and explore Olmo-core 3 on GitHub.
Olmo-core 3 是我们下一步的地基：下一代 Olmo 将用 MoE 架构，目标是最强的一代 Olmo——最大数据集、最长上下文窗口。新栈让我们越过往日的 MoE 工作，也留足随模型与硬件演进调整训练的灵活度。而且完全开放——研究者与开发者可以用 Olmo-core 3 训自己的 MoE、适配不同硬件、实验路由与并行等各部分。这也是我们对开放模型开发的立场：权重背后的基建与训练决策也开放，权重才更有用。系统设计、实验、消融与试过的路线，见技术报告与 GitHub 上的 Olmo-core 3。