Native-script data for 22 languages, romanized data for twelve, and a reusable GPU-accelerated pipeline for translation, quality assessment, and repair.
22 种语言的原生文字数据、12 种语言的罗马化数据，外加一条可复用、GPU 加速的翻译-质量评估-修复管线。
Authors: AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.
作者：AI4Bharat（Mohammed Safi Ur Rahman Khan、Mitesh Khapra、Anoop Kunchukuttan）与 NVIDIA（Niket Agarwal、Raviraj Joshi、Anusha Kamath、Markus Kliegl、Yev Meyer、Sriharsha Niverty、Kiran Praveen、Utkarsh Vaidya）。
We're excited to introduce Nemotron-pretraining-Indic-v1: approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it. These are the 22 languages listed in India's Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census. Developed by NVIDIA with IIT Madras, India's premier engineering institute, and its AI4Bharat lab, this release gives model builders a substantial new resource for Indic pretraining. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material and generate more data for their needs.
我们很高兴发布 Nemotron-pretraining-Indic-v1：横跨 22 种印度语言、约 2.8 万亿（2.8T）token 的语料，随同发布的还有造出它的开放 GPU 加速数据生成管线。这 22 种语言是印度宪法列出的语言：按 2011 年人口普查，是 97% 印度人（近 12 亿，1.2 billion 公民）的第一语言。由 NVIDIA 与印度顶尖工学院 IIT Madras 及其 AI4Bharat 实验室联合开发，给模型构建者提供一份可观的印度语言预训练新资源。管线完整记录了生成配方——提示词、LLM 工作流与质检——开发者可以适配自己的源材料、按需生成更多数据。
AI4Bharat, IIT Madras' Indian language research lab, builds open-source datasets and models for Indian languages. Its resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, have helped lay the foundation for open Indic language-model research. Our joint open-source release represents a substantial expansion of the open synthetic Indic training data that will be available to the community. It brings a broad mix of curated English material, including mathematics, code, and question-answer content, into Indian languages, giving teams more training material for building models that can explain technical concepts, work through problems, and answer questions in those languages. Every record traces back to a curated source document, and each output is checked against its input, so the synthetic text stays grounded in real information.
AI4Bharat 是 IIT Madras 的印度语言研究实验室，构建印度语言的开源数据集与模型。它的资源——比如 22 种印度语言的开源预训练语料 Sangraha——为印度语言模型的开源研究铺了地基。我们的联合开源发布大幅扩充了社区可用的开放合成印度语言训练数据：把一批精心筛选的英文材料（数学、代码、问答内容等）引入印度语言，给团队更多训练素材，去构建能用这些语言讲解技术概念、推演问题、回答问题的模型。每条记录都能溯源到筛选过的源文档，每个输出都对照输入检查过——合成文本始终扎根于真实信息。
Mitesh Khapra, who heads AI4Bharat and made the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because "we do not have enough data for Indian languages." We're answering that call with more data, plus an open recipe for generating more.
AI4Bharat 负责人、2025 年 TIME100 AI 榜单人物 Mitesh Khapra 点明过核心问题：印度语言技术落后于英文，是因为「我们没有足够的印度语言数据」。我们用更多数据、外加一份能生成更多数据的开放配方来回应这个呼吁。
The corpus also accounts for how people write. It includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Our workflows combine document-level translation, quality assessment, and targeted repair, with checks to preserve content such as equations, tables, and code. This data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.
这份语料也照顾到了人们实际的书写方式：22 种语言全有原生文字数据，12 种另有罗马化文本，共 34 种语言-文字配置。我们的工作流组合了文档级翻译、质量评估与定向修复，并带保护公式、表格与代码的检查。这些数据是自然创作的印度语言文本的补充——后者对本地知识与文化语境仍然不可或缺。
We're sharing the open generation recipe so the community can not just reproduce the data but take this work further. Teams can use the corpus directly, adapt the prompts and quality checks, or bring their own source documents to the workflow. By opening up both the data and the process, we aim to make large-scale Indic data generation more accessible and enable more developers to contribute to stronger models across Indian languages.
我们分享开放生成配方，是为了让社区不只是复现数据，还能把这项工作推得更远：团队可以直接用语料、改写提示词与质检，或者把自己的源文档带进工作流。数据与流程双双开放，我们想让大规模印度语言数据生成更可及，让更多开发者能贡献于更强的印度语言模型。
Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text. Our technical report on the model covers the full story. We plan to build on this corpus in future Nemotron models.
Nemotron 3.5 Super VL 的继续预训练（CPT）加入了 100 亿印度语言 token，取自本语料的十二语言子集与自然印度语言文本。模型的完整故事见我们的技术报告；未来的 Nemotron 模型还会继续在这份语料上建设。
What is in the corpus? · 语料里有什么
Figure 1. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.
图 1：语料含 2,601.0B 原生文字 token 与 224.6B 罗马化 token（Gemma-4 tokenizer 口径）。文档记录包含同一来源的翻译与文字变体，不是去重后英文文档的计数。
The corpus has two groups of language configurations. Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu. Native-script data: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.
语料分两组语言配置。**原生文字+罗马化双轨**：孟加拉语、古吉拉特语、印地语、卡纳达语、马拉雅拉姆语、马拉地语、尼泊尔语、奥里亚语、旁遮普语、泰米尔语、泰卢固语、乌尔都语（12 种）。**仅原生文字**：阿萨姆语、博多语、多格拉语、克什米尔语、孔卡尼语、迈蒂利语、曼尼普尔语、梵语、桑塔利语、信德语（10 种）。
Here, native-script describes the writing system: the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. For each of the twelve dual-script languages, romanization covers a subset of the accepted native-script translations.
这里的「原生文字」描述的是书写系统——文本仍是从英文翻译来的；「罗马化」指用拉丁字母书写的印度语言文本，不是英文翻译。十二种双文字语言的罗马化数据，各取自已接受的原生文字翻译的一个子集。
This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.
构造训练配比时这个区分很关键：原生与罗马化记录提供不同的文字暴露，但可能表达同一底层内容。token 计数描述的是留存语料（含被接受的修复重写），不是某次训练运行的 token 预算；计数也依赖所用 tokenizer——语料盘点与训练 token 数应该分开读。
From English source documents to Indic training data · 从英文源文档到印度语言训练数据
To build this corpus, we start with documents from 15 Nemotron-CC-v2 subsets, covering web text, question-answer material, mathematics, code, and other specialized content. We use NeMo Data Designer, NVIDIA's open-source framework for generating synthetic data, to run translation, quality checks, and targeted repair. Throughout the workflow, we retain each source identifier so developers can trace an output back to its original document. Clear provenance like this is part of what makes open data practical to audit and build on. Figure 2 shows the workflow for twelve languages.
语料从 15 个 Nemotron-CC-v2 子集的文档起步，覆盖网页文本、问答材料、数学、代码与其他专门内容。我们用 NeMo Data Designer（NVIDIA 的开源合成数据生成框架）跑翻译、质检与定向修复。整个工作流保留每个来源标识符，开发者能把输出追溯回原始文档——这种清晰溯源正是开放数据可审计、可续建的原因之一。图 2 展示十二种语言的工作流。
Figure 2. Four-stage Indic synthetic-data pipeline for twelve languages, implemented using NeMo Data Designer and Gemma-4-31B-it. We translate English documents, assess translation quality, repair low-scoring outputs, and convert a subset of accepted translations into Latin script. The other ten languages use the separate Indic Translate workflow described below.
图 2：十二语言的四阶段合成数据管线（NeMo Data Designer + Gemma-4-31B-it 实现）：翻译英文文档→评估翻译质量→修复低分产出→把已接受翻译的子集转成拉丁字母。另外十种语言用下文的 Indic Translate 独立工作流。
### The twelve-language workflow · 十二语言工作流 For these twelve languages, we translate documents in full so the model can use context across paragraphs, tables, and equations. We selected Gemma-4-31B-it by comparing models on complete documents, checking translation quality and preservation of technical content. Our prompts distinguish prose to translate from content to preserve, including code, equations, URLs, and formatting.
这十二种语言采用整文档翻译，让模型能利用跨段落、表格与公式的上下文。我们靠在完整文档上对比各模型（查翻译质量与专业技术内容的保留）选定了 Gemma-4-31B-it。提示词区分「要翻译的散文」与「要保留的内容」——代码、公式、URL 与格式。
After translation, we use the same model to assess each output against its English source. A rubric inspired by Multidimensional Quality Metrics (MQM), an open framework for classifying translation errors by type and severity, covers meaning, fluency, terminology, style, and formatting. We retain only outputs scoring at least 75 out of 100 (a rubric score, not an accuracy percentage). For lower-scoring outputs, we use the assessment feedback to guide repairs, then reassess them against the same threshold. This process recovered 438.4 billion tokens across native-script and romanized data, already included in the corpus total. Sampled human reviews and tests with deliberately introduced errors helped identify omissions and residual English, informing improvements to our prompts and checks.
翻译之后，用同一个模型把每个输出对照英文源自评。评分表受多维质量指标（Multidimensional Quality Metrics，MQM——按类型与严重度分类翻译错误的开放框架）启发，覆盖语义、流畅度、术语、风格与格式。只保留 100 分制里 75 分以上的输出（评分表的分数，不是准确率百分比）；低分输出用评估反馈指导修复，再按同一阈值复评。这个过程在原生文字与罗马化数据中挽回了 438.4B token（已计入语料总量）。抽样人工审查与故意注入错误的测试帮助发现漏译与英文残留，反过来改进了提示词与检查。
Accepted native-script translations also provide the starting point for romanized data. We convert a subset into Latin script and check each result against its native-script input. This separates translation from script conversion. In our initial human comparison, this two-step approach produced fewer hallucinated additions and less residual English than generating romanized text directly from English.
已接受的原生文字翻译也是罗马化数据的起点：把一个子集转成拉丁字母，每个结果对照其原生文字输入检查——翻译与转写就此分离。初期人工对比显示，这两步流程比从英文直接生成罗马化文本，幻觉增补更少、英文残留更少。
### The ten-language workflow · 十语言工作流 For the other ten languages, we adapt the workflow to address long-document quality and script-stability issues observed in our comparisons. We translate with Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit center for AI in education incubated at IIT Madras. Gemma-4-31B-it then assesses each translation against its English source. For eight languages, we keep the whole-document approach. For longer Santali documents, we translate overlapping windows and reconstruct the document; for Manipuri, we translate into Bengali script before converting to Meitei Mayek, its native script. These workflows retain outputs scoring at least 75 out of 100 and produce native-script data only, without feedback-based rewriting.
另外十种语言的工作流针对对比中观察到的长文档质量与文字稳定性问题做了适配：翻译用 Indic Translate（Bodhan AI——IIT Madras 孵化的教育 AI 非营利中心——的英文↔22 印度语言文档级翻译模型），再由 Gemma-4-31B-it 对照英文源评估。八种语言保持整文档翻译；较长的桑塔利语文档用重叠窗口翻译再拼接重建；曼尼普尔语先译成孟加拉文字再转换为其原生文字 Meitei Mayek。这些工作流保留 75/100 以上的输出、只产原生文字数据，不做基于反馈的重写。
What changed after Indic continued pretraining? · 印度语言继续预训练改变了什么
The Nemotron 3.5 Super VL technical report compares the checkpoint immediately before Indic continued pretraining (CPT) with the checkpoint after that stage. The configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mixture of synthetic and natural Indic text. The remaining 90% maintained the preceding training mixture's relative proportions.
Nemotron 3.5 Super VL 技术报告对比了印度语言 CPT 阶段前后的两个 checkpoint。配置预算为 1000 亿训练 token，其中 10%（100 亿）分给合成+自然印度语言文本的混合，其余 90% 维持前一阶段训练配比的相对比例。
Scores increased on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K.
五项已报告的印度语言知识与推理基准全部上涨：MILU、Indic MMLU-Pro、Indic ARC-Challenge、Indic ARC-Easy、Indic GSM8K。
Figure 3. Nemotron 3.5 Super VL before and after the Indic CPT stage. Gains range from 4.98 to 9.82 percentage points. The comparison reports the change across a training stage that combines synthetic and natural Indic data, rather than isolating the synthetic corpus's contribution. Language coverage differs by benchmark; these are stage-level checkpoints, not a comparison of final model releases.
图 3：Nemotron 3.5 Super VL 印度语言 CPT 阶段前后对比，涨幅 4.98 到 9.82 个百分点。该对比报告的是「合成+自然印度语言数据联合训练阶段」的整体变化，没有隔离合成语料的独立贡献；各基准的语言覆盖不同，且这是阶段级 checkpoint 对比，不是最终模型发布的比较。
These gains support the combined synthetic-and-natural Indic CPT recipe, while leaving the choice of mixture for other models and languages to further study.
这些涨幅支持「合成+自然」联合的印度语言 CPT 配方；其他模型与语言该用什么配比，留待进一步研究。
The companion Indic report examines source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data for the twelve-language group. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step for those controlled studies.
配套的印度语言报告用十二语言组「首轮即接受」的数据考察了来源配比、合成数据池规模与文字覆盖；对完整语料（含修复重写与另外十种语言）的评估仍是这些受控研究的下一步。
### Broader Indic capabilities in Nemotron 3.5 Super VL · Nemotron 3.5 Super VL 更广的印度语言能力 Beyond the CPT-stage results, how does the released model compare with the previous generation? Figure 4 compares Nemotron 3.5 Super VL with Nemotron 3 Super across Indic task performance and language quality. These results reflect the released model's full training pipeline, including Indic CPT and post-training.
CPT 阶段结果之外，发布模型对比上一代如何？图 4 对比 Nemotron 3.5 Super VL 与 Nemotron 3 Super 的印度语言任务表现与语言质量。这些结果反映发布模型的完整训练管线（含印度语言 CPT 与后训练）。
Nemotron 3.5 Super VL scores higher on five of the six task benchmarks shown, spanning knowledge, reasoning, mathematics, code, and translation. Instruction following on IFEval 1.5 dipped 2.04 points. It also improves across all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages. Together, these evaluations examine both what the model can do and how well it communicates in Indian languages.
Nemotron 3.5 Super VL 在展示的六项任务基准中五项得分更高，覆盖知识、推理、数学、代码与翻译；IFEval 1.5 的指令遵循微降 2.04 分。它还在 IndiVibe（Sarvam AI 的印度语言助手式提示开放基准）全部四个维度上改善：有用性、流畅度、语言与文字保真、详略得当。这些评估合起来，既看模型能做什么，也看它用印度语言沟通得好不好。
Figure 4. Indic task performance and language quality across Nemotron Super generations. Left: six task benchmarks. Right: preliminary IndiVibe conditional preference scores, not direct head-to-head win rates. Labels show changes in percentage points, except WMT24++, which uses chrF points. See the companion Indic technical report for benchmark and evaluation details.
图 4：Nemotron Super 两代的印度语言任务表现与语言质量。左：六项任务基准；右：IndiVibe 初步条件偏好分（不是直接对战胜率）。标签为百分点变化，WMT24++ 除外（用 chrF 分）。基准与评估细节见配套印度语言技术报告。
Start building · 开始构建
### Load the corpus · 加载语料 Start with the languages and scripts your model needs, inspect a sample, and review the dataset card and license before training. Configurations use language, script, and region tags: for example, hi-Deva-IN denotes Hindi in Devanagari, while hi-Latn-IN denotes romanized Hindi.
从你的模型需要的语言与文字开始，抽样检查、训练前过一遍数据卡与许可。配置用「语言-文字-地区」标签：hi-Deva-IN 表示天城文的印地语，hi-Latn-IN 表示罗马化印地语。
The example below asks for the repository ID from the dataset card, then discovers the available splits and streams a sample. (from datasets import get_dataset_split_names, load_dataset; config = "hi-Deva-IN"; data = load_dataset(dataset_id, name=config, split=splits[0], streaming=True)). Hugging Face Datasets supports streaming so readers can inspect records without downloading the entire corpus. Once the release is available, pin a dataset revision for reproducible experiments.
示例先要数据卡里的仓库 ID，再发现可用 split 并流式抽样（datasets 库：get_dataset_split_names + load_dataset(name="hi-Deva-IN", split=splits[0], streaming=True)）。HF Datasets 支持流式读取，不下载整个语料就能检查记录。发布可用后，钉定数据集版本号以保证实验可复现。
For a first training study, compare mixtures at a fixed token budget and record the natural/synthetic and native/romanized shares. Track per-language results alongside English capability retention and language/script quality. When creating held-out splits, account for records derived from the same source across languages and scripts.
第一个训练研究建议：固定 token 预算下对比不同配比，记录自然/合成与原生/罗马化的份额；逐语言追踪结果，同时盯英文能力保持与语言/文字质量。切留出集时，注意跨语言与文字派生自同一来源的记录要归组处理。
To build a fuller recipe, pair the corpus with Nemotron post-training data, such as the Chat & Instruction or Post-Training Blends collections. For India-specific synthetic data, seed NeMo Data Designer with Nemotron-Personas-India, synthetic personas in English and Hindi grounded in real-world distributions for India.
要配更完整的配方，可把语料与 Nemotron 后训练数据组合（如 Chat & Instruction 或 Post-Training Blends 合集）。要印度特有的合成数据，用 Nemotron-Personas-India（基于印度真实分布构建的英文+印地语合成人设）给 NeMo Data Designer 做种子。
### Adapt the generation recipe · 适配生成配方 The open-source Nemotron Bharat translation plugin for NeMo Data Designer provides translation and evaluation prompts, optional feedback-based rewriting, document windowing, and script-remapping support. Use its configuration reference to choose models and languages, adapt prompts to your domain, and set rewrite parameters. The accompanying developer note explains the workflow and engineering choices.
NeMo Data Designer 的开源 Nemotron Bharat 翻译插件提供翻译与评估提示词、可选的反馈式重写、文档窗口与文字重映射支持。用配置参考选模型与语言、把提示词适配到你的领域、设重写参数；随附的开发者注记解释工作流与工程取舍。
Start with a small sample of your own source documents. Inspect translation quality and preservation of code, mathematics, and formatting, then refine the prompts and checks with native-language reviewers before scaling up. Treat generation and corpus acceptance as separate steps: the plugin returns candidates and scores, while your filtering policy determines which records enter the training corpus.
从你自己的源文档小样本开始：检查翻译质量与代码/数学/格式的保留，放大规模前与母语评审一起打磨提示词与检查。把「生成」与「入库」当两步：插件返回候选与分数，你的过滤策略决定哪些记录进入训练语料。
Scope and limitations · 范围与局限
Translation broadens access to the source material, but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture. We recommend combining this corpus with appropriately sourced natural Indic text and evaluating the resulting mixture for the intended use case.
翻译拓宽了来源材料的可及性，但不能替代以印度语言原生创作的文本，也不保证覆盖本地知识与文化。建议把这份语料与来源妥当的自然印度语言文本组合，并针对目标用例评估混合结果。
Quality assessment is primarily automated, complemented by sampled human review whose coverage varies across languages. For the twelve-language workflow (Figure 2), the translator also serves as the judge. Retained text can still contain errors, and the ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates; they are not semantic deduplication of similar content.
质量评估以自动化为主、抽样人工审查为辅（覆盖随语言而异）。十二语言工作流（图 2）里译者兼任评委；留存文本仍可能含错，十语言工作流需要更广的母语者验证。来源 ID 检查剔除的是处理性重复，不是相似内容的语义去重。
Alongside the corpus, AI4Bharat at IIT Madras, with support from NVIDIA, is building Indic-LLM-Eval-Suite, an open and evolving collection of benchmarks for Indic model capabilities. It helps model builders assess both task performance and language quality. See the companion blog, Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite.
与语料并行，IIT Madras 的 AI4Bharat 在 NVIDIA 支持下正在建设 Indic-LLM-Eval-Suite——印度语言模型能力的开放基准集合（持续演进），帮助模型构建者同时评估任务表现与语言质量。另见配套博客《更好的模型需要更好的基准》。
Explore the language configurations, test a training mixture, or adapt the open recipe to a new domain. Sharing per-language results and improvements to the generation workflow can help the community build on this work.
探索语言配置、测试训练配比，或把开放配方适配到新领域。分享逐语言结果与生成工作流的改进，能帮助社区在这项工作上继续建设。
Further reading and acknowledgments · 延伸阅读与致谢
The companion reports, Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs and the Nemotron 3.5 Super VL and Lightning technical report, provide the corpus accounting, generation methods, experiments, and evaluation details. Evaluation is the other half of this work. For the benchmark side, see the companion blog from AI4Bharat and NVIDIA, Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite.
配套报告《Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs》与《Nemotron 3.5 Super VL and Lightning 技术报告》提供语料账目、生成方法、实验与评估细节。评估是这项工作的另一半——基准侧见 AI4Bharat 与 NVIDIA 的配套博客《Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite》。
This work brings together NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI. We also acknowledge the model and tooling contributions used in the pipeline: Google's Gemma, Bodhan AI's Indic Translate, and NeMo Data Designer. Additional NVIDIA data resources are available in the Nemotron Pre-Training Datasets collection.
这项工作汇聚了 NVIDIA、IIT Madras、AI4Bharat 与 Bodhan AI。同时致谢管线中使用的模型与工具贡献：Google 的 Gemma、Bodhan AI 的 Indic Translate、NeMo Data Designer。更多 NVIDIA 数据资源见 Nemotron Pre-Training Datasets 合集。