Glonce

NVIDIA announces Nemotron-pretraining-Indic-v1, about 2.8T tokens in 22 Indian languages

researchThe Glonce editorsHugging Face🔥64Glonce rating64Impact55Novelty60Relevance95Surprise30Players70Credibility70
NVIDIA announces Nemotron-pretraining-Indic-v1, about 2.8T tokens in 22 Indian languages

NVIDIA, working with IIT Madras and its AI4Bharat lab, has announced Nemotron-pretraining-Indic-v1: approximately 2.8 trillion tokens of synthetic pretraining text across 22 Indian languages. The corpus is "releasing soon" alongside the open, GPU-accelerated data generation pipeline used to create it. The post gives no release date beyond that. The 22 languages are those listed in India's Constitution, the first languages of 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census.

The data is translated from English. The source documents come from 15 Nemotron-CC-v2 subsets covering web text, question-answer material, mathematics, code and other specialized content. The authors say this brings curated English material, including maths, code and Q&A, into Indian languages so that models can explain technical concepts, work through problems and answer questions in them. Every record traces back to a curated source document through a retained source identifier, and each output is checked against its input. The pipeline is built with NeMo Data Designer, NVIDIA's open-source synthetic-data framework, and it captures the full recipe: prompts, LLM workflows and quality checks. Developers can adapt it to their own source material. The authors stress that the synthetic data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.

The corpus holds 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. It spans 34 language-script configurations: native script for all 22 languages and romanized text for twelve. "Native-script" here describes the writing system only; the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. The twelve dual-script languages are Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu. The other ten, native script only, are Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali and Sindhi. The authors warn that document records include translations and script variants of shared sources, so they are not counts of unique English documents, and that corpus token counts should be read separately from a training run's token budget.

For the twelve-language group, documents are translated whole by Gemma-4-31B-it, chosen by comparing models on complete documents. The same model then scores each output against its English source with a rubric inspired by Multidimensional Quality Metrics (MQM), covering meaning, fluency, terminology, style and formatting. Only outputs scoring at least 75 out of 100 are kept (a rubric score, not an accuracy percentage). Lower-scoring outputs are repaired using the assessment feedback and reassessed against the same threshold. That repair step recovered 438.4 billion tokens across native-script and romanized data, already counted in the total. Romanized data is made in a second step: a subset of accepted native-script translations is converted to Latin script and checked against its native-script input. In the authors' initial human comparison, this produced fewer hallucinated additions and less residual English than generating romanized text directly from English.

The other ten languages use Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit center for AI in education incubated at IIT Madras. Gemma-4-31B-it assesses each translation. Eight languages keep the whole-document approach. Longer Santali documents are translated in overlapping windows and reconstructed. Manipuri is translated into Bengali script first, then converted to Meitei Mayek, its native script. These workflows use the same 75-out-of-100 cut-off and produce native-script data only, without feedback-based rewriting.

The post also reports what Indic continued pretraining (CPT) did for Nemotron 3.5 Super VL, citing that model's technical report. The configured CPT budget was 100 billion training tokens, 10% of which (10 billion) went to a mixture of synthetic and natural Indic text, drawn from a twelve-language subset of this corpus plus natural Indic text. The other 90% kept the preceding mixture's proportions. Scores rose on all five reported Indic knowledge and reasoning benchmarks (MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K), by 4.98 to 9.82 percentage points. The authors note that this comparison does not isolate the synthetic corpus's contribution.

Comparing the released Nemotron 3.5 Super VL with Nemotron 3 Super, which reflects the full training pipeline including Indic CPT and post-training, the newer model scores higher on five of the six task benchmarks shown, which span knowledge, reasoning, mathematics, code and translation. Instruction following on IFEval 1.5 dipped 2.04 points. The model also improves on all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages, and the scores shown are preliminary conditional preference scores, not head-to-head win rates. The authors plan to build on the corpus in future Nemotron models.

The post closes with a getting-started section: pick the languages and scripts you need, inspect a sample, and review the dataset card and license before training. Configurations carry language, script and region tags, so hi-Deva-IN is Hindi in Devanagari and hi-Latn-IN is romanized Hindi. A short Hugging Face Datasets example streams a sample without downloading the whole corpus, and the authors advise pinning a dataset revision once the release is available.

Key facts

  • Nemotron-pretraining-Indic-v1 has approximately 2.8 trillion tokens across 22 Indian languages: 2,601.0 billion native-script and 224.6 billion romanized (Gemma-4 tokenizer).
  • It is "releasing soon" with the open, GPU-accelerated generation pipeline (prompts, LLM workflows, quality checks) built on NeMo Data Designer; no release date is given.
  • Text is translated from English documents in 15 Nemotron-CC-v2 subsets; outputs must score at least 75 out of 100 on an MQM-inspired rubric, and repair recovered 438.4 billion tokens.
  • Nemotron 3.5 Super VL, after an Indic CPT stage with 10 billion Indic tokens, gained 4.98 to 9.82 percentage points on five Indic benchmarks, though the gain is not attributed to the synthetic data alone.
  • The released Nemotron 3.5 Super VL beats Nemotron 3 Super on five of six Indic task benchmarks, with IFEval 1.5 down 2.04 points.

Why it matters

AI4Bharat's head Mitesh Khapra, as quoted in the post, says Indian language technology trails English because "we do not have enough data for Indian languages." This release is pitched as an answer on two fronts: a large body of synthetic Indic text, and an open recipe for generating more. The authors describe it as a substantial expansion of the open synthetic Indic training data available to the community, building on AI4Bharat's earlier open corpus Sangraha. Publishing the prompts, workflows and quality checks, with source identifiers kept on every record, is meant to make the data practical to audit and extend.

Who it affects

Model builders training on Indian languages are the main audience. The corpus covers all 22 constitutionally listed languages, including low-resource ones such as Bodo, Dogri, Santali and Manipuri, and the pipeline is aimed at teams that want to run it on their own source documents. Speakers of those languages are the intended long-term beneficiaries, since the data targets models that can explain technical concepts, work through problems and answer questions in them. NVIDIA says it plans to use the corpus in future Nemotron models.

How to use it

The corpus is not yet released; the post says it is coming soon. Once it is, the authors suggest starting with the languages and scripts your model needs, inspecting a sample, and reading the dataset card and license before training. Configurations use language, script and region tags (hi-Deva-IN for Hindi in Devanagari, hi-Latn-IN for romanized Hindi). Hugging Face Datasets streaming lets you look at records without a full download, and the authors advise pinning a dataset revision for reproducible experiments. Teams can also adapt the prompts and quality checks or feed in their own source documents. The authors warn that native and romanized records can express the same underlying content, which matters when building a training mixture, and that corpus token counts depend on the tokenizer and are not a training budget.

How solid is it

This is a vendor announcement written by the builders, with authors from AI4Bharat and NVIDIA, and the corpus is not yet downloadable, so none of the data has been independently inspected. The benchmark figures are cited from the Nemotron 3.5 Super VL technical report and a companion Indic report. The authors themselves say the CPT comparison covers a stage that combines synthetic and natural Indic data and does not isolate the synthetic corpus's contribution, and that language coverage differs by benchmark. They say the gains support the combined recipe while leaving the best mixture for other models and languages to further study. The translation-quality scores come from an LLM judge (Gemma-4-31B-it) on a rubric, supported by sampled human reviews and tests with deliberately introduced errors; the romanization finding rests on an initial human comparison.

Risks and caveats

The data is machine-translated from English, so it carries English-origin content and may miss local knowledge and cultural context; the authors say naturally authored Indic text remains essential. Nothing in the post gives a release date beyond "soon", and the license and repository ID are left to the dataset card. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step; the companion report used first-pass accepted data for the twelve-language group. The ten-language workflow has no feedback-based rewriting. Instruction following on IFEval 1.5 dipped 2.04 points in the newer model, and the IndiVibe scores are preliminary. GSM8K excludes Odia because evaluation exceeded the serving context limit.

“we do not have enough data for Indian languages.”

— Mitesh Khapra, who heads AI4Bharat, as quoted in the NVIDIA post