K2-Horizon releases six fully open language models, up to 375B MoE

K2-Horizon is a fleet of six fully open language models. The range runs from a 0.9B dense model to a 375B-A23B mixture-of-experts (MoE) model. The authors say the models are built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation and beyond. They claim the models lead or are competitive with strong open models of their size on agentic and coding benchmarks.
The paper also introduces three technical pieces. The first is Mixture-of-Value Attention (MoVA), which applies MoE to the value projections in attention and is introduced to strengthen reasoning. The second is softplus attention gating, described as a more flexible, theoretically motivated gate. The authors report that their 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. The third is Uno, a set of lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model.
The release follows the LLM360 definition of open source, aimed at transparency and reproducibility. It covers the full development history: intermediate checkpoints from every training stage, the pre-training data, training logs, and all training, evaluation and infrastructure code. The code includes the production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack.
The authors put particular weight on the data. Beyond the data itself, they release and document in detail every pipeline that built it, which they call the part of model development that open releases most rarely disclose. Those pipelines produced roughly 10T synthetic pre-training tokens, which the authors say shows synthetic data can be a major ingredient of pre-training at scale. Four of the six models share one 22T-token pre-training corpus and recipe, which the authors say reveals consistent scaling behavior and allows controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/.
Key facts
- Six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, aimed at reasoning and agentic work including coding, deep research and office automation.
- New methods: Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno diffusion adapters that speed up decoding by up to 2.2x without a separate draft model.
- The 36B-A4B model, using MoVA and softplus gating, nearly matches a 32B dense model trained on the same data with one-seventh the active parameters.
- The release follows the LLM360 open-source definition: checkpoints from every training stage, pre-training data, training logs, and all training, evaluation and infrastructure code, including xLLM and RL360.
- Data pipelines are documented and released; they produced roughly 10T synthetic pre-training tokens, and four models share one 22T-token corpus and recipe.
Why it matters
Few open releases show how a model was actually made. K2-Horizon ships checkpoints from every training stage, the pre-training data, logs and the code, and the authors single out the data pipelines as the part open releases most rarely disclose. Four models sharing one 22T-token corpus and recipe also give a controlled setup for studying scale and sparsity. On the technical side, the 36B-A4B result is the headline: it nearly matches a 32B dense model on the same data with one-seventh the active parameters.
Who it affects
Researchers studying scaling, sparsity, attention design or synthetic data can work from the released checkpoints, data and logs. Teams building their own training stacks get xLLM for dense and MoE training and the RL360 reinforcement-learning stack. Developers who want open models for coding, agentic tasks, deep research or office automation get a range of sizes, from 0.9B dense up to a 375B-A23B MoE.
How to use it
All artifacts are listed at https://ifm.ai/k2/. The release covers model checkpoints, pre-training data, training logs, and the training, evaluation and infrastructure code. Uno adapters are meant for faster decoding: up to 2.2x, lossless, with no separate draft model. Teams can also reuse the documented data pipelines, which produced roughly 10T synthetic pre-training tokens.
How solid is it
The text is the paper's abstract, so every claim is the authors' own. The performance claim, that the models lead or are competitive with strong open models of their size on agentic and coding benchmarks, comes without detail here: no benchmark names, scores or named competitor models are given. The 36B-A4B result is worded as nearly matching, not equalling, the 32B dense model. The 2.2x figure is a maximum (up to). The ten-trillion-token figure is approximate (roughly 10T) and is the output of the pipelines, not a share of the 22T-token corpus.
Risks and caveats
The sizes of the four models other than the 0.9B dense, 36B-A4B and 375B-A23B are not stated, and neither is which four share the 22T-token corpus. No licence is named. No hardware, training cost or compute budget is given. No author names or institutions are named in the text. The "fully open" label rests on the LLM360 definition the authors say they follow, and the benchmark claims need independent checking.
“we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose”
— K2-Horizon paper abstract