Intern Lumina U2 uses multi-codebook visual tokens in a diffusion language model
The paper starts from a known split in unified multimodal generation. Discrete approaches, which turn images into tokens from a single-codebook visual tokenizer, fit naturally with text modeling but have limited representational capacity. Continuous and hybrid approaches keep richer visual representations, but they model text and images in different spaces with different training objectives. The authors say this can create optimization conflicts between modalities whose representation spaces do not align. The question they pose is how to model high-capacity visual representations alongside text in one shared optimization space, inside a single backbone.
Their answer is Intern Lumina U2, which they describe as, to their knowledge, the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space. Each visual position is represented by a group of codes drawn from complementary codebooks. That raises the capacity of each position while keeping the representation compatible with discrete text modeling. At the input, the embeddings of the codes are concatenated at each position. At the output, a lightweight multi-codebook autoregressive head predicts the codes within a position one after another, while the diffusion backbone processes all positions in parallel.
The model is built on the unified 4D representation of AToken. It takes image, video and 3D inputs within a single backbone. The authors claim leading performance among unified models in visual understanding, image generation and image editing.
Two engineering points follow. The authors built a full-stack training, inference and evaluation system on both Ascend and NVIDIA platforms, to verify that the model works across the two. They also improved inference efficiency with a stability-aware sampling strategy that combines confidence-plateau-guided unmasking with drift-triggered cache refreshing. According to the paper, this accelerates image generation by 4.45x.
Key facts
- Intern Lumina U2 is a diffusion language model that represents each visual position with a group of codes from complementary codebooks, in a fully discrete space.
- The authors say it is, to their knowledge, the first diffusion language model to do this with multi-codebook visual tokens.
- At the output, a lightweight autoregressive head predicts the codes within each position sequentially, while the diffusion backbone handles all positions in parallel.
- Built on AToken's unified 4D representation, it supports image, video and 3D inputs in one backbone.
- A stability-aware sampling strategy (confidence-plateau-guided unmasking plus drift-triggered cache refreshing) accelerates image generation by 4.45x; the system was also built on both Ascend and NVIDIA platforms.
Why it matters
Unified models that handle both understanding and generation must decide how to represent images next to text. The paper frames the trade-off plainly: single-codebook discrete tokens match text modeling but carry limited visual detail, while continuous and hybrid designs keep more detail at the cost of separate spaces and objectives for text and images. Intern Lumina U2 tries to keep the discrete, text-compatible setup and gain capacity by giving each position several codes instead of one. The authors say no earlier diffusion language model has done this in a fully discrete space.
Who it affects
Mainly researchers working on unified multimodal models, visual tokenizers and diffusion language models. The support for image, video and 3D inputs in one backbone, and the system built on both Ascend and NVIDIA platforms, are relevant to teams that need to train or run such models on either hardware.
How to use it
This is a research paper, and the abstract mentions no release of code, weights or a demo. For now it is something to read and compare against, not something to run. The design idea to take from it: concatenate the embeddings of several codes at each position on input, and predict them sequentially with a small head on output while the main backbone works in parallel.
How solid is it
The account rests on the paper's abstract, so every performance claim is the authors' own. They say the model achieves leading performance among unified models in visual understanding, image generation and image editing, but the abstract names no benchmarks, scores or comparison models. The 4.45x speedup is likewise reported without a stated baseline, hardware or image resolution. The abstract also names no authors or institutions.
Risks and caveats
Treat the headline numbers as unverified until the full results are read. A 4.45x speedup is a relative figure, and it is unclear what it is measured against. "Leading among unified models" depends on which models and tests are included. The abstract does not discuss limitations, model size or the number of codebooks, so the cost of the multi-codebook design cannot be judged from it.
“to our knowledge the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space”
— Intern Lumina U2 paper abstract