Study finds multimodal pretraining recipe hits strong results on 5% of compute

Vision is increasingly treated as a critical input for foundation models, pushing the field toward natively unified multimodal pretraining, training language and vision together within a single model from the start. The paper argues that despite this shift, the design space and the underlying mechanics of how modalities interact during unified training have remained underexplored. In a new paper, Junlin Han and co-authors set out to close that gap with a systematic, empirical study of multimodal pretraining, running controlled experiments on both synthetic and large-scale real-world datasets. The result is four insights the authors frame as findings about the 'physics' of multimodal pretraining.

The first insight concerns knowledge flow: how language ability, visual understanding and visual generation transfer knowledge to and from one another once they share a single unified model. The authors disentangle the three capabilities and report that the transfer between them is not symmetric or uniform: distinct patterns of influence emerge, meaning some capabilities shape the others more than they are shaped in return. The text describes the asymmetry and its patterns without specifying which capability leads and which lags.

The second insight addresses when modalities help each other and when they compete for capacity. The authors find that the complexity of the training data is largely what decides the outcome: more complex data tends to make modalities synergistic. They also identify architecture choices that favor synergy, sharing attention and normalization layers across modalities while keeping the feed-forward layers modality-specific, and show that this pattern holds across different visual tokenizer designs, the components that convert images into a form the model can process alongside text.

The third insight is about timing. Unifying modalities from the very earliest stage of training and optimizing them jointly outperforms both aligning modalities later in training and training them one after another in sequence. Delaying that integration, the authors find, produces what they call a 'vision laziness' phenomenon: models that unify late tend to fall back on language priors, leaning on what they already know from text rather than learning fully from the visual input.

The fourth insight is practical. Building on the first three, the authors derive pretraining recipes that reach strong generative performance while using only 5% of the compute budget that would otherwise be required. They then check that all four findings hold up outside the smaller controlled settings by training multiple 13.5-billion-parameter Mixture-of-Experts (MoE) models on 2 trillion tokens, describing the results as validating the insights at scale. The authors frame the paper as a step toward a principled understanding of multimodal pretraining that can guide how future systems are designed and scaled; the text does not mention releasing any code, dataset or model alongside the paper.

Key facts

A systematic study of multimodal pretraining, from Junlin Han and co-authors, runs controlled experiments on synthetic and large-scale real-world datasets to isolate four insights about how modalities interact during unified training.
Knowledge flow between language, visual understanding and visual generation is asymmetric: the three capabilities transfer knowledge to each other in distinct, uneven patterns rather than boosting one another equally.
Whether modalities cooperate or compete is largely set by data complexity; sharing attention and normalization layers while keeping feed-forward layers modality-specific promotes synergy, and this holds across different visual tokenizer designs.
Unifying modalities from the earliest training stage beats late alignment or sequential training; delayed integration causes a 'vision laziness' effect in which models fall back on language priors instead of learning fully from vision.
The derived pretraining recipes reach strong generative performance using only 5% of the compute budget, and all four findings are validated at scale by training multiple 13.5-billion-parameter MoE models on 2 trillion tokens.

Why it matters

Vision is increasingly treated as a critical input for foundation models, and the field is shifting toward natively unified multimodal pretraining, training language and vision together within one model. The paper argues that despite this shift, the underlying mechanics, how the modalities actually interact once training is unified, have remained underexplored. The authors respond with a systematic, controlled empirical study that maps when combining language, visual understanding and visual generation in one model helps, when it hurts, and which architectural and scheduling choices tip the balance. That distinction matters for anyone deciding how to structure a unified multimodal training run rather than leaving the choice unexamined.

Who it affects

The findings speak most directly to teams building or scaling unified multimodal foundation models: engineers deciding how to mix training data, whether to share or separate attention, normalization and feed-forward layers across modalities, and whether to unify vision and language from the first training step or align them later. The recipe's efficiency, reaching strong generative performance using only 5% of the compute budget, is also relevant to anyone assessing the resource cost of that kind of training run, though the source states this only as a relative figure rather than an absolute cost in dollars or GPU-hours. The paper does not name a specific company, product or lab behind the work, so no particular commercial user or customer is identified in the text beyond the research and engineering community working on multimodal pretraining.

How to use it

The practical takeaways are architectural and procedural rather than a downloadable product. Unify modalities from the earliest stage of training rather than aligning them later or training them in sequence; share attention and normalization layers across modalities while keeping feed-forward layers modality-specific to encourage synergy; and account for how training-data complexity affects whether modalities help or compete. The authors say the resulting recipes reach strong generative performance using just 5% of the compute budget. The text does not mention releasing code, a dataset or a trained model alongside the paper, so applying the findings currently means following the paper's own methodology rather than using a published tool.

How solid is it

The authors base their four insights on controlled experiments run across both synthetic and large-scale real-world datasets rather than a single setting, and they explicitly test whether the architecture finding, shared attention and normalization layers combined with modality-specific feed-forward layers, generalizes across different visual tokenizer designs. They then check that the same four insights hold at scale by training multiple 13.5-billion-parameter MoE models on 2 trillion tokens, rather than resting the claims on the smaller controlled experiments alone. The source text does not, however, name the authors' institutions or compare the results to specific prior models, so some of the usual points of external verification are not available from the abstract itself.

Risks and caveats

The abstract states the recipe's efficiency only as a relative figure, using 5% of the compute budget, without giving an absolute cost in GPU-hours, dollars or wall-clock time, so it is hard to judge from the text alone how expensive or accessible the full training run actually was. It draws no comparison to specific prior multimodal models or baselines, so readers cannot tell from the source how these results stack up against existing systems. No code, dataset or trained model is mentioned as released alongside the paper, so outside groups cannot yet verify or reproduce the findings without repeating the methodology themselves. The text also names no authors' institutions, so the affiliations and funding behind the work are not stated. Finally, the knowledge-flow and 'vision laziness' findings are described at the level of the pattern, asymmetric influence between modalities, reliance on language priors, without the abstract spelling out the underlying mechanism.