UniSpark-9B unified multimodal model mixes full and linear attention

A paper on Hugging Face presents UniSpark-9B, a unified multimodal model that handles both visual understanding and visual generation. Its central question is hybrid attention: the model combines full attention, used for explicit token interactions, with GatedDeltaNet linear attention, used for compact context propagation.
The backbone is a Mixture-of-Transformer-Experts with approximately 9B parameters in each branch. Two training objectives run together: autoregressive text prediction and visual flow matching. Between them they support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction.
Interleaved generation is where the hybrid design needs special handling. Noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction in two ways: attention masks in the full-attention layers, and recurrent-state retention rules in the linear-attention layers.
The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum trains understanding and generation jointly. It refines answer accuracy and instruction following, and it extends image synthesis to 2K resolution.
On benchmarks, UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. The authors also point to qualitative examples of interleaved image generation, video synthesis, and video editing. They conclude that these results demonstrate hybrid attention to be an effective backbone for unified multimodal understanding and generation.
Key facts
- UniSpark-9B is a unified multimodal model for visual understanding and generation, built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch.
- It combines full attention (explicit token interactions) with GatedDeltaNet linear attention (compact context propagation).
- Training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences, used in a four-stage curriculum that extends image synthesis to 2K.
- Reported scores are 2470.3 on MME-S and 0.83 on GenEval.
- For interleaved generation, a blockwise protocol uses attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers to keep noisy visual targets apart from later predictions.
Why it matters
Unified models try to do image understanding and image generation in one network. UniSpark explores a specific design choice for that: mixing full attention with GatedDeltaNet linear attention instead of relying on one attention type. The authors frame the work as an investigation of hybrid attention and argue that their results show it is an effective backbone for unified multimodal understanding and generation.
Who it affects
The paper is most relevant to researchers building models that both answer questions about images and generate or edit them. The listed tasks are visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, with qualitative examples for video synthesis and video editing.
How to use it
No release, code, weights, license or availability is mentioned. For now this is a description of an approach and its reported scores. The practical takeaway is the design: full attention for explicit token interactions, linear attention for compact context propagation, and a blockwise protocol that handles masks and recurrent state in interleaved generation.
How solid is it
The claims are the authors' own, drawn from the paper's abstract. The two headline numbers are 2470.3 on MME-S and 0.83 on GenEval. No comparison with other models or baselines is given; the MME-S and GenEval scores come without any reference scores. Video synthesis and video editing are shown only through qualitative examples.
Risks and caveats
Without reference scores, the benchmark numbers cannot be placed against other models. No scale for the GenEval score is stated beyond the number 0.83, and no sub-scores for MME-S are given. No details of the four stages of the curriculum are given, and no total parameter count is stated; only approximately 9B parameters in each branch. No training compute, hardware or training duration is given. No authors or institutions are named.
“these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation”
— UniSpark paper abstract