DreamX-Creator 1.0 pairs a 7B model with 2K audio-video generation
The authors have presented DreamX-Creator 1.0, a compact system that generates video and audio jointly rather than producing them in separate stages, as most video generators do today. The system is built around a 7B parameter generator: given a first frame and a text prompt, it denoises audio and video streams together. For the first half of the network the two streams are processed independently; only in the second half are they coupled, through a mechanism the authors call Gated Cross-Modal Attention, which uses per-token and per-head output gates to control how much each cross-modal attention head contributes. Training runs through what the authors call Progressive Joint Training: two audio-video pre-training stages followed by a high-quality finetuning stage, drawing on a purpose-built Audio-Video Data System that filters clips for temporal coherence, generates structured multimodal annotations, and sorts the results into capability-oriented data pools. After pre-training, an Audio-Video Reinforcement Learning stage further refines the generator using what the authors term Modality-Aware Multimodal Feedback, which routes separate video, audio and cross-modal feedback signals to their respective streams rather than mixing them into one reward. For high-resolution output, the team built an Autoregressive 1-Step 2K Refinement pipeline: a bidirectional, multi-step teacher model is first converted into an autoregressive multi-step refiner, then distilled into a student model that needs only one denoising evaluation per temporal chunk to produce 2K output. The authors state that DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems, though the abstract gives no benchmark numbers or named comparison systems to back that claim. They are releasing both the 7B generator and the 2K refiner, with the stated goal of making native audio-video generation more accessible and giving other researchers a foundation to build on.
Key facts
- DreamX-Creator 1.0 is a native audio-video generator built around a 7B parameter model, conditioned on a first frame and a text prompt.
- Video and audio streams are processed independently for the first half of the network, then coupled via Gated Cross-Modal Attention for the rest.
- Progressive Joint Training runs two audio-video pre-training stages plus a high-quality finetuning stage, followed by reinforcement learning with modality-specific feedback.
- An Autoregressive 1-Step 2K Refinement pipeline distills a multi-step teacher into a student needing one denoising evaluation per temporal chunk to hit 2K resolution.
- The authors are releasing the 7B generator and the 2K refiner openly, describing the move as an effort to democratize native audio-video generation.
Why it matters
Most video generation systems today either skip audio entirely or bolt it on afterward in a separate pass, which limits how well the visual motion and the sound can inform each other. DreamX-Creator's architecture denoises both modalities together from the start, with cross-modal attention only kicking in partway through the network, which the authors position as a more genuine form of joint audio-video modeling rather than post-hoc synchronization.
Who it affects
The release targets researchers and developers working on multimodal generative models. By publishing a 7B generator, which is small enough to be broadly accessible compared with larger proprietary systems, alongside the 2K refinement pipeline, the authors are aiming this at people who want to build on or study native audio-video generation rather than at end users of a consumer product.
How to use it
The abstract states that the 7B generator and the 2K Refiner are being released, but gives no license terms, hosting location, or availability timeline for the weights or code.
How solid is it
The claim of being competitive with state-of-the-art open-source systems comes directly from the authors and is qualitative: the abstract names no specific benchmarks, metrics, or competing systems to substantiate it, so the comparison cannot be independently checked from this text alone.
Risks and caveats
Beyond the unquantified performance claim, the abstract does not specify the training data volume or compute budget behind the Audio-Video Data System, nor does it define what counts as a 'temporal chunk' in the refinement pipeline or give any inference speed or hardware requirements, so practical cost and deployment questions remain open.
“By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.”
— DreamX-Creator 1.0 paper