SepGen generates video with one audio stem per sound source
A new paper presents SepGen, a model that tackles a limitation of today's joint audio-video generators. Those systems can synthesize a video and its soundtrack, but the soundtrack comes out as a single audio mix in which the individual sources cannot be reached separately.
The authors motivate the work with 4D audio-visual scenes: a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires every source to be available as an individual waveform, so it can be localized in the scene and propagated to the observer before the signals are mixed.
SepGen extends a pretrained audio-video generator so that one joint sampling run emits the video, the mixed soundtrack, and one waveform per captioned source. It supports two complementary modes. In generation mode, each source caption specifies what its stem contains; a two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio mix stays clean while the captions specify what to extract, so the model can decompose a recording from a free-text description.
Generation is evaluated on scenes synthesized from text. Separation is evaluated on scenes rendered by other generators and on real recordings of speech, music and sound effects. Given an audio mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints and datasets are available at the project page, sepgen.github.io.
Key facts
- SepGen extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run.
- Two modes: generation, where each source caption says what its stem contains, and separation, where captions say what to extract from a clean input mix.
- A two-speaker dialogue in generation mode comes out as one stem per speaker, in the original turn order.
- Evaluated on text-synthesized scenes (generation) and on scenes from other generators plus real speech, music and sound-effect recordings (separation); it outperforms language-conditioned separators, most clearly on speech.
- Code, checkpoints and datasets are released at sepgen.github.io.
Why it matters
Joint audio-video generators produce a soundtrack as one flat mix, which blocks any use that needs the sources apart. The paper's framing is 4D audio-visual scenes, where each sound source has its own position and trajectory and must be an individual waveform before it can be placed in the scene and propagated to a listener at a new viewpoint. SepGen aims to supply those per-source waveforms from the generator itself, in the same sampling run that produces the video and the mix. The same model also works as a text-conditioned separator for existing recordings.
Who it affects
The work is aimed at researchers building audio-visual generation and rendering systems, especially those who need separate sound sources for novel-viewpoint rendering. It is also relevant to people working on language-conditioned audio source separation, since the paper compares against separators of that kind on speech, music and sound effects.
How to use it
Code, checkpoints and datasets are available at https://sepgen.github.io/. In generation mode you write one caption per source, and each caption specifies what its stem contains. In separation mode you supply an audio mix plus captions describing what to extract, and the model decomposes the recording from that free-text description. No license for the code, checkpoints or datasets is stated.
How solid is it
The claims come from the paper's abstract. It reports evaluation on text-synthesized scenes for generation, and on scenes rendered by other generators and real recordings of speech, music and sound effects for separation, with SepGen ahead of language-conditioned separators, most clearly on speech. No quantitative results are given, so the size of the margins cannot be judged from the source, and the baseline separators and the pretrained generator being extended are not named.
Risks and caveats
The source gives no metric values, no margins over baselines and no named baselines, so the strength of the lead is unknown. The abstract says the lead is clearest on speech; it does not say the model is better on music or sound effects than on speech. No model size, training data volume, compute cost or inference speed is given, and the abstract names no authors or institutions.
“A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order.”
— SepGen paper abstract