AuK unifies speech generation and editing in one open-source model

A new technical report introduces AuK, an open-source foundational model that brings speech generation and speech editing together behind a single interface: natural-language instructions paired with audio context. Instead of separate systems for the two jobs, AuK is built to handle both from the same instructions-plus-audio input.

To support that range, the team built about 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision, spanning five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing.

AuK's architecture combines three pieces: a multimodal large language model for semantic conditioning, a VAE trained jointly on speech, general audio and music for acoustic conditioning, and a hybrid rectified-flow Transformer that generates output through dual-stream MMDiT blocks followed by unified single-stream DiT blocks.

Training starts with a generation-only warm-up phase, then moves into joint pre-training on generation and editing together. Two post-training steps follow: human-feedback preference optimization for open-ended editing, and reward-based reinforcement learning for speech generation.

To cut inference cost, the team distilled AuK into a second model, AuK-Flash, using consistency initialization and a task-routed version of Decoupled DMD. AuK-Flash performs 4-step inference without classifier-free guidance and reaches a 4.5x wall-clock speedup over the full AuK model under matched conditions.

In the reported experiments, AuK achieves leading performance on zero-shot and instruction-controlled speech generation and on general instruction-guided editing, while remaining only competitive, not leading, on signal-level restoration tasks. The abstract names the task categories behind these results but not the specific benchmarks, datasets or comparison models. The team says it is releasing both the source code and the model weights, though the abstract does not give a license, a repository location, a parameter count, or a date for when AuK was developed, submitted or released.

Key facts

AuK is an open-source foundational model that unifies speech generation and editing behind one interface of natural-language instructions plus audio context.
It was trained on about 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing.
Its architecture pairs a multimodal large language model for semantic conditioning and a VAE jointly trained on speech, general audio and music for acoustic conditioning with a hybrid rectified-flow Transformer using dual-stream MMDiT blocks followed by single-stream DiT blocks.
Training runs a generation-only warm-up, then joint generation-editing pre-training, followed by human-feedback preference optimization for editing and reward-based reinforcement learning for generation.
A distilled variant, AuK-Flash, performs 4-step inference without classifier-free guidance and reaches a 4.5x wall-clock speedup over the full model under matched conditions; the team is releasing the source code and model weights.

Why it matters

AuK's central move is unification. Rather than separate systems for generating speech and for editing it, one open-source model handles both through a common interface of natural-language instructions and audio context, spanning five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. Behind it is a large training effort, about 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision, and the team is releasing the source code and model weights rather than keeping the model closed.

Who it affects

The immediate audience is speech and audio researchers and developers: anyone building text-to-speech, voice editing, audio enhancement and separation, or instruction-driven audio tools gets an open, general-purpose base model instead of having to assemble separate single-task systems. The report frames the release, source code and model weights together, as being specifically for reproducibility and further research, so academic and open-source communities are the stated target rather than an end-user product.

How to use it

The team says it is releasing both the source code and the model weights, which points to hands-on access rather than an API-only or closed release. For lower-latency use, the distilled AuK-Flash variant completes inference in 4 steps without classifier-free guidance, reaching a 4.5x wall-clock speedup over the full AuK model under matched conditions, making the choice between the two mainly about latency against using the full model.

How solid is it

The claims here come from the technical report's own abstract, which states that AuK achieves leading performance on zero-shot and instruction-controlled speech generation and on general instruction-guided editing. It names the task categories behind that claim but not the specific benchmarks, datasets, or comparison models, so the comparisons can't be checked from what's given here. The abstract also does not name any authors or institutions, which limits how much can be independently verified before the full report is available.

Risks and caveats

The report's own numbers include one trade-off: AuK leads on generation and instruction-guided editing but is only competitive, not leading, on signal-level restoration tasks. Adoption is also constrained by what the abstract leaves out: no parameter count or model size, no license or repository location, and no date for when AuK was developed, submitted, or released, so anyone planning to build on it has to wait for those specifics rather than assume typical open-source terms.