Glonce

WaveGen generates speech waveforms from text in one model, no codec or aligner

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty60Relevance95Surprise25Players10Credibility75
WaveGen generates speech waveforms from text in one model, no codec or aligner

A new preprint introduces WaveGen, a text-to-speech framework that the authors call fully end-to-end. It is diffusion-based and built on what they name Spectral Forcing. A single model jointly learns an internal spectral trajectory and the target waveform trajectory, so it can go directly from text to a waveform.

The pitch is about what WaveGen leaves out. Unlike recent TTS pipelines, it does not rely on separately trained or externally decoded intermediate representations: no neural audio codecs, no Mel-spectrogram vocoders, no VAE-based latent models. It also aligns text and speech internally, so external alignment modules such as duration predictors are not needed. To keep the formulation truly end-to-end, the framework also avoids self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, and semantic distillation methods for training optimization.

To make this work, the authors design Spectral Forcing, a model architecture with disentangled diffusion heads, together with a training objective based on a multi-scale spectral v-loss. They also test scalability by scaling the model from 0.1B to 0.6B parameters.

On results, the authors say that with a single architecture, open-source data and one-stage training, and without external modules such as a duration predictor or REPA, WaveGen achieves state-of-the-art performance on the Emilia training dataset. In particular, they report a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are published on GitHub at sh-lee-prml/WaveGen.

Key facts

  • WaveGen is a diffusion-based text-to-waveform framework built on Spectral Forcing; one model jointly learns an internal spectral trajectory and the target waveform trajectory.
  • It uses no neural audio codec, Mel-spectrogram vocoder or VAE latent model, and no external alignment module such as a duration predictor.
  • It also avoids BERT-based text embeddings, Wav2Vec 2.0-based semantic representations and semantic distillation.
  • The design combines disentangled diffusion heads with a multi-scale spectral v-loss; the authors scaled the model from 0.1B to 0.6B parameters.
  • Reported on Seed-en: WER 1.47 and SIM 0.75. Code and checkpoints are on GitHub.

Why it matters

Most current TTS systems are pipelines: a codec, a vocoder or a latent model sits between the text model and the audio, and a separate component often handles alignment. WaveGen's claim is that one diffusion model can do all of it and output the waveform directly. If that holds up, it removes several separately trained parts from the speech stack. The authors also say the approach scales, testing sizes from 0.1B to 0.6B parameters.

Who it affects

Mainly speech and TTS researchers and engineers who build or study codec-based, vocoder-based or latent-based pipelines. Since code and checkpoints are public, they can try the model directly.

How to use it

Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen. The source gives no details on usage steps or hardware needs.

How solid is it

This is a preprint, and the claims come from the authors themselves. They report a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark, and say WaveGen reaches state-of-the-art performance on the Emilia training dataset. The text does not say which model size produced the Seed-en numbers, and it names no competing systems or their scores, so the state-of-the-art claim cannot be checked from the summary alone.

Risks and caveats

The text gives no audio-quality or MOS figures, no inference speed and no training cost, so practical trade-offs are unclear. It does not say which languages beyond English (Seed-en) were evaluated. The license of the code and checkpoints is not stated. Independent replication would be needed to confirm the end-to-end approach matches pipeline systems.