Glonce

KVAE tokenizers released for audio, image and video generation

researchThe Glonce editorsHugging Face🔥78Glonce rating78Impact60Novelty65Relevance100Surprise45Players40Credibility85
KVAE tokenizers released for audio, image and video generation

Latent diffusion models rely on a tokenizer to compress an input signal into a latent representation before generation happens, which makes the tokenizer's design a direct factor in learning speed, sample quality and what can be built on top of it later. A new technical report presents KVAE, a family of three such tokenizers, each aimed at subsequent text-conditioned generation in a different modality. KVAE-Audio is a continuous, full-band 48 kHz tokenizer with a 50 Hz latent made of 64 channels. KVAE-3D covers video with two causal tokenizers, one at 4x16x16 compression and one at 4x8x8 compression. KVAE-2D handles images, compressing the input by a factor of 8 into a 32-channel latent.

The authors report that reconstruction metrics (PSNR, LPIPS, PESQ, among others) and generation metrics, both objective (Frechet Distance, CLIP score, CLAP score, among others) and subjective (side-by-side human evaluation), match or surpass frontier open-source tokenizers, specifically the VAEs used in Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. The report does not give the actual numeric scores behind this comparison, only the metric names and the list of baseline tokenizers.

Citing the difficulty of building tokenizers like these, the authors say they are sharing training details, their model selection method and ablation studies on design choices with the community. Code for both parts of the family is public, split across two GitHub repositories: kandinskylab/kvae and kandinskylab/kvae-audio, both hosted under the GitHub account kandinskylab. The report names no individual authors or affiliated institution, and gives no publication date, venue or model size and training compute figures.

Key facts

  • KVAE is a family of three tokenizers, for audio, video and image, all built for later text-conditioned generation.
  • KVAE-Audio is a continuous, full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels.
  • KVAE-3D provides two causal video tokenizers, at 4x16x16 and 4x8x8 compression.
  • KVAE-2D compresses images by a factor of 8 into a 32-channel latent.
  • The authors say reconstruction and generation metrics match or surpass tokenizers from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio, and have released the code at two GitHub repositories under the account kandinskylab.

Why it matters

The tokenizer sits at the front of every latent diffusion pipeline, so its quality caps what the generation model built on top of it can do: how fast it trains, how clean its output is, and what applications become feasible later. Releasing one family that covers audio, video and image together, rather than a single modality, is a broader claim than most tokenizer papers make.

Who it affects

Teams building or fine-tuning text-conditioned generative models for audio, image or video, and anyone currently relying on the VAEs bundled with Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio or MMAudio, since those are the exact tokenizers KVAE is benchmarked against.

How to use it

Code is public at github.com/kandinskylab/kvae for the image and video tokenizers and github.com/kandinskylab/kvae-audio for the audio tokenizer. The report states no license terms, pricing or usage restrictions, and none should be assumed.

How solid is it

The authors evaluate against a specific, named set of frontier open-source baselines using both objective metrics (PSNR, LPIPS, PESQ for reconstruction; Frechet Distance, CLIP score, CLAP score for generation) and a subjective side-by-side comparison, and describe sharing training details and ablations on design choices given how hard these models are to build. The report itself does not print the numeric results of that comparison.

Risks and caveats

The claim of matching or surpassing frontier tokenizers is not backed by any score in the text, only by the metric names and baseline list, so it cannot be checked without running the released code. The report also gives no individual authors, no affiliated institution, no publication date or venue, and no model size or training compute figures, leaving attribution and scale unverified from the source alone.