Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov, Kirill Chernyshev, Kirill Malakhov, Ilia Vasiliev, Ilia Trushkin, Valeriya Kobenko, David Chikovani, Alexander Ivanov, Azat Saginbaev, Egor Silvestrov, Ivan Mikheev, Konstantin Zakharov
7 min
Abstract
Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized samples and lay foundation for later applications. This report presents series of KVAE tokenizers for audio, image and video, all designed for subsequent text-conditioned generation: KVAE-Audio, a continuous full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels; KVAE-3D -- two causal video tokenizers for 4x16x16 and 4x8x8 compression; KVAE-2D, an image model, compressing input by factor of 8 with 32 channels. We demonstrate that reconstruction (PSNR, LPIPS, PESQ, etc.) and generation results on objective (Frechet Distance, CLIP score, CLAP score, etc.) and subjective (side-by-side evaluation) metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. Considering difficulty of development, we share with community training details, model selection method and ablation on design choices. The code is publicly available at https://github.com/kandinskylab/kvae and https://github.com/kandinskylab/kvae-audio.
Sam: And I'm guessing KVAE is designed to produce smoother maps?
Alex: That's the goal. The framework moves away from rigid, discrete encoding—where data gets sorted into fixed categories—and instead uses a continuous, flowing space that aligns more naturally with how modern generative models work. They also fine-tune what's called the decoder, which is the component that translates the compressed map back into a usable image or video.
Sam: You mentioned the compressed data is sometimes called "latents." Can you unpack that word?
Alex: Sure. Imagine reading a long, complex novel. A latent is like a carefully written summary—it captures the main themes, the character arcs, the emotional tone—without reproducing every sentence. It's an abstract representation that holds the essence of the original without the bulk. KVAE is designed to produce latents that are not just accurate summaries, but summaries that are easy for a generative model to work with.
Sam: So the quality of the summary matters as much as the quality of the creative engine reading it.
Alex: Precisely. And to make that measurable, the researchers developed a specific metric—a way to put a number on diffusability. They call it the Correlation Decay Slope. The idea is to measure how information spreads across the latent map. If you look at one point on the map, how related is it to the points nearby? And how quickly does that relationship fade as you move further away?
Sam: So if the information fades too fast, the map is jagged. If it fades at just the right rate, it's smooth and navigable.
Alex: Exactly. A well-calibrated decay means the AI can make reliable predictions about what's nearby on the map, which makes the generation process more stable. The researchers tested this across fourteen different model configurations and found a strong correlation between this slope value and how humans actually rated the quality of the generated output.
Sam: That's a meaningful efficiency gain. Instead of training a full model for weeks just to find out if the tokenizer is any good, you can check this slope value early and know whether you're on the right track.
Alex: It functions as a diagnostic tool. And it also explains why reconstruction scores have been misleading—a tokenizer can reproduce an input beautifully while still producing a latent map that's difficult for a generative model to navigate. The two qualities are genuinely separate.
Sam: You mentioned the paper also made changes to the architecture itself—the internal structure of the model. What was the key change there?
Alex: This one comes down to how the model handles time, which matters a great deal for video generation. Older systems used a technique called GroupNorm—a way of standardizing the data that looked at the entire video clip at once, including frames that hadn't been generated yet.
Sam: It was looking at the future?
Alex: In a sense, yes. And that's a problem. When you're generating a video frame by frame, the model should only be able to see what came before—not what comes next. If it can peek ahead, it's essentially cheating, and the model won't work properly in real-world conditions where the future genuinely isn't available yet.
Sam: Like trying to predict the next word in a sentence while accidentally reading the end of the paragraph. It breaks the whole exercise.
Alex: Exactly. By switching to a spatial-only normalization approach—one that only looks at the current frame and what preceded it—the researchers ensured the model stays what they call causal. It builds each moment of a video based only on what it has already processed, which is how a real-time system needs to behave.
Sam: So the paper is really a set of careful, deliberate adjustments: a better way to organize compressed data, a better way to measure whether that organization is working, and a fix to how the model processes time. None of it is flashy, but each piece addresses a real gap.
Alex: That's a fair summary. What the researchers are arguing, at a fundamental level, is that the tokenizer deserves to be treated as a first-class component—not an afterthought. The quality of generation isn't just about having a more powerful creative engine. It's about giving that engine well-organized, navigable material to work with.
Sam: It's a bit like the difference between a great chef and a great kitchen. You can have all the talent in the world, but if the ingredients are poorly prepared and the workspace is chaotic, the meal suffers.
Alex: And KVAE is, in essence, an argument for taking the kitchen seriously. If this approach becomes more widely adopted, it suggests that meaningful improvements in AI-generated media may come not from scaling up the generative model itself, but from being more deliberate about how we compress and organize the data it learns from.
Sam: That's a useful reframing. Thanks for walking through it, Alex.
Alex: Thanks for listening to ResearchPod.