Yeonkyeong Lee, Hyunsung Go, Jongmin Kim, Sewoong Lim, Donghoon Lee
5 min
Abstract
Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.
Alex: Both, and that's what makes the result worth paying attention to. The compression ratio at higher resolutions confirms that the adaptive approach scales far better than fixed-grid methods — the token budget grows with scene complexity rather than frame count. But the more interesting finding is that aggressive pruning of uninformative tokens enhances generative fidelity. By forcing the model to focus its capacity on motion-rich regions, you're not just saving compute — you're improving what the model attends to. The efficiency and quality gains are mechanistically linked, not independent.
Sam: That's a meaningful distinction. It's not a quality-efficiency tradeoff — it's a case where the constraint itself acts as an inductive bias.
Alex: Right. The sparsity pressure is doing regularization work. The model can't afford to spread capacity uniformly, so it learns to concentrate it where the signal is.
Sam: Where would a careful referee push back?
Alex: A few places. First, the cascaded architecture introduces a dependency on the mask-prior model. If that prior fails — say, in scenes with complex, unpredictable motion — the main model receives poor structural conditioning, and generation quality degrades in exactly the regions that matter most. Second, the paper focuses heavily on transformer-based backbones, which leaves open how this approach transfers to other architectures. Third, and the authors acknowledge this, the method shows blurring artifacts when high-frequency texture coincides with large motion — the gate struggles to decide whether those tokens are informative, and reconstruction suffers in those specific regions.
Sam: So the failure mode is precisely the case where the prior's assumptions break down — high motion, high texture, unpredictable structure.
Alex: Which is also, not coincidentally, the hardest case for video generation generally. The robustness evidence across datasets like Kinetics-600 is encouraging, but those are the conditions where you'd want to see more ablation work before drawing strong conclusions about generalizability.
Sam: The broader implication seems architectural. If this holds, future video models might move toward dynamic token budgets as a default — expanding compute allocation based on scene complexity rather than fixing it at the frame level.
Alex: That's the direction the paper points toward. Static grids are a convenient approximation, but they're not a principled one. KATok is a concrete demonstration that treating token selection as a learned, content-adaptive decision is both tractable and beneficial. The open question is how far that principle extends — to longer sequences, to more diverse motion patterns, to architectures beyond the transformer setting they tested.
Sam: A well-scoped result with a clear mechanism and honest limitations.
Alex: Thanks for listening to ResearchPod.