Eren Çetin, Lucas Relic, Yuanyi Xue, Markus Gross, Christopher Schroers, Roberto Azevedo
6 min
Abstract
We present a perceptually-driven video compression framework integrating implicit neural representations (INRs) and pre-trained video diffusion models to address the extremely low bitrate regime (<0.05 bpp). Our approach exploits the complementary strengths of INRs, which provide a compact video representation, and diffusion models, which offer rich generative priors learned from large-scale datasets. The INR-based conditioning replaces traditional intra-coded keyframes with bit-efficient neural representations trained to estimate latent features and guide the diffusion process. Our joint optimization of INR weights and parameter-efficient adapters for diffusion models allows the model to learn reliable conditioning signals while encoding video-specific information with minimal parameter overhead. Our experiments on UVG, MCL-JCV, and JVET Class-B benchmarks demonstrate substantial improvements in perceptual metrics (LPIPS, DISTS, and FID) at extremely low bitrates, including improvements on BD-LPIPS up to 0.214 and BD-FID up to 91.14 relative to HEVC, while also outperforming VVC and previous strong state-of-the-art neural and INR-only video codecs. Moreover, our analysis shows that INR-conditioned diffusion-based video compression first composes the scene layout and object identities before refining textural accuracy, exposing the semantic-to-visual hierarchy that enables perceptually faithful compression at extremely low bitrates.
Alex: Wait, how do they tweak such a huge pre-trained model with so few changes? Like, what's the trick there?
Sam: Large diffusion models have billions of internal settings. Normally, customizing one for a specific video means retraining almost everything—like rewriting a whole cookbook to add one recipe. Instead, they add lightweight adapters: fixed random patterns that stay the same, plus a handful of simple numbers that adjust how much each pattern influences key parts of the model.
Alex: Okay, so these adapters sit inside the diffusion transformer—the main engine doing the denoising?
Sam: Exactly. The diffusion transformer step-by-step cleans up noisy video latents. They place adapters in 30 of its blocks, targeting attention layers where it focuses on important parts and feed-forward layers that process features. This adapts the model to the video's style—like soccer motion or crowd scenes—without bloating the file size.
Alex: And everything happens in this latent space? With predicted signals and masks?
Sam: Right—the latent space is a squeezed-down version of the video, like compressing a big photo album into a few key sketches. The INR outputs two things there: predicted latent features, which are like rough sketches of what each frame should look like, and adaptive masks that score confidence in those sketches—highlighting reliable areas like steady backgrounds versus fuzzy fast action. The masks weigh how much to trust the INR versus letting the diffusion fill in from its general knowledge.
Alex: Does that mean they judge success by how it looks to humans, not just perfect pixel matches?
Sam: Precisely—a perceptually-driven setup. They optimize and measure with tools that mimic human vision. The paper shows a clear edge over codecs like HEVC at under 0.05 bits per pixel, with about 91 times better FID scores—meaning generated videos match real ones far more naturally, avoiding blur in motion where others fail.
Alex: Huh... so the masks and adapters together make it reliable even when data's scarce.
Alex: But if it's perceptually better, how exactly do they measure that?
Sam: They focus on tools that check how close the video looks to the original in ways that match human eyes, rather than exact pixel-by-pixel copies. These are called LPIPS and DISTS, and a third, FID, measures if the generated clips statistically match real video distributions.
Alex: So it's trading perfect reconstruction for something that feels right to watch.
Sam: Exactly—the paper notes this deliberate trade-off. Traditional measures like PSNR reward exact matches but ignore how we see blur or artifacts; here, perceptual scores show a clear edge over codecs like HEVC and VVC, and even neural methods, especially on datasets with fast motion like sports clips.
Alex: But with all that training, what's the catch—like time or hardware needs?
Sam: Training takes about 15 hours per short clip on a high-end GPU, spread over three stages. Inference runs at around one frame per second, fitting consumer hardware. Bitrate splits with most going to the INR, a small slice to adapters.
Alex: Huh... so for archive or satellite use, where speed isn't real-time critical, this holds up well.
Sam: The evidence points to yes—strong perceptual gains on benchmarks like UVG and JVET with complex motion, validating the hybrid for extreme low rates where codecs fail, though pixel metrics lag as expected for realism-focused design.
Alex: So overall, this setup prioritizes how videos look and feel to people, even if it doesn't match every single pixel perfectly. Well put. It's a solid step for handling video where data is the real squeeze. Thanks for breaking it down, Sam—this has been a clear look at pushing compression limits thoughtfully. Thanks for listening to ResearchPod.