We present a perceptually-driven video compression framework integrating implicit neural representations (INRs) and pre-trained video diffusion models to address the extremely low bitrate regime (<0.05 bpp). Our approach exploits the complementary strengths of INRs, which provide a compact video representation, and diffusion models, which offer rich generative priors learned from large-scale datasets. The INR-based conditioning replaces traditional intra-coded keyframes with bit-efficient neural representations trained to estimate latent features and guide the diffusion process. Our joint optimization of INR weights and parameter-efficient adapters for diffusion models allows the model to learn reliable conditioning signals while encoding video-specific information with minimal parameter overhead. Our experiments on UVG, MCL-JCV, and JVET Class-B benchmarks demonstrate substantial improvements in perceptual metrics (LPIPS, DISTS, and FID) at extremely low bitrates, including improvements on BD-LPIPS up to 0.214 and BD-FID up to 91.14 relative to HEVC, while also outperforming VVC and previous strong state-of-the-art neural and INR-only video codecs. Moreover, our analysis shows that INR-conditioned diffusion-based video compression first composes the scene layout and object identities before refining textural accuracy, exposing the semantic-to-visual hierarchy that enables perceptually faithful compression at extremely low bitrates.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a challenge in video streaming—how to send high-quality footage over very limited bandwidth without it turning into a blurry mess.
Sam: This episode centers on a research paper called DiV-INR, which tackles extreme low-bitrate video compression. The core idea is using a special compact map of the video to guide a smart generator that fills in details, allowing clear video at under 0.05 bits per pixel—about one fiftieth the data of normal methods.
Alex: So this paper is basically asking how to transmit something like 4K sports footage over a satellite link, where bandwidth is tiny, without losing sharp details in fast action?
Sam: Yes, exactly. Traditional video compressors squeeze data by cutting out less important bits. But below about 0.05 bits per pixel, they start blurring motion or blocking up the image, because they can't hold onto fine details like player faces or ball paths in sports clips.
Alex: Right, so the problem isn't just making files smaller—it's keeping the video looking real to our eyes when the data is so limited.
Sam: Precisely. Older codecs rely on sending key snapshots of frames and guessing the rest. But at these rates, the guesses fail, creating artifacts that make fast scenes—like a soccer kick—unwatchable.
Alex: And this paper's approach avoids those snapshots altogether?
Sam: It does, by pairing two tools: one builds a lightweight, whole-video blueprint from coordinates in space and time, and the other uses learned patterns from millions of videos to recreate details faithfully. This combo encodes the video's global structure efficiently to steer the generation.
Alex: Huh. So instead of shipping blurry keys, they're sending a smart guide that tells the receiver how to rebuild the whole thing.
Sam: That's the key insight. The paper shows this preserves perceptual quality—how it looks to humans—far better, outperforming standards by up to 91 times on one measure at the lowest rates.
Alex: Okay, so the bottleneck is that low data can't capture motion well, but this blueprint-plus-generator fixes it without needing much bandwidth.
Sam: Correct. And since it's optimized together, the guide learns exactly what the generator needs, making it practical even on regular hardware for real-world low-bandwidth uses like remote broadcasts.
Alex: So they're optimizing this blueprint and the generator together—which means the guide gets tuned specifically for what the generator can handle best?
Sam: Yes. They train both parts at the same time: the INR that makes the conditioning signals, and small tweaks to the diffusion model itself. This joint optimization ensures the signals match exactly what the generator expects, without wasting bits on mismatched info.
Alex: Wait, how do they tweak such a huge pre-trained model with so few changes? Like, what's the trick there?
Sam: Large diffusion models have billions of internal settings. Normally, customizing one for a specific video means retraining almost everything—like rewriting a whole cookbook to add one recipe. Instead, they add lightweight adapters: fixed random patterns that stay the same, plus a handful of simple numbers that adjust how much each pattern influences key parts of the model.
Alex: Okay, so these adapters sit inside the diffusion transformer—the main engine doing the denoising?
Sam: Exactly. The diffusion transformer step-by-step cleans up noisy video latents. They place adapters in 30 of its blocks, targeting attention layers where it focuses on important parts and feed-forward layers that process features. This adapts the model to the video's style—like soccer motion or crowd scenes—without bloating the file size.
Alex: And everything happens in this latent space? With predicted signals and masks?
Sam: Right—the latent space is a squeezed-down version of the video, like compressing a big photo album into a few key sketches. The INR outputs two things there: predicted latent features, which are like rough sketches of what each frame should look like, and adaptive masks that score confidence in those sketches—highlighting reliable areas like steady backgrounds versus fuzzy fast action. The masks weigh how much to trust the INR versus letting the diffusion fill in from its general knowledge.
Alex: Does that mean they judge success by how it looks to humans, not just perfect pixel matches?
Sam: Precisely—a perceptually-driven setup. They optimize and measure with tools that mimic human vision. The paper shows a clear edge over codecs like HEVC at under 0.05 bits per pixel, with about 91 times better FID scores—meaning generated videos match real ones far more naturally, avoiding blur in motion where others fail.
Alex: Huh... so the masks and adapters together make it reliable even when data's scarce.
Alex: But if it's perceptually better, how exactly do they measure that?
Sam: They focus on tools that check how close the video looks to the original in ways that match human eyes, rather than exact pixel-by-pixel copies. These are called LPIPS and DISTS, and a third, FID, measures if the generated clips statistically match real video distributions.
Alex: So it's trading perfect reconstruction for something that feels right to watch.
Sam: Exactly—the paper notes this deliberate trade-off. Traditional measures like PSNR reward exact matches but ignore how we see blur or artifacts; here, perceptual scores show a clear edge over codecs like HEVC and VVC, and even neural methods, especially on datasets with fast motion like sports clips.
Alex: But with all that training, what's the catch—like time or hardware needs?
Sam: Training takes about 15 hours per short clip on a high-end GPU, spread over three stages. Inference runs at around one frame per second, fitting consumer hardware. Bitrate splits with most going to the INR, a small slice to adapters.
Alex: Huh... so for archive or satellite use, where speed isn't real-time critical, this holds up well.
Sam: The evidence points to yes—strong perceptual gains on benchmarks like UVG and JVET with complex motion, validating the hybrid for extreme low rates where codecs fail, though pixel metrics lag as expected for realism-focused design.
Alex: So overall, this setup prioritizes how videos look and feel to people, even if it doesn't match every single pixel perfectly. Well put. It's a solid step for handling video where data is the real squeeze. Thanks for breaking it down, Sam—this has been a clear look at pushing compression limits thoughtfully. Thanks for listening to ResearchPod.