Junbo Wang, Liangyu Fu, Yuke Li, Yining Zhu, Ya Jing, Xuecheng Wu, Jiangbin Zheng
10 min
Abstract
Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on Diffusion model for Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.
Alex: So those test scores suggest it's closing the quality gap meaningfully... But how does the noise-cleaning part actually use the video to sharpen the text without getting overwhelmed by it?
Sam: Let's break it down step by step. First, they run the video through a visual encoder—a tool that watches the frames and pulls out key features like shapes, motions, and objects into a compact set of visual tokens, similar to summarizing a movie scene with bullet points. The real caption text gets turned into a matching set of text tokens by a text encoder, creating a clean starting point.
Alex: Got it—so both video and text become these token lists that can talk to each other.
Sam: Exactly. During training, they sprinkle Gaussian noise—random fuzz like static on an old TV screen—onto the clean text tokens over many steps, making them blurrier each time until they're pure mess. The denoiser then reverses this: it takes the fuzzy text as the main focus, or "query," and uses the clean visual tokens strictly as reference points, or "keys" and "values," in a cross-attention setup. This keeps the video's guidance separate from the text's own cleanup, so the visuals don't get drowned out—like using a blueprint to chisel a statue from noisy clay without the clay smearing the blueprint.
Alex: Wait—cross-attention? You're saying the text asks questions of the video features without mixing everything in one big conversation?
Sam: Yes, precisely. In older setups, everything shared the same attention space, diluting the video's role as text features competed for focus. Here, text queries pull exactly what it needs—like a verb focusing on action clips or a noun on objects—producing a sharpened text representation. That feeds into the language model, which refines it through stacked self-attention layers, then predicts all words in parallel via simple projections.
Alex: Oh, so that separation is why it matches the slow methods' quality... Without the visuals fading into the background.
Sam: The paper's tests bear that out—it hits state-of-the-art scores on datasets like MSVD and VATEX, comparable to autoregressive models on MSR-VTT, all while generating two to three times faster for longer captions.
Alex: Makes sense why speed alone wasn't enough before... It's the precision in how they connect the senses.
Alex: So that separation keeps the video's input clear and strong throughout... But how do they actually train this to make sure the cleaning works reliably?
Sam: During training, they use two main guides to shape the model. One checks how closely the cleaned-up text matches the original clean version, like measuring the distance between a rough sketch and the final drawing with a ruler—the smaller the gap, the better. They call this mean squared error, or MSE loss. The other guide tests how well the final word predictions line up with the true caption words, by comparing the model's guessed probabilities for each spot to the actual choice. That's cross-entropy loss. Together, these losses fine-tune the whole system.
Alex: Okay, so one loss for the noise removal accuracy, the other for getting the words right at the end.
Sam: Right. At test time, or inference, they speed things up further by starting with pure random noise and running fewer cleaning steps than during training—maybe just a handful instead of hundreds—while keeping the output quality steady. This uses a sampling trick called DDIM, which predicts the path from noise to clean text more directly, like jumping ahead on a map instead of following every twisty road.
Alex: Fewer steps sounds efficient... Does the paper show this holds up in actual tests, beyond just the speed?
Sam: The tests on datasets like MSR-VTT—with thousands of short video clips each paired with human-written descriptions—confirm it. DiffVC reaches scores matching the best step-by-step models, while being two to three times faster overall. An ablation test removing the discriminative denoiser dropped quality sharply, as mixing video and text in shared self-attention weakened both text grammar and visual links.
Alex: Huh—so without that targeted querying, the text modeling gets sloppy too...
Sam: Exactly. Qualitative examples highlight this: DiffVC spots rare details like a sunset that others miss, and distinguishes fine points like "girl" versus "woman" in scenes, producing more complete and accurate descriptions than prior fast methods. The paper notes it's the best among non-autoregressive approaches on most metrics across MSR-VTT, MSVD, and VATEX datasets.
Alex: That ties the mechanism back to real gains... Without overcomplicating the blend.
Sam: The paper backs this with tests that tweak one piece at a time—researchers call these ablation studies—to isolate what drives the gains. Removing the targeted querying from the denoiser hurt scores across the board, confirming its role in keeping visuals sharp. Skipping the refining layers in the language model also weakened the final text, as raw cleaned embeddings alone couldn't polish grammar and flow reliably.
Alex: So those tweaks show each part pulls its weight... But were there spots where adding more layers or steps didn't help?
Sam: Yes, balance matters. Deeper denoiser stacks peaked at twelve layers before scores dipped, as extra depth added noise without gain—like over-polishing a sculpture until details blur. The language model hit its sweet spot at six blocks; too few starved text refinement, too many overfit to training quirks. Inference settled on twenty cleaning steps for a practical trade-off: fewer rushed quality, more slowed speed without much lift.
Alex: Right, so even with the speed edge, it's not perfect—quality dips a bit behind the top slow models on some tests, like shorter clips where precision rules.
Sam: The evidence points to that caution—on MSVD, it trails the best autoregressive by about ten points on one key metric, though it leads all fast rivals. Those step choices highlight the ongoing tension: push for peak quality, and generation crawls; prioritize speed, and details soften slightly. Still, it sets a notable benchmark for non-autoregressive work.
Alex: Makes sense as a step forward... not flawless, but closing the gap where it counts most, like longer real-world clips.
Sam: Pulling it together, DiffVC delivers captions that match slow methods' quality at two-to-three times the speed by cleanly linking video guides to text sharpening—addressing error buildup and dilution head-on. Practically, this could enable instant subtitles for accessibility, quick search tagging of uploads, or content checks at scale, without users waiting. The paper suggests it's a meaningful advance for reliable, fast video understanding.
Alex: Yeah, that grounded push toward usable speed feels right... especially with those real gains on details like actions or scenes. Thanks, Sam—this has been a clear look at how careful design tackles a tough bottleneck.
Sam: My pleasure, Alex. That's DiffVC in essence. Thanks for listening to ResearchPod.