Transcript: OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a challenge that anyone who's tried to get an AI to follow specific instructions has likely faced.

Sam: We're discussing a new study on what researchers call "Omni-modal Large Language Models," or OLLMs. Think of these as AI systems that can read text, listen to audio, and watch video all at the same time—rather than handling just one type of information. The puzzle the paper explores is that these models often struggle when you give them complex, multi-part instructions. For example, asking an AI to summarize a video in a specific table format while also tracking precise timestamps.

Alex: So the paper is asking: why does adding a strict formatting requirement seem to get in the way of the AI actually understanding the content?

Sam: Exactly. The researchers identified what they call a "format-content tradeoff." Here's the core idea: when you force an AI to follow rigid structural rules—say, outputting its answer as a perfectly formatted spreadsheet file—it actually pulls mental resources away from the task of reasoning accurately about what's in the video. Think of it like asking someone to write a thoughtful essay while simultaneously keeping every sentence to exactly ten words. The effort spent counting words degrades the quality of the ideas. The same thing appears to happen inside these AI models.

Alex: So the model has a kind of limited budget of attention, and formatting burns through that budget before it even gets to the substance.

Sam: That's a good way to put it. And to actually measure this, they built a new testing tool called "OmniCap-IF." The key design choice was splitting the evaluation into two completely separate checks. One automated system looks purely at structure—does the table have the right number of columns? Are the timestamps in the right place? A second, separate AI acts as a judge and asks whether the content itself is actually accurate. By keeping those two checks apart, they can pinpoint exactly which part of the task is breaking down.

Alex: So instead of a single pass/fail grade, they're diagnosing the problem—separating whether the model got the *shape* of the answer right from whether it got the *meaning* right.

Sam: Precisely. And that diagnosis led to a meaningful finding: simply training the model on more data doesn't resolve the tension. The real fix required rethinking the process itself. They developed what they call a "decomposed generation strategy." The idea is straightforward—have the model focus entirely on getting the content right first, then, in a separate follow-up stage, apply the formatting rules. It's the same logic as writing a rough draft before worrying about layout and presentation. By separating the two jobs, the model never has to juggle both at once.