Transcript: The Attention Triangle in Audio-Video Models

Alex: Welcome to another episode of ResearchPod. Today we're looking at a specific failure mode in joint audio-video diffusion models — the problem of source attribution. When you prompt one of these models to generate a scene with a parrot speaking while a pirate stands silently nearby, the model frequently ignores that instruction and has the human speak instead. This paper asks why that happens structurally, and whether you can fix it without retraining.

Sam: So it's not just a prompt-following failure in the usual sense — the model isn't confused about what you want, it's actively overriding it?

Alex: That's the argument. The authors frame it as systematic semantic leakage. The cross-modal attention mechanisms have learned strong statistical priors from training data — humans speak, objects make ambient noise — and those priors can dominate over the explicit source-sound bindings in the prompt. The model isn't hallucinating randomly; it's taking the path of least resistance through its own learned associations.

Sam: And the paper claims this is structural — baked into how the attention is architected across modalities?

Alex: Right. They introduce what they call the Attention Triangle: three cross-attention edges connecting text, audio, and video streams — text-to-video, text-to-audio, and audio-to-video. The key insight is that these edges are coupled. When the model processes a prompt, information doesn't flow cleanly along each edge independently. The coupling means a strong prior on one edge — say, audio-to-video associating speech with human faces — can propagate and corrupt the attribution decision even when the text edge is correctly aligned to the parrot.

Sam: So the failure isn't localized to one attention head or one modality pair. It's a triangle-wide routing problem.

Alex: Exactly. And that framing has a direct implication for intervention. If you only fix one edge, you haven't addressed the coupled system. The authors test this explicitly — steering individual edges in isolation partially improves attribution, but the full effect only emerges when you steer all three jointly.

Sam: How do they actually implement the steering?

Alex: At inference time, during the denoising process, they modify the pre-softmax attention logits directly. They define masks over token positions that correspond to the intended source-sound binding — the parrot in this case — versus the conflicting association — the pirate. Then they apply additive biases: reinforcing attention toward the prompt-aligned tokens and suppressing it toward the bias-driven ones. No gradient updates, no fine-tuning. The intervention happens entirely within a single forward pass at each denoising step.

That's a fairly surgical intervention. Does it actually hold up across varied prompts, or is it tuned to specific cases?