Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage. We study these models by probing and analyzing the ``attention triangle,'' comprising the three cross-attention edges connecting the text, audio, and video streams, and examine how semantic information is routed across modalities during generation. Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage: when prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward visually canonical but incorrect outcomes. These effects suggest that semantic artifacts arise not merely from attention spreading beyond its intended target, but from structured, bias-driven interactions along specific pathways. Building on this perspective, we extract attention-derived signals that expose how semantics are distributed and grounded across modalities, and use them as a diagnostic tool to both analyze and deliberately incur leakage under controlled conditions. This enables us to probe the internal dynamics of cross-modal routing and isolate the role of individual interactions. We further leverage these signals to guide inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments support our analysis and demonstrate improved semantic grounding while preserving generation quality.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a specific failure mode in joint audio-video diffusion models — the problem of source attribution. When you prompt one of these models to generate a scene with a parrot speaking while a pirate stands silently nearby, the model frequently ignores that instruction and has the human speak instead. This paper asks why that happens structurally, and whether you can fix it without retraining.
Sam: So it's not just a prompt-following failure in the usual sense — the model isn't confused about what you want, it's actively overriding it?
Alex: That's the argument. The authors frame it as systematic semantic leakage. The cross-modal attention mechanisms have learned strong statistical priors from training data — humans speak, objects make ambient noise — and those priors can dominate over the explicit source-sound bindings in the prompt. The model isn't hallucinating randomly; it's taking the path of least resistance through its own learned associations.
Sam: And the paper claims this is structural — baked into how the attention is architected across modalities?
Alex: Right. They introduce what they call the Attention Triangle: three cross-attention edges connecting text, audio, and video streams — text-to-video, text-to-audio, and audio-to-video. The key insight is that these edges are coupled. When the model processes a prompt, information doesn't flow cleanly along each edge independently. The coupling means a strong prior on one edge — say, audio-to-video associating speech with human faces — can propagate and corrupt the attribution decision even when the text edge is correctly aligned to the parrot.
Sam: So the failure isn't localized to one attention head or one modality pair. It's a triangle-wide routing problem.
Alex: Exactly. And that framing has a direct implication for intervention. If you only fix one edge, you haven't addressed the coupled system. The authors test this explicitly — steering individual edges in isolation partially improves attribution, but the full effect only emerges when you steer all three jointly.
Sam: How do they actually implement the steering?
Alex: At inference time, during the denoising process, they modify the pre-softmax attention logits directly. They define masks over token positions that correspond to the intended source-sound binding — the parrot in this case — versus the conflicting association — the pirate. Then they apply additive biases: reinforcing attention toward the prompt-aligned tokens and suppressing it toward the bias-driven ones. No gradient updates, no fine-tuning. The intervention happens entirely within a single forward pass at each denoising step.
Sam: That's a fairly surgical intervention. Does it actually hold up across varied prompts, or is it tuned to specific cases?
Alex: Their ablations suggest it generalizes reasonably well across the prompt types they tested. Joint steering restores both visual appearance and audio attribution — meaning the parrot looks like a parrot and sounds like one. But I'd flag the scope here: the evaluation is on a constrained set of scenarios where the conflicting prior is well-defined. The harder question — whether the masking procedure degrades generation quality in cases where the model's prior is actually correct — gets less attention in the paper.
Sam: That feels like the natural place a referee would push back. If you're suppressing learned associations globally, you'd expect some cost to overall coherence.
Alex: The authors acknowledge it, but frame it as an acceptable trade-off given the targeted nature of the masks. Their position is that you're only steering when you've identified a specific conflict — you're not applying this blindly. Whether that holds in practice, when users don't know in advance where the model's priors will misfire, is a fair open question.
Sam: And the deeper limitation they flag?
Alex: The model's internal representation of space and time is what they call somewhat degenerate — meaning the model doesn't have a clean, factored representation of which entity occupies which spatial region at which moment. The Attention Triangle framing and the logit steering are working around that, not fixing it. So this is explicitly positioned as an inference-time patch, not a solution to the underlying representational problem. A model with a properly factored spatiotemporal representation probably wouldn't need this intervention in the first place.
Sam: So the contribution is partly the fix, but also the diagnostic — the Attention Triangle as a framework for understanding why these failures occur.
Alex: That's probably the more durable contribution. Moving from "the model hallucinated the wrong speaker" to "here is the specific coupled routing mechanism that produced that failure, and here is how you can steer it" is a meaningful shift in how we reason about these systems. It turns a black-box failure into something mechanistically interpretable and, at least in controlled settings, correctable.
Sam: Which suggests the next step is either architectural — building models that don't have this coupling problem — or developing more robust masking procedures that don't require the user to pre-specify where the conflict is.
Alex: Both seem like natural follow-ons. For now, the paper gives practitioners a concrete handle on a failure mode that's otherwise difficult to diagnose, and gives the research community a cleaner vocabulary for talking about cross-modal attribution in diffusion models.
Alex: Thanks for listening to ResearchPod.