Na Min An, Taekyung Kim, Jaemyung Yu, Hyunjung Shim, Dongyoon Han
8 min
Abstract
Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive $\rightarrow$ infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.
Alex: Similarity to the response sounds circular. The response is whatever the model happened to say. Is the score stable?
Sam: They checked. They resampled responses several times from the same prompt. Token-level scores were largely consistent, and more so for larger models. Random filler text was the least stable, which fits: its score depends on incidental word overlap.
Alex: What did TMA show?
Sam: Two things. First, attention, how much the decoder looks at each input while answering, splits sharply. Text tokens get far more attention than audio or video tokens. That holds even for text unrelated to the question, wherever it's inserted. Audio and video tokens with higher TMA do get somewhat more attention, but the link is modest and the absolute attention stays low.
Alex: And the second?
Sam: A masking test. They zero out input tokens in low, middle, or high TMA tiers. Then they check two things: can the model still describe the input, and can it still answer the question? Description ability barely moves. Answering drops, and drops more when high-TMA tokens are masked. That separation is their evidence that TMA tracks access to generation, not just whether information is present.
Alex: So text is the channel the decoder actually reads. The fix, I'd guess, is to turn the audio into text.
Sam: That's the method, called Omni-Catalyst. Perceive, then infer. The model writes its own description of each modality, then answers using the original inputs plus that description. Their analysis of attention shows the description step mostly helps the weaker modality, usually audio. There's a two-pass version and a cheaper single-pass version that writes description and answer together.
Alex: A prior method, OmniXR, already extracts modality information as text before reasoning. What's different?
Sam: OmniXR replaces the modalities with the text. Omni-Catalyst keeps the originals and adds the description. The authors tested replacement directly, and swapping inputs for descriptions degraded performance. The gain depends on keeping both.
Alex: Let's get to numbers. What's the headline, and against what?
Sam: Across seven omni-models, the average gain is about 7 percent. On Qwen2.5-Omni 7-billion, the average across benchmarks went from about 56 to 64. On the same model, Chain-of-Modality, a planner-reasoner-decider approach, gained about 6. ThinkOmni, which hands reasoning to an external text-only model, actually lost about 17 points. On the two benchmarks that require combining vision and audio, the competing approach gained up to 3 points, and Omni-Catalyst gained 5 to 7.
Alex: Couldn't any second pass help, just from extra compute?
Sam: They tested that with self-correction, a second chance to revise the first answer. It gained roughly 3 percent, with almost no change in modality balance. For a ceiling, they fed in ground-truth audio descriptions, which gained about 13 percent. Omni-Catalyst moves in that same direction. Existing methods that shrink the modality gap in embedding space left performance and modality balance largely unchanged.
Alex: Does this go beyond audio and video?
Sam: They tried brain scans, IMU data from wearable motion sensors, and 3D point clouds. Gains were significant, larger for weaker modalities. One caveat: the authors built those tasks themselves. They mapped captions to categories with keyword scoring, so these aren't standard benchmarks.
Alex: Can you train this behaviour in?
Sam: Partly. Training the model only to describe its inputs gave the biggest training gain, under 3 points. That beat training it to answer directly, and beat several self-training methods. Adding the inference-time description on top added another 5 to 6 points. Every training variant then landed in a similar range. The authors read this as: improve perception before optimizing the answer.
Alex: What's the limitation that matters most?
Sam: They don't explain why the imbalance exists. They list candidates, like the language-centric architecture, pretraining data, or the difficulty of aligning very different signals, and leave it open. And the gains are uneven. The largest model, Qwen3-Omni 30-billion, gained only about one point on one benchmark, while OmniVinci gained over 11. Some categories even went down. The two-pass version also costs an extra inference call. And their failure cases are questions needing inference beyond what's visible or audible, like off-screen sounds or intentions.
Alex: So who should read the full paper, and where should they start?
Sam: Anyone building or evaluating omni-modal models. Start with the masking analysis that separates what the model can describe from what it can answer. Then read the comparison of adding descriptions versus replacing inputs with them. If you work on representation alignment, the section showing that modality-gap fixes don't improve answers is worth a careful look.
Alex: And for everyone else?
Sam: The model often heard it. It just didn't say it until it wrote it down.
Alex: Worth trying before you retrain anything.