Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive $\rightarrow$ infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.
Sam: A model can hear the crucial line of audio and still answer as if it heard nothing. Ask it to write down what it heard first, and the right answer often comes back.
Alex: So the problem isn't perception. The information is in there, and it doesn't make it out. Whose work is this?
Sam: It's a method paper from researchers at KAIST and NAVER AI Lab, posted to arXiv as a preprint in October 2026. They study omni-modal language models, meaning models that take video, audio, and sometimes other signals alongside text. They call the failure "knows but cannot say."
Alex: Why should someone who doesn't build these models care?
Sam: Because it changes where you look when these models underperform. The usual suspects are training data and training recipes. This paper argues that part of the gap is about access: the model encoded the audio, but the part that writes the answer barely uses it. So we'll cover how they showed that, the simple fix they propose, and how far to trust the numbers.
Alex: Give me the concrete case first.
Sam: They have a video clip and a question: what gift did the man prepare for his daughter? Answering directly, the model says neither the image nor the audio gives any detail about a gift. Then they ask the same model to describe what it sees and hears. It says a room full of people are looking up at the ceiling, and a girl says "Let there be light!" Given those descriptions, it answers: a petal-shaped chandelier. Correct.
Alex: That example is more than two modalities stacked. Does the problem change as you add inputs?
Sam: It gets worse. The authors report that perception holds up as modalities are added, while answer accuracy drops with each one. Their reading is that the bottleneck is in read-out, not encoding.
Alex: People already measure the "modality gap," how far apart image and text embeddings sit. Why wasn't that enough?
Sam: Those measures come from dual-encoder models like CLIP, where an image and its caption are trained to match. In an omni-model the inputs aren't paraphrases of each other. An audio track isn't a restatement of the video. And each input is hundreds or thousands of tokens, so one summary vector blurs which tokens matter. The question here isn't whether audio matches video. It's whether each audio token connects to what the model is generating.
Alex: So they measure each input token against the answer, not against the other inputs.
Sam: Right, and that measure is called Token-level Modality Alignment, or TMA. They take the last decoder layer's hidden state for every input token. They find the most similar token in the generated response and record that similarity. Then they keep the whole distribution, rather than averaging it away.
Alex: Similarity to the response sounds circular. The response is whatever the model happened to say. Is the score stable?
Sam: They checked. They resampled responses several times from the same prompt. Token-level scores were largely consistent, and more so for larger models. Random filler text was the least stable, which fits: its score depends on incidental word overlap.
Alex: What did TMA show?
Sam: Two things. First, attention, how much the decoder looks at each input while answering, splits sharply. Text tokens get far more attention than audio or video tokens. That holds even for text unrelated to the question, wherever it's inserted. Audio and video tokens with higher TMA do get somewhat more attention, but the link is modest and the absolute attention stays low.
Alex: And the second?
Sam: A masking test. They zero out input tokens in low, middle, or high TMA tiers. Then they check two things: can the model still describe the input, and can it still answer the question? Description ability barely moves. Answering drops, and drops more when high-TMA tokens are masked. That separation is their evidence that TMA tracks access to generation, not just whether information is present.
Alex: So text is the channel the decoder actually reads. The fix, I'd guess, is to turn the audio into text.
Sam: That's the method, called Omni-Catalyst. Perceive, then infer. The model writes its own description of each modality, then answers using the original inputs plus that description. Their analysis of attention shows the description step mostly helps the weaker modality, usually audio. There's a two-pass version and a cheaper single-pass version that writes description and answer together.
Alex: A prior method, OmniXR, already extracts modality information as text before reasoning. What's different?
Sam: OmniXR replaces the modalities with the text. Omni-Catalyst keeps the originals and adds the description. The authors tested replacement directly, and swapping inputs for descriptions degraded performance. The gain depends on keeping both.
Alex: Let's get to numbers. What's the headline, and against what?
Sam: Across seven omni-models, the average gain is about 7 percent. On Qwen2.5-Omni 7-billion, the average across benchmarks went from about 56 to 64. On the same model, Chain-of-Modality, a planner-reasoner-decider approach, gained about 6. ThinkOmni, which hands reasoning to an external text-only model, actually lost about 17 points. On the two benchmarks that require combining vision and audio, the competing approach gained up to 3 points, and Omni-Catalyst gained 5 to 7.
Alex: Couldn't any second pass help, just from extra compute?
Sam: They tested that with self-correction, a second chance to revise the first answer. It gained roughly 3 percent, with almost no change in modality balance. For a ceiling, they fed in ground-truth audio descriptions, which gained about 13 percent. Omni-Catalyst moves in that same direction. Existing methods that shrink the modality gap in embedding space left performance and modality balance largely unchanged.
Alex: Does this go beyond audio and video?
Sam: They tried brain scans, IMU data from wearable motion sensors, and 3D point clouds. Gains were significant, larger for weaker modalities. One caveat: the authors built those tasks themselves. They mapped captions to categories with keyword scoring, so these aren't standard benchmarks.
Alex: Can you train this behaviour in?
Sam: Partly. Training the model only to describe its inputs gave the biggest training gain, under 3 points. That beat training it to answer directly, and beat several self-training methods. Adding the inference-time description on top added another 5 to 6 points. Every training variant then landed in a similar range. The authors read this as: improve perception before optimizing the answer.
Alex: What's the limitation that matters most?
Sam: They don't explain why the imbalance exists. They list candidates, like the language-centric architecture, pretraining data, or the difficulty of aligning very different signals, and leave it open. And the gains are uneven. The largest model, Qwen3-Omni 30-billion, gained only about one point on one benchmark, while OmniVinci gained over 11. Some categories even went down. The two-pass version also costs an extra inference call. And their failure cases are questions needing inference beyond what's visible or audible, like off-screen sounds or intentions.
Alex: So who should read the full paper, and where should they start?
Sam: Anyone building or evaluating omni-modal models. Start with the masking analysis that separates what the model can describe from what it can answer. Then read the comparison of adding descriptions versus replacing inputs with them. If you work on representation alignment, the section showing that modality-gap fixes don't improve answers is worth a careful look.
Alex: And for everyone else?
Sam: The model often heard it. It just didn't say it until it wrote it down.
Alex: Worth trying before you retrain anything.