Xiaoai Plus, SeerRay Team
8 min
Abstract
Multimodal large language models (MLLMs) answer questions about whole images well, yet they still struggle with fine-grained region understanding. A key reason is that text-only supervision never specifies which regions support an answer. To close this gap, we present Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 that predicts masks from its own visual representations. Specifically, a lightweight decoder combines vision-encoder patch features with visual latents, the language model's mask-token states, so mask supervision reaches the shared backbone without an additional segmentation encoder. Because each latent decodes to a mask, the model can interleave reasoning with region-specific latents and thus think with visual latents, while grounded reasoning traces supervise the regions it visits during chain-of-thought. As a result, across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in referring comprehension and achieves the highest segmentation cIoU among compared models. Moreover, it surpasses Qwen3.5 across all perception benchmarks. Homepage: \href{https://seerray-lab.github.io/Xiaomi-Lens-0}{this https URL}
Alex: So it keeps a more detailed grid for outlining. What tells that grid which object the current sentence means?
Sam: The model generates a special mask token. Its contextual state is what the authors call a visual latent: a language-model state representing the region being discussed. A lightweight decoder combines that state with the shared image grid to predict the mask.
Alex: That also suggests a way to mark several objects during an explanation, rather than outlining only the final target.
Sam: The report does exactly that with grounded reasoning traces: explanations whose region references have mask annotations. In one example, the model counts people in separate boats, attaching a mask to each person before adding the counts. The intermediate references get explicit spatial training, alongside supervision of the text and final answer.
Alex: Calling that “thinking with visual latents” could imply the model examines each outline before continuing. Does it actually use the decoded masks to correct its reasoning?
Sam: No. During inference, mask decoding runs alongside text generation. Neither crops nor region features from those masks are fed back into the language model. The masks make intermediate region references inspectable, but this is not a visual feedback loop.
Alex: Then the training examples carry a lot of responsibility. How do the authors keep plausible explanations from getting attached to the wrong regions?
Sam: For referring instructions, they establish target regions first, then generate descriptions and check that those descriptions recover the intended objects. For reasoning data, an independent checker must reproduce the proposed answer before traces are generated. Accepted region references are paired with masks, with a box-overlap check for consistency.
Alex: Let's turn to the evidence. How much better is the finished model at finding a described object?
Sam: Compared with its Qwen backbone, it gains an average of 5.6 percentage points in referring expression comprehension—locating an object from a description. It improves on all eight tested splits of the established referring datasets. The gains also persist when evaluation demands tighter box placement.
Alex: Finding the right box is one test. Does the lightweight decoder also produce competitive pixel-level boundaries?
Sam: It has the highest pooled mask-overlap score among compared models on every tested referring-segmentation split. Its average is about 81 percent, compared with about 79 for Qwen3-VL-Seg, the strongest baseline on average. That score pools pixel intersections and unions, so large targets receive more weight.
Alex: Those are spatial outputs. The broader claim needs improvement when the model only gives a text answer.
Sam: It outperforms its backbone on every perception benchmark reported, covering general images, high-resolution detail, counting, and color. On CountQA, counting accuracy rises from about 44 percent to about 51 percent. Those evaluations require no box or mask output.
Alex: But the finished model has additional training data and box supervision too. How much of that improvement can we actually credit to masks?
Sam: The most useful comparison disables mask loss while keeping the same later-stage grounding and reasoning data. Box-only training already improves localization and perception substantially. Adding mask supervision raises every evaluated metric further, but those additional gains are smaller than the headline differences from the original backbone.
Alex: So the full improvement isn't all a mask effect. The controlled comparison supports an extra benefit from supervising region extent.
Sam: That's the narrower, supported conclusion. Another ablation removes grounded reasoning data while retaining masks. Most referring gains remain; adding the reasoning traces gives smaller improvements, especially in mask quality. I'd read the evidence as strong for these benchmark gains, without making the reasoning traces carry the whole explanation.
Alex: Where does performance break down? Familiar photographs and direct object descriptions may not capture the situations researchers care about.
Sam: The authors introduce LensBench to broaden that test. It contains 4,800 instruction-level samples across general, aerial, driving, and panoramic images. Each reviewed unit pairs direct and indirect wording for the same targets, and separately tests a single target versus multiple targets.
Alex: Holding the targets fixed makes the wording comparison useful. Can the model identify something through its function instead of its category name?
Sam: Sometimes, but that remains a weakness. Xiaomi-Lens-0 leads mask overlap in aerial, driving, and panoramic imagery, not general imagery. Its score falls substantially for indirect instructions. Selecting multiple targets in panoramic scenes is particularly difficult, so strong outlining does not guarantee reliable interpretation of what to select.
Alex: Given those strengths and limits, who should read the full report, and where should they begin?
Sam: Researchers training image-and-language models should start with the supervision ablation in Section 4.4, then the shared-feature architecture in Section 2.2. Researchers evaluating grounding should read LensBench's paired design and its indirect-instruction results. For everyone else, remember this: teaching a model where an answer comes from can improve what it says.
Alex: And knowing what to outline remains a separate test.