Multimodal large language models (MLLMs) answer questions about whole images well, yet they still struggle with fine-grained region understanding. A key reason is that text-only supervision never specifies which regions support an answer. To close this gap, we present Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 that predicts masks from its own visual representations. Specifically, a lightweight decoder combines vision-encoder patch features with visual latents, the language model's mask-token states, so mask supervision reaches the shared backbone without an additional segmentation encoder. Because each latent decodes to a mask, the model can interleave reasoning with region-specific latents and thus think with visual latents, while grounded reasoning traces supervise the regions it visits during chain-of-thought. As a result, across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in referring comprehension and achieves the highest segmentation cIoU among compared models. Moreover, it surpasses Qwen3.5 across all perception benchmarks. Homepage: \href{https://seerray-lab.github.io/Xiaomi-Lens-0}{this https URL}
Sam: Teaching a model to outline objects can improve its answers about what it sees. This report finds that pixel-level training helps not just object outlines, but answers about counting, color, and other visual details.
Alex: That sounds useful beyond systems that draw outlines. Is the claim that locating visual evidence helps a model answer ordinary questions?
Sam: Yes, within the benchmarks tested. This is Xiaomi-Lens-0, a technical report from the Xiaoai Plus and SeerRay team. Their model builds on Qwen3.5, an image-and-language model, and trains it to identify precisely which pixels belong to requested regions.
Alex: So this matters to researchers building visual assistants, not just object-selection tools. The question is whether spatial training improves the answers people actually receive.
Sam: That is the central test. We'll follow the shared-feature design, the comparisons that separate its contributions, and the conditions where it still struggles.
Alex: Start with the problem. Why wouldn't training a model on correct answers already teach it where to look?
Sam: The authors argue that a text answer leaves the supporting region unspecified. A model can receive credit for an answer without explicit supervision about which object supports it, or where that object ends. They connect this gap to failures like miscounting nearby objects and misreading small regions.
Alex: But models that produce object outlines already exist. What changes here beyond adding another output?
Sam: An outline here is a segmentation mask: pixel labels marking the target region. Systems such as LISA and Sa2VA get detailed spatial features from an additional segmentation encoder. Xiaomi-Lens-0 instead requires the image features already used by the language model to support mask prediction.
Alex: That changes where the training pressure lands. But wasn't there also prior work using the model's own image features?
Sam: There was. PixelLM and MLLMSeg do that, but keep their vision encoders frozen. Here, joint training lets the mask error update both the shared image encoder and the language-model states identifying targets. The proposed benefit is better shared representations, not simply a better separate outlining tool.
Alex: Make that architecture concrete. How does the model connect a phrase about an object to the pixels belonging to it?
Sam: Its image encoder produces a grid of patch features, each associated with an image location. The language pathway receives a compressed version of those features. The mask decoder uses the grid before that compression, preserving spatial detail that the compression can discard.
Alex: So it keeps a more detailed grid for outlining. What tells that grid which object the current sentence means?
Sam: The model generates a special mask token. Its contextual state is what the authors call a visual latent: a language-model state representing the region being discussed. A lightweight decoder combines that state with the shared image grid to predict the mask.
Alex: That also suggests a way to mark several objects during an explanation, rather than outlining only the final target.
Sam: The report does exactly that with grounded reasoning traces: explanations whose region references have mask annotations. In one example, the model counts people in separate boats, attaching a mask to each person before adding the counts. The intermediate references get explicit spatial training, alongside supervision of the text and final answer.
Alex: Calling that “thinking with visual latents” could imply the model examines each outline before continuing. Does it actually use the decoded masks to correct its reasoning?
Sam: No. During inference, mask decoding runs alongside text generation. Neither crops nor region features from those masks are fed back into the language model. The masks make intermediate region references inspectable, but this is not a visual feedback loop.
Alex: Then the training examples carry a lot of responsibility. How do the authors keep plausible explanations from getting attached to the wrong regions?
Sam: For referring instructions, they establish target regions first, then generate descriptions and check that those descriptions recover the intended objects. For reasoning data, an independent checker must reproduce the proposed answer before traces are generated. Accepted region references are paired with masks, with a box-overlap check for consistency.
Alex: Let's turn to the evidence. How much better is the finished model at finding a described object?
Sam: Compared with its Qwen backbone, it gains an average of 5.6 percentage points in referring expression comprehension—locating an object from a description. It improves on all eight tested splits of the established referring datasets. The gains also persist when evaluation demands tighter box placement.
Alex: Finding the right box is one test. Does the lightweight decoder also produce competitive pixel-level boundaries?
Sam: It has the highest pooled mask-overlap score among compared models on every tested referring-segmentation split. Its average is about 81 percent, compared with about 79 for Qwen3-VL-Seg, the strongest baseline on average. That score pools pixel intersections and unions, so large targets receive more weight.
Alex: Those are spatial outputs. The broader claim needs improvement when the model only gives a text answer.
Sam: It outperforms its backbone on every perception benchmark reported, covering general images, high-resolution detail, counting, and color. On CountQA, counting accuracy rises from about 44 percent to about 51 percent. Those evaluations require no box or mask output.
Alex: But the finished model has additional training data and box supervision too. How much of that improvement can we actually credit to masks?
Sam: The most useful comparison disables mask loss while keeping the same later-stage grounding and reasoning data. Box-only training already improves localization and perception substantially. Adding mask supervision raises every evaluated metric further, but those additional gains are smaller than the headline differences from the original backbone.
Alex: So the full improvement isn't all a mask effect. The controlled comparison supports an extra benefit from supervising region extent.
Sam: That's the narrower, supported conclusion. Another ablation removes grounded reasoning data while retaining masks. Most referring gains remain; adding the reasoning traces gives smaller improvements, especially in mask quality. I'd read the evidence as strong for these benchmark gains, without making the reasoning traces carry the whole explanation.
Alex: Where does performance break down? Familiar photographs and direct object descriptions may not capture the situations researchers care about.
Sam: The authors introduce LensBench to broaden that test. It contains 4,800 instruction-level samples across general, aerial, driving, and panoramic images. Each reviewed unit pairs direct and indirect wording for the same targets, and separately tests a single target versus multiple targets.
Alex: Holding the targets fixed makes the wording comparison useful. Can the model identify something through its function instead of its category name?
Sam: Sometimes, but that remains a weakness. Xiaomi-Lens-0 leads mask overlap in aerial, driving, and panoramic imagery, not general imagery. Its score falls substantially for indirect instructions. Selecting multiple targets in panoramic scenes is particularly difficult, so strong outlining does not guarantee reliable interpretation of what to select.
Alex: Given those strengths and limits, who should read the full report, and where should they begin?
Sam: Researchers training image-and-language models should start with the supervision ablation in Section 4.4, then the shared-feature architecture in Section 2.2. Researchers evaluating grounding should read LensBench's paired design and its indirect-instruction results. For everyone else, remember this: teaching a model where an answer comes from can improve what it says.
Alex: And knowing what to outline remains a separate test.