Huanyao Zhang, Jiepeng Zhou, Runhao Zhao, Yanzhe Shan, Jiaoyang Chen, Bowen Zhou, Bo Li, Fang Wang, Jialong Wu, Zhengwei Tao, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
7 min
Abstract
Multimodal large language models (MLLMs) have advanced visual understanding and reasoning, yet their static parametric knowledge limits their ability to address knowledge-intensive and dynamically evolving open-world problems. To move beyond this limitation, multimodal deep search has emerged as a key direction for open-world information access, evolving from single-turn factual retrieval toward long-horizon, multi-turn search guided by visual evidence. However, existing methods typically confine vision to the input or answer stage, overlooking its role in intermediate reasoning, and lack designs tailored to long-horizon interaction. Consequently, visual evidence rarely drives continued retrieval, constraining both interaction depth and reasoning span. To address these limitations, we propose DeepVoyager-VL, a long-horizon multimodal deep-search framework for vision-in-the-loop search. Specifically, we construct a multimodal event graph to drive data synthesis, yielding problems with intermediate visual dependencies and long reasoning chains. We then design an agent framework for active visual acquisition and on-demand image loading. Finally, we fine-tune models on the synthesized data without reinforcement learning. Extensive experiments across ten multimodal search benchmarks demonstrate the effectiveness of our method.
Sam: So it's being selective. It doesn't process every image it stumbles across—only the ones that function as those keys we talked about.
Alex: Right. And that selectivity is what makes the whole system efficient. The model isn't guessing; it's following a logical chain where visual evidence is a required step, not an optional extra.
Sam: Now, they build this without using reinforcement learning—the kind of trial-and-error training you often hear about. They use something called supervised fine-tuning instead. Why does that distinction matter?
Alex: Reinforcement learning is a bit like teaching someone to ride a bike by letting them fall repeatedly until they figure it out. It can work, but it's unpredictable and hard to control, especially over long, multi-step tasks. Supervised fine-tuning is more like a guided apprenticeship—you show the learner exactly what a correct solution looks like, step by step, and they learn the underlying logic from those examples.
Sam: So the "teacher trajectories" they mention are basically recordings of the AI doing the search correctly—a blueprint for it to study?
Alex: Exactly. It learns when to crop an image, when to run a reverse image search, when to move on—all from watching those correct examples, rather than stumbling toward them through trial and error.
Sam: But if it's learning from pre-set paths, doesn't it struggle when it hits a situation it hasn't seen before?
Alex: That's the clever part. It's not just memorizing the actions—it's learning the rules. The event graph makes the dependencies explicit, so the model understands that step B is only possible once you have the specific visual data from step A. It can apply that logic to new situations it's never encountered.
Sam: So it knows it can't skip to the finish line without the key, even in a brand-new scavenger hunt.
Alex: Precisely. And the results reflect that. On the benchmarks tested, this approach improved performance by more than fifty percent for smaller models compared to the same models trained without this method. That's a meaningful gap—and it comes from the quality of the reasoning, not just raw speed.
Sam: What's the mechanism there? Why does training on these structured examples produce such a clear improvement?
Alex: Because the model is trained on long, logical chains where every step is justified. It doesn't develop shortcuts or bad habits. When it encounters a new problem, it applies the same disciplined process: find the required visual evidence, use it to unlock the next step, repeat.
Sam: It's the difference between a student who memorized answers and one who actually understands the method.
Alex: That's a fair comparison. And that understanding is what makes it reliable on problems it hasn't seen before.
Sam: So where does this fall short? Because it sounds like a capable system, but I imagine it's not ready to write a full research paper.
Alex: Not at all. The system is optimized for answer-oriented search—finding a specific, verifiable fact. It hasn't been trained for open-ended tasks like synthesizing a long report, reconciling conflicting sources, or forming a hypothesis from scratch.
Sam: So it's more like a very capable research assistant that tracks down the right facts, rather than a researcher who builds a new argument.
Alex: That's the right framing. There's also a cost issue. The system uses an additional model to summarize what it finds on webpages, and that processing adds time and expense. The more pages it visits, the more that adds up.
Sam: So there's a trade-off between the quality of evidence it gathers and how quickly it can work.
Alex: Exactly. Future work will likely focus on making that evidence processing more efficient—through smarter, selective summarization, or caching results so the same page isn't processed twice.
Sam: And what about memory? As a search gets longer, does the agent start losing track of what it's already found?
Alex: That's a real limitation. Right now, its visual memory is essentially a growing pile of images with no internal organization. It lacks a way to compress, prioritize, or discard old evidence. As searches get longer, finding the right clue in that pile becomes harder.
Sam: So it needs a better filing system, not just a bigger memory.
Alex: Well put. The next step is likely a hierarchical memory—one that tracks what's important, where it came from, and how it connects to the current goal.
Sam: It sounds like the longer-term vision is moving from a search agent to something more like a research agent—one that manages its own plan and keeps track of its evidence.
Alex: That's where the field is heading. The paper points toward agents that maintain a clear record of where every claim comes from, so that everything they report can be traced back to a verifiable source. Right now, DeepVoyager-VL is a meaningful step in that direction—it demonstrates that vision can be a genuine driver of search reasoning, not just a passive input. But the harder problem of autonomous, open-ended research is still ahead.
Sam: So this is the foundation, not the finished building.
Alex: That's a fair way to put it. It's a clear improvement in how AI systems use visual information to think through a problem—and it points toward a more disciplined, verifiable kind of machine research. Thanks for listening to ResearchPod.