Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how AI systems understand the physical world — specifically, how they make sense of three-dimensional space.
Sam: We're discussing a new paper on a system called S-AGENT. The central argument is that current AI models struggle with 3D space because they treat it like a series of disconnected photographs, rather than a continuous environment that exists and changes over time.
Alex: So the problem isn't just that the models are too small or too slow — it's that they're missing something more fundamental about how space works?
Sam: Exactly. Think about how you'd map a house if you could only peek through a keyhole, one glimpse at a time, with no memory of what you'd seen before. That's essentially what most AI models do. They process each image as a frozen snapshot. The researchers argue that real spatial understanding isn't passive — it's an active process of gathering clues, building a picture piece by piece.
Alex: It's the difference between seeing a photograph of a car and actually watching it move down a street. One gives you a moment; the other gives you context.
Sam: That's a good way to put it. And the system they've built tries to replicate that second kind of understanding. They call their approach "Hierarchical Spatial Evidence Acquisition," which is a formal name for something fairly intuitive: a structured, three-stage method for building a mental model of a space from raw image data.
Alex: What are the three stages?
Sam: The first stage identifies objects in a flat, two-dimensional image — finding a chair, a table, a doorway. The second stage takes those flat shapes and calculates their depth — working out how far away each object actually is, and turning those 2D outlines into genuine 3D positions. The third stage brings in specialized tools to answer specific spatial questions, like whether the chair is to the left of the table, or how tall a particular object is.
Alex: So it's a bit like a detective building a case file. You don't try to solve the whole thing at once — you gather evidence, organize it, and then draw conclusions.
Sam: That analogy holds up well. The system acts as what the paper calls a "semantic planner" — essentially a coordinator that decides what information is still missing and which tool to call next. Crucially, the main AI doesn't try to do all the complex spatial mathematics itself. It delegates. It tells the specialist tools what it needs, and they handle the hard calculations.
Alex: That's an interesting design choice. Why not just have one powerful model do everything?
Sam: Because trying to do everything at once is where these systems tend to fail. If you ask a model to simultaneously identify objects, calculate depths, and reason about their relationships — all from a single image — the errors compound. By separating those tasks, each step can be done well, and mistakes don't cascade.
Alex: And what holds it all together as the camera moves through a space?
Sam: That's where the memory system comes in, and it's one of the more distinctive parts of the design. There are two separate memory stores. The first, which the paper calls "Scene Memory," holds persistent facts about the environment — the size and position of objects the system has already figured out. So if the camera pans away from a chair and comes back, the system doesn't have to recalculate where the chair is. It already knows.
Alex: And without that, the model would just forget everything the moment the view changed.
Sam: Precisely. The second store is called "Agent Memory," and it records something different — not what the system has seen, but how it has been thinking. It logs which tools were used, which ones produced useful results, and what questions are still open. This prevents the system from going in circles, repeating the same failed approach because it has no record of having tried it before.
Alex: So the memory isn't just a database of images. It's a record of the reasoning process itself.
Sam: That's the key distinction. And it's what makes the system more reliable than a model that just looks at a single frame and guesses. This one investigates. It accumulates evidence over time, updates its understanding, and keeps a log of its own work.
Alex: Does that mean a smaller model using this approach could outperform a much larger one that doesn't?
Sam: That's one of the paper's more significant claims. Their S-AGENT system, built on a relatively compact model, performs comparably to much larger closed-source systems on spatial reasoning tasks. The argument is that the bottleneck wasn't raw processing power — it was the inability to hold onto information across time and reason about it systematically. A well-organized smaller model can outperform a larger one that's essentially working blind.
Alex: Which suggests the design of the reasoning process matters as much as the size of the model.
Sam: And the paper takes that one step further. They use S-AGENT as a kind of teacher. By recording the step-by-step reasoning trajectories the system produces — the actual sequence of tool calls, observations, and conclusions — they can use that data to train even smaller, more compact models. The compact model learns not just the answers, but the method for arriving at them.
Alex: So the insight transfers. You don't need to run the full system every time — you can bake the reasoning approach into a smaller model.
Sam: In principle, yes. Though the paper is careful to note a real limitation: the whole chain depends on the quality of the initial measurements. If the depth estimation in stage two is inaccurate, that error propagates through everything that follows. The reasoning framework is only as reliable as the tools it's built on.
Alex: That's a useful caveat. It's a meaningful step forward, but not a complete solution.
Sam: That's a fair summary. The contribution is architectural — a clearer way of organizing how an AI gathers and retains spatial information. Whether that translates into reliable real-world performance depends on the quality of the underlying tools, and that remains an open question.
Alex: Thanks for walking us through it, Sam. And thanks for listening to ResearchPod.