Spatial Tool-Use Elicits Reasoning for Spatial Intelligence | Yalun Dai et al. | ResearchPod

Transcript: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Alex: Welcome to another episode of ResearchPod. Today, we're looking at how AI systems understand the physical world — specifically, how they make sense of three-dimensional space.

Sam: We're discussing a new paper on a system called S-AGENT. The central argument is that current AI models struggle with 3D space because they treat it like a series of disconnected photographs, rather than a continuous environment that exists and changes over time.

Alex: So the problem isn't just that the models are too small or too slow — it's that they're missing something more fundamental about how space works?

Sam: Exactly. Think about how you'd map a house if you could only peek through a keyhole, one glimpse at a time, with no memory of what you'd seen before. That's essentially what most AI models do. They process each image as a frozen snapshot. The researchers argue that real spatial understanding isn't passive — it's an active process of gathering clues, building a picture piece by piece.

Alex: It's the difference between seeing a photograph of a car and actually watching it move down a street. One gives you a moment; the other gives you context.

Sam: That's a good way to put it. And the system they've built tries to replicate that second kind of understanding. They call their approach "Hierarchical Spatial Evidence Acquisition," which is a formal name for something fairly intuitive: a structured, three-stage method for building a mental model of a space from raw image data.

Alex: What are the three stages?

Sam: The first stage identifies objects in a flat, two-dimensional image — finding a chair, a table, a doorway. The second stage takes those flat shapes and calculates their depth — working out how far away each object actually is, and turning those 2D outlines into genuine 3D positions. The third stage brings in specialized tools to answer specific spatial questions, like whether the chair is to the left of the table, or how tall a particular object is.

Alex: So it's a bit like a detective building a case file. You don't try to solve the whole thing at once — you gather evidence, organize it, and then draw conclusions.

Sam: That analogy holds up well. The system acts as what the paper calls a "semantic planner" — essentially a coordinator that decides what information is still missing and which tool to call next. Crucially, the main AI doesn't try to do all the complex spatial mathematics itself. It delegates. It tells the specialist tools what it needs, and they handle the hard calculations.

Alex: That's an interesting design choice. Why not just have one powerful model do everything?

Because trying to do everything at once is where these systems tend to fail. If you ask a model to simultaneously identify objects, calculate depths, and reason about their relationships — all from a single image — the errors compound. By separating those tasks, each step can be done well, and mistakes don't cascade.

Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Yalun Dai, Hao Li, Shulin Tian, Runmao Yao, Yuhao Dong, Fangzhou Hong, Zhaoxi Chen, Fangfu Liu, Baoliang Tian, Dingwen Zhang, Tao Wang, Kim-Hui Yap, Ziwei Liu

Jun 18, 2026·5:19·arxiv·Original Paper

S-AgentSpatio-temporal evidence accumulationDual-memory systemVLM semantic plannerSpatial IntelligenceS-AGENT

Listen

5 min

Briefing

Abstract

Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

Spatial Tool-Use Elicits Reasoning for Spatial Intelligence: Research Paper Podcast

Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Listen