Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen
5 min
Spatial intelligence is a core requirement for embodied agents, robotic planning, and multimodal assistants. Although modern vision-language models (VLMs) have advanced rapidly, spatial reasoning remains difficult. Existing approaches generally follow two paths: post-rescaling training (such as supervised fine-tuning and reinforcement learning) or agentic paradigms that invoke external spatial tools like depth estimators and 3D reconstruction systems at inference time. This paper investigates an underexplored alternative: parameter-update-free self-evolution. Specifically, can a frozen VLM improve its spatial reasoning by maintaining an external memory bank of reusable lessons without modifying model weights or depending on external spatial tools?
The authors introduce Spatial Memory Agent (SMA), an experience-grounded runtime framework that converts verified spatial experience into reusable procedural memories. During an experience acquisition phase, SMA queries a frozen VLM, evaluates the prediction using a verifier, and applies verifier-guided reflection to distill compact transferable lessons. These lessons are then stored in an external memory bank and deployed for read-only inference on new spatial tasks.
The SMA framework operates across two main phases: experience acquisition and read-only deployment. In the acquisition phase, spatial problems consisting of visual inputs, natural-language tasks, and verified targets are processed by the frozen VLM. A reflection model then compresses successful or failed rollouts into structured memory cards containing a source task, a short summary, and a transferable lesson.
A key innovation in SMA is its two-stage retrieval and calibration mechanism. Standard retrieval relies solely on semantic similarity, which can mistakenly prioritize superficially similar but unhelpful procedures. SMA addresses this by combining semantic filtering with a Transfer Reliability Score (TRS). The TRS is initialized uniformly and updated online using a visit-evidence estimator that tracks how frequently a retrieved memory successfully guides subsequent tasks. During read-only deployment, the memory bank remains fixed, and retrieved procedures guide the frozen VLM inference without any parameter updates or writeback operations.
The authors evaluate SMA across five representative spatial benchmarks—RoboSpatial, ERQA, Omni3D, SAT, and EmbSpatial—using four different frozen base VLMs (Qwen3.5-9B, Qwen3.5-122B-A10B, Qwen3.6-27B, and Qwen3.6-35B-A3B). Across all evaluated base models, SMA achieves the highest macro average accuracy compared to standard inference without memory, traditional retrieval-augmented generation (RAG), and existing memory-reinforcement baselines. For instance, on Qwen3.6-27B, SMA achieves a macro average accuracy of 69.8%, outperforming the strongest baseline by 1.7 points and non-memory inference by 6.5 points. These results demonstrate that reliability-aware procedure selection provides a robust, scalable path for spatial self-evolution across different model scales.
Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.
Alex: That's exactly the problem the researchers address with what they call a Transfer Reliability Score, or TRS. Think of it as a track record rating for each memory card. Every time a card is used to help solve a new problem, the system notes whether it actually led to a correct answer. Cards with a strong track record get a high score. Cards that tend to mislead get a low score.
Sam: So when a new problem comes in, the system isn't just searching for the most similar past lesson — it's also filtering by trustworthiness.
Alex: Right. It's a two-stage process. First, find lessons that seem relevant to the current problem. Then, among those, rank them by reliability. The model ends up drawing on advice that has genuinely proven useful before, not just advice that superficially resembles the current situation.
Sam: And all of this happens without touching the model's internal weights — its core learned parameters — at all?
Alex: Correct. The model itself never changes. What improves is the quality of the external library it consults. It's the difference between making someone smarter and giving them access to a better, more curated set of reference materials.
Sam: That's a meaningful architectural choice. Did the results actually bear it out?
Alex: The researchers tested SMA across five different spatial reasoning benchmarks — a range of tasks designed to probe different aspects of understanding space from images. In nearly every case, SMA outperformed the comparison methods. The pattern held consistently, which suggests the approach is robust rather than tuned to one specific type of problem.
Sam: What strikes me about that is what it implies about intelligence more broadly. It's not purely about the raw capacity of the model. It's about how well a system can organize and selectively retrieve its own past experience.
Alex: That's a fair reading of what the paper suggests. And it points toward something worth thinking about for AI development generally — that self-improvement doesn't necessarily require retraining from scratch. A system that reflects carefully on what worked, stores those insights reliably, and retrieves them wisely can get meaningfully better over time, simply by learning from itself. Thanks for listening to ResearchPod.