Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan
5 min
Most Vision-Language-Action (VLA) models operate under a Markovian assumption, predicting actions based solely on the current observation and instruction. This approach struggles with long-horizon tasks that require understanding past state transitions, task progress, and temporal dependencies. The authors ask: can historical experience be represented as context-native latent memory—stored, retrieved, and consumed within the same continuous embedding space used for multimodal reasoning?
LaMem-VLA introduces a framework that treats robotic history as a generative latent faculty. It consists of four key components:
By keeping memory within the latent space, the model avoids the limitations of external memory banks, which often treat historical data as an auxiliary scaffold rather than an integral part of the decision-making process.
LaMem-VLA demonstrates significant improvements in long-horizon robotic manipulation tasks. In evaluations on the LIBERO and SimplerEnv-Bridge benchmarks, the model consistently outperformed existing baselines, including CogACT and standard VLA models. The results suggest that weaving dual-scale latent memory into the reasoning process enhances robustness, particularly when action generation depends heavily on task progress and historical cues.
Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
Sam: Precisely. And to retrieve from those vaults, the system uses a "seeker" module — it looks at the robot's current state and pulls out only what's relevant to the immediate decision. That retrieved information then passes through a "condenser," which compresses it down to a fixed size, and finally a "weaver," which stitches it into the robot's main reasoning sequence.
Alex: If the selection is that aggressive, doesn't it risk losing something important? What if two moments look similar but one of them actually matters?
Sam: The system handles that with what the researchers call a "curator." When a vault reaches its storage limit, rather than simply deleting the oldest entry, the curator calculates how similar neighbouring memories are to each other and merges the most redundant pair. It's a bit like condensing two nearly identical paragraphs into one summary sentence — you keep the gist without keeping every word.
Alex: So it's compressing history, not erasing it.
Sam: Right. And because these compressed memories are stored in the same format the robot uses to process vision and language, the robot can reason about its own past as naturally as it reasons about what it sees in front of it.
Alex: That's a meaningful shift. But I want to push on the limits. If you just keep adding more memory tokens, does the system keep getting better — or is there a point where it starts to struggle?
Sam: There's a clear trade-off. More tokens do help, up to a point — they allow the robot to hold onto finer-grained visual details over longer task horizons. But beyond a certain threshold, performance drops. The system becomes overloaded with redundant information, and the condenser has to work much harder to filter out the noise. The researchers found a balanced default that provides enough history without overwhelming the model's reasoning.
Alex: So it's not about having the biggest possible memory — it's about keeping the active workspace focused on what's actually useful.
Sam: That's the right way to think about it. The goal is to maximise the relevance of the information, not the volume. When they tested this on standard robotic benchmarks in simulation, the approach showed clear improvements over systems that treat memory as an afterthought.
Alex: Since this was all done in simulation, how much should we read into those results? A controlled virtual environment is a long way from a real warehouse.
Sam: That's the primary limitation the authors themselves acknowledge. Simulations are useful because they're controllable and repeatable, but the real world introduces things that are hard to model — changing lighting, physical wear on hardware, unexpected objects in the workspace. The results in simulation are a meaningful signal, but proving the approach holds up on actual robotic arms is the next, considerably harder step.
Alex: So the architecture represents a genuine advance in how we think about robotic memory, but the real test is still ahead.
Sam: That's a fair summary. And if the approach does transfer to physical hardware, the longer-term implication is worth noting: robots that can build up knowledge of a user's preferences, or a specific tool's quirks, over time — without needing to be retrained from scratch. The memory wouldn't just help within a single task. It could accumulate across many.
Alex: That's a genuinely different way of thinking about what a robot can be — less a machine that executes instructions, and more one that learns from its own experience as it goes. Thanks for walking us through it.
Sam: Thanks for listening to ResearchPod.