Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs | Junhao Shi et al. | ResearchPod

Transcript: Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to train robots, and it starts with a problem that's been slowing down the whole field. Sam, what's the core issue?

Sam: We're hitting what researchers call a "data wall." Right now, to teach a robot to do something useful, a human has to manually guide it through every single movement—like using a remote control—while also providing spoken or written instructions. That process is slow, expensive, and almost impossible to scale up to the thousands of tasks we'd want a robot to handle.

Alex: So this paper is basically asking: can we get around the need for all that human-guided "homework"?

Sam: Exactly. And the researchers argue we've been making a mistake by trying to teach two very different things at the same time. Think of it this way: a robot needs to learn how to move its arm and interact with objects—that's the physical side. But it also needs to learn what a human actually wants it to do—that's the reasoning side. The paper argues these two goals should be trained separately.

Alex: That's like the difference between learning how to hold a pen versus learning how to write a specific sentence. The grip is just mechanics. The sentence requires understanding.

Sam: That's a perfect way to put it. And here's the key insight: only the second part—the "what to do" part—actually needs a human to label the data. The "how to move" part can be learned from simple, messy, unlabeled footage. The researchers call this approach "Task-Agnostic Pretraining," or TAP.

Alex: If the robot doesn't need a human guiding it, how does it actually learn to move?

Sam: They use a method called Inverse Dynamics. Imagine a baby flailing its arms around—not because anyone told it to, but just to figure out what its body can do. The robot does something similar. It watches video recordings of itself moving and tries to answer one question: "Given where I started and where I ended up, what action must I have taken?" By doing this over and over, it starts to build an internal model of how the physical world works.

Alex: So it's playing a game of "guess the move" based on the outcome?

Sam: Exactly. And through that game, the robot learns what physicists would call "physical priors"—basically, a foundation of common sense. It learns how objects tend to move, how to grip things without dropping them, how much force is needed. All of that, before it ever receives a single instruction from a human.

Alex: And the data for this is just sitting around in most robotics labs?

Sam: It is. Most labs accumulate thousands of hours of video from experiments that "failed" or were just exploratory—robots moving randomly, bumping into things. That footage usually gets discarded. This paper argues it's actually valuable training material, precisely because it captures the messy, unpredictable reality of physical interaction.

Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Junhao Shi, Siyin Wang, Xiaopeng Yu, Li Ji, Jingjing Gong, Xipeng Qiu

Jul 2, 2026·5:38·arxiv·Original Paper

Vision-Language-Action (VLA) modelsTask-Agnostic Pretraining (TAP)Decomposition HypothesisInverse DynamicsEmbodied AIVision-Language-Action (VLA) Models

Listen

6 min

Summary

Research Question

Vision-Language-Action (VLA) models are currently limited by the high cost of collecting expert demonstrations—triplets of observations, instructions, and actions. The authors investigate whether robots can learn fundamental physical competence (how to move) independently of semantic task instructions (what to do), thereby bypassing the need for massive, human-annotated datasets.

Approach

The authors propose Task-Agnostic Pretraining (TAP), a two-stage framework based on the Decomposition Hypothesis. In the first stage, the model learns physical priors from abundant, unlabeled interaction data—such as discarded off-task trajectories or autonomous robot play—using a self-supervised Inverse Dynamics objective. This objective forces the model to predict the action required to transition between two observations, effectively teaching the robot to understand object affordances and kinematics without language labels. In the second stage, the model is fine-tuned on a minimal set of expert demonstrations to align these physical priors with high-level semantic instructions.

Main Findings

TAP demonstrates that physical grounding can be effectively decoupled from semantic intent. On the SIMPLER benchmark, TAP matches the performance of models trained on over 1 million expert trajectories while using significantly less labeled data, achieving a 10% absolute gain over standard behavior cloning. In real-world experiments on a WidowX platform, TAP shows superior robustness, retaining 25% success under severe camera perturbations where internet-scale baselines collapse to 0%. The authors show that pretraining on task-agnostic data acts as a regularizer, preventing the model from overfitting to spurious visual correlations and enabling it to focus on causal interactive dynamics.

Why It Matters

This work challenges the prevailing paradigm that scaling expert human demonstrations is the only path to general-purpose embodied AI. By demonstrating that robots can learn from autonomous, curiosity-driven exploration, the authors provide a scalable, cost-effective foundation for building more capable and robust robotic agents that can operate in diverse, unstructured environments.

Key Terms and Definitions

Vision-Language-Action (VLA) Model — A neural network architecture that integrates visual perception, language instructions, and robotic action generation into a unified framework.
Inverse Dynamics Objective — A self-supervised training task where the model learns to predict the action taken to transition between two observed states, focusing on physical motion rather than task goals.
Decomposition Hypothesis — The core insight that robotic learning can be split into two stages: learning physical competence (how to move) and semantic alignment (what to do).
Task-Agnostic Data — Interaction trajectories that lack explicit task labels or language instructions, such as random robot play or discarded data from unrelated tasks.
Behavior Cloning — A supervised learning approach where a robot policy is trained to mimic expert actions provided in a dataset of demonstrations.

Abstract

Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiring physical competence (how to move) and acquiring semantic alignment (what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we propose Task-Agnostic Pretraining (TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via a self-supervised Inverse Dynamics objective. A lightweight second stage then grounds these priors in language using minimal expert data. On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path forward for Embodied AI.

Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs: Research Paper Podcast