PAIR: BRIDGING PERCEPTION AND ACTION IN VISION–LANGUAGE–ACTION MODELS | Kaixi Feng et al. | ResearchPod

Transcript: PAIR: BRIDGING PERCEPTION AND ACTION IN VISION–LANGUAGE–ACTION MODELS

Sam: Robot models can improve when we explicitly teach them how perception connects to movement. In the tested systems, learning that connection improved simulated manipulation and physical tasks, including pouring water and loading drawers.

Alex: So the contribution is not better eyesight for the robot. It is a better connection between understanding the task and carrying it out?

Sam: That is the claim of PAIR, a method paper available as an arXiv preprint. It is by Kaixi Feng, Guoheng Sun, and Ang Li at the University of Maryland, College Park.

Alex: For researchers building robot policies, that makes the interface itself something to design and test.

Sam: And the physical comparison gives us a reason to look closely: success rose from about half to roughly two-thirds. We will unpack the learned connection, examine the evidence, and distinguish improvements in these systems from evidence of broader robot competence.

Alex: Start with the disconnect. If a robot model already learns from demonstrations, what is missing?

Sam: These are vision–language–action models: robot policies that turn images and instructions into movements. Their perception features describe objects, relationships, and the requested task. But executable control needs an ordered sequence of movements.

Alex: The demonstrations already pair those descriptions with movements. Why would the model need another learning objective?

Sam: The authors argue that many continuous-action policies leave the transition implicit. Training tells the model whether its final action prediction was correct. But the model must discover both which scene information matters and how to organize it for control.

Alex: So this is about making part of that learning problem explicit. How is that different from earlier intermediate representations?

Sam: Earlier approaches expose motion, geometry, or affordances—the interactions an object permits. PAIR does not prescribe one of those physical descriptions. It learns its intermediate representation from the paired observations, instructions, and expert movements.

Alex: There is also prior work on learned action representations. What separates this from simply compressing the movements?

Sam: Compression is only one part. The authors distinguish PAIR from action tokenization methods, which structure how actions themselves are represented or predicted. They also contrast it with LARA, which aligns visual dynamics with intermediate action-processing features during generation. PAIR instead builds perception-derived features that carry action information before action refinement starts.

PAIR: BRIDGING PERCEPTION AND ACTION IN VISION–LANGUAGE–ACTION MODELS

Kaixi Feng, Guoheng Sun, Ang Li

Oct 6, 2026·9:11·arxiv·Original Paper

Vision–language–action (VLA) modelsPerception–action representation gapPAIR frameworkMasked Action AutoencoderBridge TokensPAIR

Listen

9 min

Summary

Understanding the Perception-Action Gap

Vision-Language-Action (VLA) models have become powerful tools for robot control, yet they often face a significant representational hurdle: the semantic gap between visual perception (interpreting the environment and instructions) and action execution (generating precise, temporally ordered control commands). Most models attempt to bridge this gap implicitly, relying on a single final action-prediction loss to teach the policy both what information is relevant and how to format it into executable control tokens. This approach can be inefficient, forcing the model to discover the structure of action from scratch.

The PAIR Framework

PAIR (Perception-Action Intermediate Representation) addresses this by making the transition from perception to action explicit. The authors propose a two-stage training approach:

Learning Action Structure: A Masked Action Autoencoder is trained to compress expert demonstration trajectories into 'Action Latent Tokens.' By training the model to reconstruct action chunks that have been masked or perturbed, it is forced to learn the underlying temporal and control structure of robot motion.
Building the Bridge: A Bridge Module processes visual-language input to extract task-relevant context. These task features are then aligned with the previously learned Action Latent Tokens. The resulting 'Bridge Tokens' represent a fusion of 'what' needs to be done (task semantics) and 'how' it should be executed (action structure).

During inference, the policy uses these Bridge Tokens to prime its own Action Tokens before it begins the refinement process, providing the model with an 'action-ready' starting point.

Why It Matters

PAIR significantly improves performance across various benchmarks, including LIBERO, LIBERO-Plus (which tests zero-shot robustness to changes like lighting or viewpoint), and the CALVIN long-horizon task suite. Crucially, it translates these simulation gains to a real-world UFACTORY robotic arm, where it demonstrably improves success rates on complex, multi-step tasks like drawer loading and cup stacking. By providing a structured, explicit interface between perception and action, PAIR enables robots to generalize better and execute longer sequences of tasks with higher reliability than models that treat this transition as an implicit, end-to-end learning problem.

Key Terms and Definitions

Vision-Language-Action (VLA) Model — A robotic policy that directly predicts continuous motor commands from inputs consisting of visual scenes and natural language instructions.
Action Latent Tokens — A compact representation of robot behavior learned by an autoencoder that captures the necessary temporal and executable structure of expert action sequences.
Bridge Tokens — An intermediate representation formed by aligning task-relevant visual-language features with action structure, designed to prime the model for action generation.
Bridge Injection — The architectural step where learned Bridge Tokens are introduced at the start of the action-decoding process to guide the policy's initial predictions.
Perception-Action Representation Gap — The semantic mismatch between descriptive visual-language scene representations and actionable, time-ordered motor commands.

Abstract

Vision-language-action (VLA) models map visual observations and language instructions to continuous robot actions. This task requires a transition from representations that describe the scene and instruction to representations that support action generation. Many continuous-action VLAs leave this transition implicit and supervise it mainly through the final action-prediction loss. We introduce PAIR, a framework that learns a shared perception-action representation between these two spaces. During training, a Masked Action Autoencoder encodes expert action chunks into horizon-aligned Action Latent Tokens. A Bridge Module extracts task-relevant features from the current visual-language representations. PAIR aligns these features with the Action Latent Tokens to form Bridge Tokens that preserve task information and capture the structure of expert actions. The Bridge Tokens are then projected into the action-token space and injected into the initial Action Tokens, providing an action-ready starting point for Action Expert refinement. At inference, the autoencoder is removed, and the Bridge Tokens are generated only from the current observation and instruction. Experiments on LIBERO, LIBERO-Plus, and CALVIN ABC-D show gains for the evaluated OpenVLA-OFT and VLA-Adapter models. On LIBERO-Plus, PAIR raises VLA-Adapter's success rate from 59.1% to 64.2%. On CALVIN, it increases VLA-Adapter's average completed sequence length from 4.42 to 4.53. Across seven real-world tasks, PAIR raises OpenVLA-OFT's success rate from 51.4% to 65.0%. Representation analyses show that Bridge Tokens retain task information while making continuous-action information accessible before Action Expert refinement. These results support a shared intermediate representation as a useful interface between perception and action in continuous-action VLAs.

PAIR: BRIDGING PERCEPTION AND ACTION IN VISION–LANGUAGE–ACTION MODELS: Research Paper Podcast