VIA: VISUAL INTERFACE AGENT FOR ROBOT CONTROL

Can general-purpose foundation models (FMs) control a robot manipulator without specialized training? The authors investigate whether the reasoning and visual capabilities of frontier agents—typically used for coding or computer use—can be repurposed for robotics by treating the robot as a software interface.

Instead of fine-tuning models into vision-language-action (VLA) policies, the authors introduce VIA (Visual Interface Agent). VIA presents the robot environment to an agent through a browser-based 3D interface. This interface provides a point-cloud reconstruction of the scene, along with third-person and wrist-camera feeds. The agent interacts with this environment using a set of Model Context Protocol (MCP) tools that allow it to "teleport" a virtual target gripper, rotate it, and execute movements. The agent operates in a closed-loop: it takes a screenshot, reasons about the scene, issues a command, observes the outcome, and adjusts its plan accordingly.

VIA enables off-the-shelf agents like Claude Code and Codex to perform complex tabletop manipulation tasks zero-shot. The system achieves high success rates, including 96.7% on LIBERO-Goal tasks and 100% on a long-horizon rainbow block assembly task using the Fable 5 model. The authors find that performance scales with the underlying model's strength, suggesting that as general-purpose reasoning agents improve, their ability to control physical robots improves automatically. Because the agent receives no privileged state information and relies solely on visual input, it demonstrates robust error recovery and planning capabilities similar to those seen in computer-use benchmarks.

This work challenges the necessity of robot-specific fine-tuning, which is often compute-intensive and prone to degrading the general reasoning capabilities of foundation models. By framing robot control as a visual tool-use task, the authors show that robotics can directly benefit from the rapid progress in general-purpose AI agents. This approach lowers the barrier to entry for deploying frontier models in physical environments, as it requires only a standardized visual interface rather than massive datasets of robot trajectories.