Dongyoung Kim, Huiwon Jang, Myungkyu Koo, Suhyeok Jang, Taeyoung Kim, Beomjun Kim, Byungjun Yoon, Changsung Jang, Daewon Choi, Dongsu Han, Donguk Lee, Heeseung Kwon, Hojin Jeon, Jaehyun Kang, Jaekyoung Bae, Jihyuk Lee, Jimin Lee, John Won, Joonwoo Ahn, Junhyeong Park, Junyoung Sung, Kyungmin Lee, Minseong Han, Minsung Yoon, Sejune Joo, Seonil Son, Seungcheol Park, Seunggeun Cho, Seungjun Moon, Seungku Kim, Yonghoon Dong, Yongjin Cho, Youngchan Kim, Jinwoo Shin
8 min
RLDX-1 is a general-purpose Vision-Language-Action (VLA) model designed to bridge the gap between versatile intelligence (scene and language understanding) and the functional requirements of real-world dexterous manipulation. While modern VLAs excel at instruction following and general reasoning, they often falter in scenarios requiring temporal dynamics (motion awareness), multi-step reasoning (long-term memory), or contact-rich interaction (physical sensing). RLDX-1 addresses these deficits through a unified architecture, a synthetic data pipeline for rare dexterous tasks, and optimized inference for real-time deployment.
The core of the model is the Multi-Stream Action Transformer (MSAT), which extends multi-modal diffusion modeling to robotic action generation. The architecture processes different modalities (cognition, proprioception, and physics) through dedicated, modality-specific streams, allowing them to remain distinct while enabling cross-modal interaction through joint self-attention. To support specific functional capabilities, the system incorporates three distinct modules: a motion-aware vision encoder that uses space-time self-similarity, a memory module that caches historical cognition tokens for long-horizon planning, and a physics stream that integrates torque and tactile signals for contact-sensitive control.
To overcome the data scarcity associated with high-degree-of-freedom humanoid robots, the researchers developed a synthetic data pipeline. This pipeline uses video generative models and inverse dynamics to augment demonstration datasets, followed by a motion-consistency filter that uses a learned classifier to discard physically implausible trajectories. The training procedure itself follows a three-stage approach: large-scale pre-training for general priors, mid-training for embodiment-specific functional capabilities, and task-specific post-training. Finally, the researchers optimized real-time performance by converting the model into a static computation graph and hand-designing fused CUDA kernels to eliminate redundant memory traffic and launch overheads.
RLDX-1 demonstrates that robotic intelligence requires more than just scaling language-based models. By explicitly modeling physical, temporal, and historical context, the researchers achieved significant performance leaps on challenging benchmarks like ALLEX humanoid manipulation and contact-intensive tasks such as peg insertion and egg handling. This work suggests that integrating specialized functional streams into a versatile foundation model is a viable pathway toward reliable real-world robotics.
While Vision-Language-Action models (VLAs) have shown remarkable progress toward human-like generalist robotic policies through the versatile intelligence (i.e. broad scene understanding and language-conditioned generalization) inherited from pre-trained Vision-Language Models, they still struggle with complex real-world tasks requiring broader functional capabilities (e.g. motion awareness, long-term memory, and physical sensing). To address this, we introduce RLDX-1, a general-purpose robotic policy for dexterous manipulation built on the Multi-Stream Action Transformer (MSAT), an architecture that unifies these capabilities by integrating heterogeneous modalities through modality-specific streams with cross-modal joint self-attention. RLDX-1 further combines this architecture with system-level design choices, including data synthesis for rare manipulation scenarios, learning procedures specialized for human-like manipulation, and inference optimizations for real-time deployment. Through empirical evaluation, we show that RLDX-1 consistently outperforms recent frontier VLAs (e.g. $π_{0.5}$ and GR00T N1.6) across both simulation benchmarks and real-world tasks that require broad functional capabilities beyond general versatility. In particular, RLDX-1 shows superiority in ALLEX humanoid tasks by achieving success rates of 86.8% while $π_{0.5}$ and GR00T N1.6 achieve around 40%, highlighting the ability of RLDX-1 to control a high-DoF humanoid robot under diverse functional demands. Together, these results position RLDX-1 as a promising step toward reliable VLAs for complex, contact-rich, and dynamic real-world dexterous manipulation.
Sam: We’ve covered the action generator. How does the information reaching it become sensitive to movement and history?
Alex: For movement, the model receives several recent video frames instead of only a current image. An added module compares visual features across space and time to capture motion. The system then compresses older visual information to reduce computation.
Sam: Recent frames only cover a short interval. Remembering what went into a box needs information from further back.
Alex: There’s a separate memory module for that. It caches compact features extracted from earlier observations and combines them with the current features. In Object-in-Box Selection, a person puts an object inside one of several boxes, then returns the box. The robot must later choose using what it saw before.
Sam: So the same current scene can demand different actions depending on its history. And physical sensing supplies another kind of missing information?
Alex: Consider their pouring task. The robot pours balls from a pot into a cup, stopping at a target weight. Visual changes offer limited guidance. Joint torque—the twisting force at a joint—provides information about the changing load.
Sam: That architecture needs training examples covering very different kinds of information. Where do those come from?
Alex: The training combines public robot demonstrations, in-house demonstrations, and synthetic videos. Broad pretraining teaches manipulation across robot types. An intermediate training stage specializes the model for particular hardware and adds capabilities such as memory and physical sensing. Task-specific training then refines deployment behavior.
Sam: Generated videos can look plausible while depicting actions a real robot couldn’t reproduce. How do they address that gap?
Alex: An inverse dynamics model predicts actions from current and future observations. The authors replay those predicted actions in a simulator. A learned classifier compares the replay with the generated video and rejects inconsistent samples. A separate filter checks visual plausibility and instruction following.
Sam: So they check both the appearance of the demonstration and whether its action labels match the motion. What evidence suggests that extra data helps?
Alex: In a controlled humanoid benchmark experiment, they hold the real dataset fixed and increase the amount of filtered synthetic data. Performance improves as synthetic coverage grows. That supports the pipeline as a useful complement to real demonstrations, not a replacement for sensor-equipped data.
Sam: Let’s get to the main results. How large is the advantage on the humanoid tasks?
Alex: Across the ALLEX tasks, the reported average is about eighty-seven percent. The comparison models, pi-zero-point-five and GR00T N1.6, average around forty percent. But that headline combines different kinds of task scores.
Sam: If some scores reward partial completion, that average wouldn’t mean the robot finishes nearly every task. What does it actually measure?
Alex: Conveyor handling and box selection use success rates. Card handling and pouring award partial credit for completed stages. So I’d call it a large advantage on this benchmark, not an overall probability of completing the full tasks.
Sam: Is there an individual result that captures the benefit without that scoring complication?
Alex: Box selection is a clean example. RLDX-1 chooses correctly about ninety-two percent of the time, while the comparison models perform around chance. That is a direct test of using information no longer visible in the current scene.
Sam: These tasks were designed around the proposed capabilities. Does the advantage extend beyond that particular humanoid?
Alex: The report also evaluates a Franka single-arm robot, including motion recognition, memory tasks, and contact-sensitive manipulation. RLDX-1 leads the comparisons across those tasks too. Those evaluations generally use twenty-four trials per policy per task, which limits how precisely I’d interpret individual rates.
Sam: Leading a comparison still leaves the question of practical reliability. Where does the system remain weak?
Alex: Plug insertion succeeds only about a third of the time, despite outperforming both comparison models. The socket is completely hidden from the cameras, making contact feedback especially relevant. Cup swapping also remains difficult. These results show improvement, not dependable mastery.
Sam: And are we seeing a general model dropped into unfamiliar tasks, or a system trained for each evaluation?
Alex: These are task-fine-tuned models, including the comparisons. Simulation results also favor RLDX-1 across all reported benchmarks, but those models receive benchmark-specific training. The evidence supports a strong training-and-architecture package. It doesn’t establish arbitrary-task deployment without adaptation.
Sam: A model could understand motion and still react too late. Did they measure the deployment cost?
Alex: They did. On their tested desktop configuration, full-model inference falls from about seventy-one milliseconds to forty-four milliseconds. They capture execution as one reusable computation graph and combine selected operations to reduce overhead and memory traffic. That’s a measured latency improvement, not a guarantee for every robot setup.
Sam: Who should read the full report, and where should they begin?
Alex: Researchers building manipulation policies should start with the ALLEX and Franka Research 3 experiment sections, especially their scoring protocols. Then read the neural architecture and staged training procedure. Deployment engineers should examine the inference strategy. For everyone else: understanding a scene is not the same as having the information needed to act.
Sam: Keep that distinction in mind when the next robot benchmark arrives.