Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling | Yufan He et al. | ResearchPod

Transcript: Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

Alex: Welcome to another episode of ResearchPod. Sam, what paper caught your eye this time?

Sam: This is about a study called Cosmos-H-Surgical, which tackles training robots to do surgery on their own. The core idea is using a special AI system to turn piles of surgery videos—without any robot movement details—into useful training data for those robots. It's a way to get around the huge shortage of the right kind of data.

Alex: So the paper is basically saying surgical robots can't learn well because we don't have enough videos matched with exactly what the robot arms are doing?

Sam: Yes, that's the key problem. In regular robot tasks like picking up household objects, there are tons of videos paired with precise records of arm positions, speeds, and grips—what researchers call kinematics. But for surgery, those paired sets are tiny, maybe just 5 to 20 examples per task, because filming in operating rooms is restricted by safety rules, privacy, and high costs. Meanwhile, there are huge libraries of surgery videos online, but they lack those movement labels, so robots can't learn from them directly.

Alex: Right, so without that matching data, even big AI models for robots stall out. How does this study plan to bridge that gap with just the unlabeled videos?

Sam: They build what's called a world model—think of it as an AI that watches the start of a surgery video and a text description like "pick up the needle," then predicts realistic future video frames of what happens next. First, they created a dataset called SATA, with expert notes on over 2,000 short surgery clips describing tool touches and body parts. That trains the world model to mimic real surgery motions. The paper suggests this lets them generate new synthetic videos that look and move like the real thing.

Alex: Okay, so it's like simulating driving from dashcam footage alone. But to train the robot, don't you still need to know what the controls are doing in those simulated videos?

Sam: Exactly—and that's where they add another step with an inverse dynamics model. Imagine needing to figure out how to twist your arm joints to touch a spot on a table from just watching the hand move. This model learns to reverse-engineer those joint movements, or pseudo-kinematics, directly from pairs of video frames. Trained on a bit of real robot data first, it labels the synthetic videos with fake-but-realistic actions, creating tons of paired training examples—about ten times more than real demos alone.

Alex: So they mix real data with this generated stuff to teach a robot policy. And the policy is one of those vision-language-action models?

Sam: Yes, like GR00T, a foundation model that takes video and language instructions to output robot controls. Picture an AI that watches a video of a task, reads a simple instruction like "pick up the needle and pass it," then figures out the exact sequence of arm bends, twists, and grips needed to copy it. It's built to handle real-world messiness, like varying lighting or slight changes in setup, by learning from huge piles of examples—researchers call these vision-language-action models, or VLAs for short. The study shows training it this way on a real surgical robot—for tasks like picking up and handing over a needle—leads to notably better performance than real data alone, with lower errors in movement predictions.

Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

Yufan He, Pengfei Guo, Mengya Xu, Zhaoshuo Li, Andriy Myronenko, Dillan Imans, Bingjie Liu, Dongren Yang, Mingxue Gu, Yongnan Ji, Yueming Jin, Ren Zhao, Baiyong Shen, Daguang Xu

Dec 29, 2025·7:07·arxiv·Original Paper

Surgical RoboticsWorld ModelingData ScarcityVision-Language-Action (VLA) ModelsInverse-Dynamics ModelCosmos-H-Surgical

Listen

7 min

Briefing

Abstract

Data scarcity remains a fundamental barrier to achieving fully autonomous surgical robots. While large scale vision language action (VLA) models have shown impressive generalization in household and industrial manipulation by leveraging paired video action data from diverse domains, surgical robotics suffers from the paucity of datasets that include both visual observations and accurate robot kinematics. In contrast, vast corpora of surgical videos exist, but they lack corresponding action labels, preventing direct application of imitation learning or VLA training. In this work, we aim to alleviate this problem by learning policy models from Cosmos-H-Surgical, a world model designed for surgical physical AI. We curated the Surgical Action Text Alignment (SATA) dataset with detailed action description specifically for surgical robots. Then we built Cosmos-H-Surgical based on the most advanced physical AI world model and SATA. It's able to generate diverse, generalizable and realistic surgery videos. We are also the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos, producing synthetic paired video action data. We demonstrate that a surgical VLA policy trained with these augmented data significantly outperforms models trained only on real demonstrations on a real surgical robot platform. Our approach offers a scalable path toward autonomous surgical skill acquisition by leveraging the abundance of unlabeled surgical video and generative world modeling, thus opening the door to generalizable and data efficient surgical robot policies.

Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling: Research Paper Podcast

Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

Listen