Shuailei Ma, Jiaqi Liao, Xinyang Wang, Jingjing Wang, Chaoran Feng, Zijing Hu, Chong Bao, Zichen Xi, Yuqi Gan, Weisen Wang, Yanhong Zeng, Qin Zhao, Zifan Shi, Wei Wu, Hao Ouyang, Qiuyu Wang, Shangzhan Zhang, Jiahao Shao, Yipengjing Sun, Liangxiao Hu, Lunke Pan, Nan Xue, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Ka Leong Cheng
5 min
Existing video generative models are primarily optimized for aesthetic quality and content creation, often failing to capture the physical realism and interaction dynamics required for embodied intelligence. This paper investigates how to bridge the gap between digital video generation and physical actuation by scaling a video foundation model specifically tailored for robotics.
To address the limitations of dense video models, the authors introduce LingBot-Video, a DiT-based (Diffusion Transformer) paradigm built on three pillars:
The authors demonstrate that their sparse MoE architecture effectively scales to handle complex spatiotemporal dynamics without the prohibitive costs of dense models. Through systematic ablation, they show that fine-grained expert routing—where tokens are routed to a larger pool of smaller experts—outperforms coarse routing, as it reduces parameter-level gradient conflicts and enables better specialization. The resulting model, LingBot-Video, provides a foundation for tasks requiring both high visual fidelity and an intrinsic understanding of physical interaction.
By providing an open-source, MoE-based video foundation model, this work offers a scalable path for using video models as implicit simulators for robotics. It addresses the critical tension between the need for massive, diverse training data and the requirement for physically grounded, efficient inference in embodied systems.
Despite the recent promise in robot control, video generative models suffer from a domain mismatch due to their primary focus on content creation. For example, their design inherently prioritizes visual fidelity and creativity over computational efficiency and physical realism. In this work, we present LingBot-Video, a DiT-based video pretraining paradigm specifically tailored for embodied intelligence. From the architecture perspective, we adopt the Mixture-of-Experts (MoE), instead of dense, framework to achieve a better trade-off between modeling capacity and inference efficiency, and manage to scale it up from scratch. From the data perspective, we construct a data profiling engine that augments standard internet videos with extensive robot-oriented footage, encompassing manipulation, navigation, and egocentric perspectives, to equip the base model with an intrinsic understanding of actions and world dynamics. From the training perspective, we develop a multi-dimensional reward system to enforce the alignment regarding physical rationality and task completion, going beyond standard criteria such as aesthetics, prompt-following, and motion consistency. Comprehensive evaluations validate its performance and efficiency as a video foundation model. We contribute LingBot-Video as the inaugural large-scale, open-source MoE video foundation model to the community, in a pioneering effort to bridge digital creativity and physical actuation.
Sam: So the training data itself is doing some of the physics teaching. But surely that's not enough on its own — you could still end up with a model that produces physically convincing-looking footage that's actually wrong.
Alex: Right, which is why the reward system is the third piece. Instead of just scoring a generated video on how realistic it looks, the system applies multiple criteria simultaneously. If the model produces a sequence where a robot arm moves in a way that would be physically impossible — say, accelerating instantly with no ramp-up, or an object floating after being released — it gets penalized. The model is being graded not on aesthetics, but on whether its imagined physics are coherent.
Sam: That's a meaningful shift. It's like the difference between grading an essay on whether it sounds confident versus whether the argument actually holds up.
Alex: Exactly. And there's a practical engineering piece that ties this together. Video data is extremely heavy — each second of footage contains an enormous amount of information. To handle that efficiently during training, they use what they call token-budgeted packing. Rather than feeding the system one short clip at a time and leaving processing capacity idle between clips, they pack the entire training batch into one continuous stream. The processors stay fully occupied, and nothing goes to waste.
Sam: So the efficiency gains aren't just coming from the sparse expert routing — they're also coming from how the data is physically organized and fed through the system.
Alex: That's the key insight. The architecture, the data curation, and the training logistics are all working together. The paper's argument is essentially that how you structure the learning process matters as much as how large the model is.
Sam: And if this approach holds up, the practical implication is that a robot could rehearse a task — folding laundry, navigating a cluttered room — thousands of times in its own imagination before ever attempting it physically. That's a meaningful capability.
Alex: It is, and the researchers are measured about it. The open question is how well these imagined physics transfer to genuinely novel environments — situations the model has never seen, even indirectly through training data. That gap between simulated rehearsal and real-world performance is where a lot of robotics research still has work to do. But as a framework for connecting video generation to physical reasoning, it's a notable step forward.
Sam: So the contribution here is really threefold: a smarter architecture that scales without exploding in cost, a data strategy that injects physical grounding, and a reward system that enforces physical coherence rather than just visual appeal.
Alex: That's a fair summary. The model is designed to treat the world not as a canvas to paint convincing images on, but as a system of rules to be understood and respected. Whether that understanding proves robust enough for real deployment is the question the field will be watching closely. Thanks for listening to ResearchPod.