NeoteAI Team, Fudan TEAI Team
5 min
Abstract
We present $N_0$-TWAM, a tactile-native world-action model for contact-rich manipulation that predicts both future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, and it shows strong capability on contact-rich tasks. We pre-train $N_0$-TWAM at large scale with visuo-tactile joint training over tactile-rich demonstrations spanning six embodiments and 450 tasks. We use NeoForce, a unified force-based tactile representation, to form a physically grounded contact signal that conditions action generation. To improve long-horizon and multi-stage manipulation, we introduce tactile contact events for task staging and advance through them during execution. For real-time efficiency, we adopt an asymmetric Mixture-of-Transformers architecture that pairs a full-width expert for video prediction with slim experts for downstream action and tactile prediction. Evaluations on both real and simulated benchmarks justify the capabilities of $N_0$-TWAM across a range of contact-rich tasks, and demonstrate the benefit of data scaling for precise tactile and action prediction. In summary, $N_0$-TWAM endows a world-action model with predictive capabilities to foresee vision, touch and action, building a solid foundation for fine-grained manipulation on open contact-rich tasks. The codebase and model checkpoints will be made publicly available to foster further research and development in tactile-enabled robotic manipulation.
Alex: So it's a predict-then-act loop. The robot imagines what the future looks and feels like, then moves accordingly.
Sam: That's the right way to think about it. And there's a clever detail in how it handles the gap between prediction and reality. The system runs two parallel streams at once. One stream carries the robot's prediction — what it expects to feel. The other carries the live sensor data — what it's actually feeling right now. By comparing those two streams in real time, the robot can confirm it's on track. If it's inserting a cable and feels the expected resistance followed by a click at the right moment, it knows the task is going well. If something feels off, it can adjust.
Alex: Does the research show this actually performs better than systems that rely on vision alone?
Sam: The results are notable. N0-TWAM outperformed vision-only systems by a meaningful margin on the tasks tested. The authors also ran what are called ablation studies — essentially, they deliberately removed parts of the system one at a time to see what broke. When they removed the predictive foresight, performance dropped. When they removed the real-time tactile observation, it dropped again. That tells you both components are doing real work — neither is just decorative. For fine-grained physical manipulation, touch isn't a bonus feature. It's load-bearing.
Alex: Are there limits to how far this scales? What happens with longer, more complex tasks?
Sam: That's where the current design shows its constraints. The model is built for speed and precision on short, well-defined movements. But it uses a strict sequential structure — each step builds directly on the last. If the robot misses a tactile cue early in a sequence, that error can compound, making recovery difficult over longer tasks. The authors identify extending the prediction window as a key next step. If the robot could anticipate further ahead in its own movement sequence, it could handle more complex chains of action.
Alex: What about the sensors themselves — is the system locked to one specific type?
Sam: Currently, yes. It's trained on vision-based tactile sensors, which work a bit like a soft gel surface that deforms under pressure and captures that deformation as an image. But the researchers point toward a broader goal: incorporating other types of feedback, like thermal or acoustic sensors. A system trained on a wider variety of inputs would be more flexible — better able to transfer to different robot hands or different materials without being retrained from scratch.
Alex: It's a meaningful shift in how we think about robots. Not just machines that see the world, but ones that can physically interact with it — and feel their way through uncertainty the way we do. Thanks for walking us through this, Sam. And thank you all for listening to ResearchPod.