Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno
5 min
Traditional neural dynamics models for aerial robotics often struggle with long-horizon forecasting because they use autoregressive rollouts, where small prediction errors at each step accumulate and lead to instability. The authors ask whether a Joint Embedding Predictive Architecture (JEPA) can learn a latent dynamics model that avoids this error accumulation while remaining efficient enough for real-time quadrotor control.
Instead of predicting future states directly, the authors propose SkyJEPA, which learns to predict future representations in a compact latent space. To bridge the gap between abstract latents and actionable control, they introduce a physics-inspired prober. This prober maps the frozen latent embeddings to physically meaningful states (such as position, velocity, and attitude) using a lightweight kinematic structure. This allows the system to enforce physical constraints and perform sampling-based optimal control on resource-constrained embedded hardware. To overcome the need for extensive real-world data, the authors developed a structured, domain-randomized simulation pipeline to generate diverse training data.
SkyJEPA demonstrates that latent-space dynamics modeling is highly effective for high-frequency aerial control. By predicting in latent space, the model avoids the compounding errors inherent in autoregressive state-prediction models. The integration of a physics-inspired prober ensures that the latent representations remain grounded in physical reality, which is critical for safety and constraint enforcement. Experimental results show that the model achieves accurate long-horizon prediction and robust zero-shot sim-to-real transfer, performing reliably in outdoor flight tests without requiring task-specific fine-tuning.
This work provides a scalable path for deploying learned dynamics models on aerial robots. By decoupling the dynamics representation from specific tasks and avoiding the pitfalls of autoregressive error accumulation, SkyJEPA offers a more generalizable and stable foundation for agile robotics. The ability to train in simulation and transfer to the real world without additional data collection significantly reduces the safety risks and costs associated with developing autonomous aerial systems.
Accurate dynamics models are critical for informed decision-making in robotic systems, particularly for agile aerial vehicles operating under uncertainty. Neural network dynamics models are attractive for capturing complex nonlinear effects, but existing predictive approaches struggle with long-horizon forecasting because their autoregressive rollout mechanism amplifies errors over time. Joint Embedding Predictive Architectures (JEPAs) offer a compelling alternative by modeling dynamics in latent space, yet prior JEPA-style methods for robot navigation have been studied primarily for kinematic-level planning, with limited investigation in high-frequency control. In this work, we introduce the JEPA-style model for real-time quadrotor control. The proposed approach combines a latent dynamics model with a novel physics-inspired prober that maps frozen latents to interpretable state, enabling physically grounded long-horizon prediction. Additionally, we combine the learned model with a sampling-based optimal control solution to take advantage of its predictive capabilities for real-time control on embedded hardware. Finally, to reduce the dependence on expensive and unsafe real-world data collection, we develop a structured pipeline for automated dataset generation. Extensive open-loop and outdoor closed-loop experiments demonstrate accurate prediction, robust zero-shot sim-to-real transfer, and strong generalization across diverse operating conditions.
Alex: And more resilient. Because it's not trying to reconstruct the entire world at every step, it doesn't get thrown off by sensor noise—a sudden gust of wind, a vibration, a brief signal dropout. The abstract map stays coherent even when the raw sensor data gets messy.
Sam: Does that mean the drone can handle genuinely unexpected conditions without needing to be retrained from scratch?
Alex: That is the goal, and it's where the training approach becomes important. The researchers used what they call "domain-randomised simulation"—think of it like training a pilot in a simulator where the weather, the aircraft weight, and the engine response all change constantly and unpredictably. The pilot can't just memorise one set of conditions. They have to learn the underlying rules of flight. SkyJEPA is trained the same way, so when it encounters something new in the real world, it's drawing on those fundamental principles rather than pattern-matching to something it's seen before.
Sam: Garbage in, garbage out, though—how did the researchers check that the simulation data was actually good enough to teach those principles?
Alex: They developed a measure they call a "Trajectory Distribution Quality" score. It's essentially a quality-control check on the training data, asking whether the simulated flight paths are diverse enough to cover the full range of ways a drone might move and react. The study found that as this score improves, the error in the drone's predictions drops meaningfully. There's a clear relationship between data quality and real-world performance.
Sam: Though I'd imagine that relationship has limits. You can't just keep improving the data forever and expect the model to keep getting better.
Alex: The authors acknowledge exactly that. Performance eventually hits a ceiling. Once the data quality is high enough, further gains are constrained by the model's own capacity, or by aspects of real-world physics that are simply difficult to capture accurately in simulation. It's a robust framework, but it's not a complete solution.
Sam: So where does the research go from here?
Alex: The team's stated next step is incorporating onboard cameras—letting the drone navigate using visual information alone, without relying on external sensors or positioning systems. That's the next significant hurdle for truly autonomous flight: operating in cluttered, unpredictable environments using only what the drone itself can see.
Sam: It's a meaningful step forward, then—not a finished product, but a more principled way of thinking about the problem.
Alex: That's a fair summary. The shift from predicting raw physical states to reasoning in abstract representations is what gives SkyJEPA its stability. Whether that approach scales to the full complexity of real-world autonomous flight is something future work will need to answer. Thanks for listening to ResearchPod.