Orca: The World is in Your Mind

Traditional foundation models are typically optimized for specific outputs—such as next-token prediction for language, next-frame prediction for video, or next-action prediction for robotics. The Orca team argues that these isolated approaches limit general intelligence. Instead, they propose that a true world foundation model should learn a unified latent representation of world states, which can then be read out to perform diverse tasks like reasoning, visual prediction, and physical manipulation.

Orca introduces a "Next-State-Prediction" paradigm. The model uses an encoder-decoder architecture where the encoder is trained to map multimodal signals (vision and language) into a unified world latent space. This learning occurs through two complementary paradigms:

1. Unconscious Learning: The model observes continuous video data to learn dense, natural state transitions (e.g., how objects move or scenes evolve) without explicit labels.
2. Conscious Learning: The model uses language instructions and event annotations to learn sparse, meaningful state transitions, effectively aligning its latent space with human-defined intentions and causal logic.

After pre-training, the backbone is frozen, and only lightweight, modality-specific decoders are trained to read out the latent space into text, images, or robot actions.

Experiments demonstrate that Orca’s learning paradigm is highly scalable; as the model size and data volume increase, the total loss decreases, and downstream performance improves consistently. Notably, despite not being trained on robot action labels, Orca’s learned world latent enables it to generate effective action trajectories in real-world settings. It outperforms similar-sized specialized baselines in text generation, image prediction, and embodied manipulation, showing that a stronger world latent directly translates to better performance across diverse downstream interfaces.