Ruslan Rakhimov, George Bredis, Yuriy Maksyuta, Daniil Gavrilov
9 min
Existing JEPA world models for visuomotor control typically commit to a single inference paradigm at training time, such as trajectory optimization (planning) or direct behavior cloning. This forces researchers to choose between deployment strategies before training. Qantara addresses this by asking whether a single, sub-billion-parameter JEPA checkpoint can support multiple inference paradigms—planning, behavior cloning, and inverse dynamics—simultaneously without requiring task-specific retraining.
Qantara introduces a joint training objective that treats the state and action axes differently to accommodate the requirements of different inference modes. On the state axis, it uses a Brownian-bridge interpolant between consecutive clean latents, which aligns with the previous latent state used during planning. On the action axis, it employs noise-to-data flow matching. To ensure the model performs well across all paradigms, the authors use edge-aligned sampling of noise levels during training. This concentrates the model's capacity on the specific regions of the noise-time square that are queried during inference (the edges and corners), rather than wasting capacity on the interior of the square.
Qantara achieves state-of-the-art performance on the OGBench-Cube benchmark and performs competitively on the LeWM control suite. Crucially, a single checkpoint successfully serves three inference paradigms: latent planning (using a goal-conditioned search), behavior-cloning action sampling, and a video-inverse composition. The authors find that the multi-paradigm capability is not just a matter of coverage; there is a significant co-regularization effect where training on multiple modes stabilizes the model's performance even for inference paths that do not explicitly query those modes. Replacing the edge-aligned sampling with uniform interior sampling leads to a substantial drop in success rates, highlighting the importance of matching training distribution to inference queries.
This work shifts the paradigm for JEPA world models from single-purpose planners to flexible, multi-paradigm controllers. By deferring the choice of inference strategy to deployment time, Qantara allows for greater flexibility in adapting to varying constraints—such as latency requirements or the availability of goal observations—without the need for expensive retraining or maintaining multiple model checkpoints.
Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels, but every existing JEPA world model commits at training time to a single inference paradigm: either trajectory optimisation in a learned dynamics model, or direct behaviour cloning. A single checkpoint that serves both would defer this choice to inference, when deployment constraints (rollout cost, observation accessibility) determine which path wins. We present Qantara, an end-to-end JEPA whose joint training objective pairs a Brownian-bridge interpolant between consecutive clean latents on the state axis with noise-to-data flow matching on the action axis. The same checkpoint serves three inference paradigms without retraining: latent planning, behaviour-cloning action sampling, and inverse dynamics, which we query through a video-inverse composition that first predicts the next latent without action conditioning, then extracts the action. Training concentrates mass on the edges of the (action-time, state-time) noise square, where inference queries the predictor: replacing it with uniform interior sampling drops Push-T planning from 90.1 to 53.3 SR at matched compute. On the LeWM control suite, Qantara reaches a 91.2 SR three-train-seed average and sets new SOTA on OGBench-Cube (+7.7 SR over DINO-WM, +19.7 over LeWM). From the same weights, the behaviour-cloning and video-inverse paths reach 82-83 SR on Push-T and 71-73 SR on Cube. These results move JEPA world models from single-paradigm planners to multi-paradigm controllers.
Alex: Those sound like different questions, with different information supplied. How does one training objective prepare the model for all of them?
Sam: That is the central idea: train on the combinations of known and unknown information that deployment will use. Sometimes the action is known and the next state is missing. Sometimes the transition is supplied and the action is missing.
Alex: So training only the forward prediction problem would leave the reverse question underprepared.
Sam: That is the authors' argument. They represent uncertainty separately for actions and states. Imagine a square, with action uncertainty along one direction and state uncertainty along the other. The deployed routines query its edges, so training concentrates there.
Alex: Why not spread training across the whole square? Wouldn't broader coverage make the model more flexible?
Sam: It would spend most training on combinations those routines never use. The authors also train along a diagonal, where actions and states become cleaner together. That diagonal helps stabilise learning, even though their default deployment routes do not use it.
Alex: There is also a bridge in the title. What does that contribute beyond choosing where to train?
Sam: Actions start as random noise, and the model learns to turn that noise into plausible actions. States start somewhere more useful: the current state's representation. A Brownian bridge, a noisy connection anchored at both endpoints, links it to the next state during training.
Alex: So the model does not have to invent a scene representation from arbitrary noise. It learns a transition from the scene it already has.
Sam: Yes. For example, during planning, the action slot contains a candidate action and the state slot starts at the current representation. The model predicts the resulting change. Training and deployment therefore begin from the same kind of state input.
Alex: Let's turn to evidence. Does this flexibility preserve useful control performance, or merely make all the routes technically callable?
Sam: The clearest planning result is simulated cube pick-and-place. Qantara reaches about ninety-four percent success, compared with eighty-six percent for DINO-WM, the prior best JEPA world-model result reported there. It also reaches the ceiling on the navigation task.
Alex: Is that a clean head-to-head comparison, or are published results coming from different evaluation pipelines?
Sam: The DINO-WM result comes from the earlier LeWM evaluation. The authors also retrain LeWM inside their own pipeline. Qantara beats that reproduction on most environments, but trails it on Push-T, where a controller pushes a T-shaped object into position.
Alex: And the published LeWM numbers do not line up perfectly with their reproduction?
Sam: They do not, especially for the precision-reaching task. That makes the published comparisons less clean than an entirely shared rerun. Still, each Qantara result averages three independent training runs, and the evaluation includes repeated episodes and evaluation seeds.
Alex: Which experiment most directly tests the training idea, rather than the whole collection of design choices?
Sam: Replacing edge-focused training with uniform coverage, while matching computation, drops Push-T planning success from about ninety percent to fifty-three percent. That supports concentrating training where this planner actually queries the model.
Alex: Does the diagonal really matter if it is never used at deployment?
Sam: Removing it destabilises Push-T training, including the planning route. Removing the policy or inverse-dynamics training mode also destabilises that task. The authors interpret this as the training modes helping stabilise their shared network, beyond simply covering separate questions.
Alex: But that does not establish every training mode as universally necessary.
Sam: No. Removing the dedicated action-free prediction mode preserves all the Push-T routes fairly well. The other environments are also less sensitive to several removals. The strongest stability claim is specific to Push-T under this recipe.
Alex: Now the deployment trade-off. What do the cheaper routes buy, and what do they give up?
Sam: They run roughly fifteen to sixty-five times faster than planning at the paper's defaults. They retain substantial success on Push-T and cube manipulation. But unlike planning, neither receives a separate encoded goal.
Alex: Then are they solving the same problem? If the target is missing, imitation could look capable only because the evaluation makes the continuation predictable.
Sam: That is an essential qualification. Push-T displays its target in the image. For navigation and cube manipulation, evaluation starts within a goal-reaching demonstration window. Following the expert's continuation can therefore succeed when the task tolerates imitation drift.
Alex: And precise targets should expose that dependence.
Sam: They do. On Reacher, where success requires matching a particular joint configuration, the imitation routes fall to near-noise performance. Planning remains useful because it receives the goal. The routes are alternatives, not interchangeable guarantees.
Alex: Given those boundaries, who should read the full document, and where should they start?
Sam: Researchers building visual world models or deployment-flexible controllers should read it. Start with the inference section, then the same-checkpoint comparison and the sampling ablations. Keep the main boundary in view: these are simulation results using near-expert offline data, not real-robot validation.
Alex: And what should someone outside that area remember?
Sam: A model becomes more useful when training prepares it for the questions deployment will actually ask. Qantara shows that one compact controller can support different ways of choosing actions, but missing goal information still matters.
Alex: Thanks for listening.