Joint-embedding predictive architecture (JEPA) world models predict future embeddings instead of pixels and plan in latent space. Two challenges remain: 2D images leave the metric geometry that manipulation acts on implicit and entangled with appearance, and what these models imagine stays hidden, as they are mostly judged in simulation by short-horizon latent planning. We present JEPA4R, a JEPA world model that observes single-depth-camera point clouds, predicts and plans in their latent space, and acts through a policy head on its frozen latents; we call this 4D point planning. A decoder on the frozen world model shows what it imagines as 4D point clouds, with a mean point error within 1.1 cm over 200 steps. Under the policy-learning protocols of OGBench and CortexBench, JEPA4R leads pixel JEPA models on OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, and ranks first on MetaWorld and Adroit Relocate with 98.4% and 88.0%, above the runner-up, R3M, at 96.0% and 65.3%. Trained only in a digital twin, JEPA4R and its policy heads run zero-shot on the real arm with both a gripper and a dexterous hand.
Sam: A robot can reach a goal in simulation yet learn poorly from the same internal representation. This study finds that giving the model depth-camera geometry improves learned manipulation policies, while making its imagined scenes inspectable.
Alex: So the claim concerns both how a robot learns to act and whether we can inspect its predictions. Whose work is this?
Sam: It’s a method preprint from Jiarui Dong, Zhiheng Liang, Yanjun Li, Zeyu Zhang and colleagues, called JEPA4R. They compare robot models using images with one using depth-camera observations, then demonstrate deployment on a real arm.
Alex: For researchers choosing what a robot should see, that could change both training and evaluation. Let’s unpack the design, the strongest comparison, and what the real-arm demonstrations actually establish.
Sam: Start with the kind of model involved. A joint-embedding predictive architecture, or JEPA, predicts future internal representations rather than complete images. Those compact representations are called latents. Given recent observations and actions, the model predicts how the scene’s latent will change.
Alex: That sounds useful for choosing actions. Why wouldn’t success at reaching a goal tell us whether those representations are good?
Sam: Because the tests ask different questions. A short-horizon planner searches candidate actions for a predicted endpoint near the goal. Policy learning asks whether a separate action-producing network can learn a skill from those representations. The authors cite earlier work where planning worked well but policy learning transferred poorly.
Alex: So being useful to a search procedure doesn’t necessarily make a representation useful to a learned controller. Where does geometry enter?
Sam: Manipulation depends on where surfaces and objects sit in physical space. An image mixes that geometry with appearance and perspective. A point cloud is a collection of surface samples with spatial coordinates. Here, a fixed depth camera supplies the visible surfaces, with color attached.
Alex: Point-cloud robot policies already exist. What makes this more than replacing an image with a different sensor input?
Sam: The combination matters. Earlier point-cloud policies lacked a world model, while many point-cloud JEPAs trained representations without actions. Other models predicted explicit future geometry. JEPA4R instead predicts compact latents, supports learned policies, and separately decodes those latents so the imagined scene becomes visible.
Alex: Walk me through one manipulation example. What does the model actually receive, and what does it produce?
Sam: Take a cube being lifted from a table and put down again. The encoder compresses recent point clouds into latents. The predictor then uses actions to roll those latents forward. A separately trained decoder turns each predicted latent back into a colored point cloud, showing the imagined movement.
Alex: Does the robot need to generate that whole visible scene before deciding what to do? That would be a different computational commitment.
Sam: No. Prediction and planning happen in latent space. The decoder is for visualization, trained afterward while the world model stays frozen. The policy head, a network producing actions, also learns from frozen latents. The decoder does not supply the control signal.
Alex: Then the comparison needs to separate better observations from simply giving one model more capacity or better demonstrations. How closely did they match things?
Sam: Images and point clouds come from the same simulated trajectories and camera viewpoint. States, actions, splits and starting conditions align. For the closest image-based comparison, LeWM, the predictor and training objective stay unchanged. The encoders have matched parameter counts, and evaluated models share the same policy heads.
Alex: That sounds controlled for the closest comparison. Were all the other competitors trained under identical conditions too?
Sam: No. Some image models were trained on the paired data; others used released pretrained weights. The CortexBench comparison also includes published representation results under its established protocol. So there’s a tightly matched comparison inside a broader benchmark comparison, not one identical training setup across every row.
Alex: Let’s hear the result that best captures the practical difference. Where did learning a policy improve?
Sam: On OGBench’s single-cube task, the point-cloud model reached about seventy-eight percent success. The strongest image-based JEPA there, PLDM, reached about forty-seven percent. The point-cloud model also led on the double-cube, triple-cube and Scene tasks.
Alex: Those are related manipulation environments. Is there evidence beyond moving cubes, especially when the robot has a more complicated hand?
Sam: On Adroit Relocate, a simulated dexterous hand picks up a ball and moves it to a target. JEPA4R reached eighty-eight percent success. The runner-up representation, R3M, reached about sixty-five percent. JEPA4R also had the highest average across the tested MetaWorld tasks.
Alex: The paper emphasizes policy learning. What happened when they returned to the conventional short-horizon planning test?
Sam: That’s a crucial qualification. In the supplementary planning comparison, JEPA4R performed on par with LeWM. Their differences stayed within the reported variation for those estimates. So the evidence is not that point clouds improve every use of a world model; the clearest gains concern learned policies.
Alex: That also explains why changing the evaluation matters. What about inspecting the imagined future? A plausible-looking reconstruction alone wouldn’t establish accurate prediction.
Sam: They compare decoded predictions with observed point clouds on OGBench Cube. Across sixty-four episodes, mean point error stayed within one-point-one centimeters. That bound held over two hundred environment steps. The model received initial context, then predicted forward from recorded actions without fresh observations.
Alex: An average over scene points might hide an error in the object the robot actually needs. Did they check anything more specific?
Sam: They also report cube-position error separately, and it is larger at the final horizon. They compare against decoding actual encoded observations and against using another episode’s actions. Those wrong actions produce much larger errors, supporting that the predictions respond meaningfully to the action sequence.
Alex: So this is a useful diagnostic, not a demonstrated safety guarantee. And the decoder itself has learned geometric knowledge behind it.
Sam: The decoder uses a large pretrained reconstruction model, then adapts it to the frozen latents. Alternative reconstruction backbones give similar errors, which strengthens the case that geometry is available in those latents. But the point-error measurement is not a collision test or a real-robot safety evaluation.
Alex: Let’s separate the real robot from the simulation scores. How much transfer did they actually show?
Sam: They trained in a digital twin, a simulation matched to the deployment arm. Without real-world policy training, the arm performed picking and pushing with a gripper, and extraction tasks with a dexterous hand. The supplement shows eight real executions. That demonstrates feasibility, not a large measured reliability rate.
Alex: And the broader limit is whether this travels beyond the particular robot and workspace, rather than just beyond simulation.
Sam: Yes. These are per-task models, with one fixed camera and workspace-focused point clouds. The simulation pipeline uses segmentation to remove background elements. A general model spanning scenes and robots is outside the paper’s scope, and the authors identify scarce diverse robot geometry data as a barrier.
Alex: Given those boundaries, who should read the full document, and where should they start?
Sam: Researchers building robot representations or world models should read “Point Clouds Outperform Pixels,” alongside the supplementary “Planning under LeWM’s setting.” That contrast is the main lesson. Then read “Real World Evaluation” to distinguish transfer demonstrations from benchmark success rates. For everyone else: a model that plans well has not yet proved it can help a robot learn well.
Alex: Keep that distinction in mind when the next robot benchmark crosses your desk.