Makoto Yamada
6 min
Abstract
Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.
Alex: So they genuinely cannot see the same information at the same time.
Sam: Right. Because the hidden pieces are always distinct, each student must infer what's missing based only on what it can see. That forces the model to learn the underlying structure of the image rather than memorizing surface-level pixel patterns.
Alex: It's like two people describing the same room while each looks through a different keyhole. They have to reason about the parts they can't see.
Sam: Exactly. And there's a second layer to the design: predictor networks. These are smaller sub-programs that work on top of the encoders. One predictor focuses on aligning the two students' overall understanding of the scene. The other tries to anticipate what the hidden portions look like, based only on what's visible.
Alex: So one checks that they're on the same page about the big picture, and the other tests whether they can fill in the blanks.
Sam: That's it. And to actually teach the model to improve, the researchers use a mathematical tool called KL divergence. Here's the intuition: the model produces two guesses about the hidden content. The first is a "prior" — a solo estimate based on one student's view alone. The second is a "posterior" — a more informed estimate that draws on both students' combined observations. KL divergence measures how far apart those two guesses are. By training the model to close that gap, you're essentially teaching it to extract only the information that holds up across both views.
Alex: So it learns to ignore the random, view-specific noise and keep only the features that are genuinely consistent.
Sam: Exactly. The model is forced to be honest — it can't rely on quirks of one particular angle or lighting condition. It has to prove its understanding by matching the combined perspective. The researchers also stop the gradient through the prior branch during training, which keeps the whole optimization process stable.
Alex: That's a well-constructed system. Each design choice seems to reinforce the others.
Sam: It is coherent. And the results back it up. Compared to older reconstruction-based methods, SiamJEPA reaches comparable accuracy in roughly a quarter of the training time. The efficiency gain isn't marginal — it's the central claim of the paper.
Alex: So where does the research go from here? Are there limitations?
Sam: Several, and the authors are candid about them. First, the study was conducted on relatively small-scale models. We don't yet know whether these efficiency gains hold when you scale the system up to the truly massive models used in industry. Scaling often introduces unexpected behaviors that smaller experiments don't predict.
Alex: So it works well at the sizes tested, but the question of whether it holds at ten or a hundred times the scale is still open.
Sam: Correct. There's also sensitivity to training settings. Finding the right configuration requires considerable trial and error — a bit like fine-tuning a high-performance engine where small adjustments can shift the output significantly. And currently, the framework is limited to still images.
Alex: What about video?
Sam: The team notes that similar designs are being explored for video data. Since video is essentially a sequence of images, the logic of predicting missing parts across time — not just across space — could be even more powerful. If this approach transfers well to video, it could help systems learn to understand movement and causality, not just static scenes.
Alex: So the next meaningful step is testing whether this works across time, not just across the frame of a single image.
Sam: That's the direction. SiamJEPA demonstrates that predicting abstract representations is a more efficient path than reconstructing raw pixels. The paper makes a clear case for that principle, even if the work of scaling and extending it remains ahead.
Alex: A genuinely interesting piece of work. Thanks for walking through it, Sam — and thanks to everyone listening. That's it for today's episode of ResearchPod.