The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models

Research Question and Approach

Joint-Embedding Predictive Architectures (JEPAs) are widely used for learning latent world models, but they are typically justified by empirical performance rather than a normative framework. This paper addresses the theoretical gap between JEPAs and Active Inference (AIF), a normative theory of intelligence based on the Free Energy Principle. The authors analyze how the choice of anti-collapse regularizer—used to prevent representational collapse in deterministic encoders—determines whether a JEPA's training objective functions as a valid variational free energy.

The Entropy-Estimator Hierarchy

The authors organize common non-contrastive regularizers (VICReg, LogDet, PairDist, and SIGReg) into a hierarchy based on how they estimate latent entropy. They define a 'prior-miscalibration gap' that measures the difference between the true latent entropy and the proxy used by the regularizer. The study demonstrates that VICReg and LogDet act as unsafe upper bounds on entropy, while PairDist acts as a safe lower bound. SIGReg is identified as the only estimator that is simultaneously safe and exact, effectively eliminating the miscalibration gap.

The SIGReg–AIF Correspondence

The central contribution is a correspondence theorem proving that under the standard constant-noise encoder model, the SIGReg objective allows the JEPA training loss to be interpreted as an exact Information Bottleneck decomposition. This preserves the AIF surprise bound and makes the latent goal cost a precise proxy for AIF pragmatic value. The authors extend this framework to multi-step expected free energy and ensemble-based epistemic value. Notably, they identify that current JEPA models lack a 'state-epistemic value' term—a future-state coverage signal—which they propose as a theoretical extension for future empirical testing.

This work bridges the gap between the self-supervised learning community and the active inference community. By providing a formal, machine-verified theoretical foundation (using Lean 4), the authors enable researchers to import tools from active inference into JEPA world models. This allows for more principled planning, such as replacing heuristic exploration bonuses with specific, signed prescriptions derived from the AIF framework.