P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture

Procedural video understanding—such as cooking, assembly, or repair—requires models to distinguish between visually similar actions that carry different meanings depending on their temporal context. Existing video foundation models often struggle with these long-range dependencies because self-attention scales quadratically with sequence length, making it computationally prohibitive to process long, untrimmed videos. The authors investigate whether a lightweight, self-supervised predictive architecture can learn to organize video features into a procedure-aware space without relying on expensive language models or explicit action labels.

To address the computational bottleneck, the authors introduce P-JEPA (Procedural Joint Embedding Predictive Architecture). Instead of processing raw RGB frames, P-JEPA operates on dense, frame-aligned features extracted from pretrained backbones like V-JEPA 2.1, TSM, or I3D. The model uses a masked latent prediction objective: it hides portions of the procedural sequence and forces the student encoder to predict the missing latent representations from the visible context.

Key technical innovations include:

• Clip-Causal Attention: A specialized attention mask that allows tokens to attend freely within their own segment and to all previous segments, while preventing access to future information. This enables streaming inference while maintaining temporal awareness.
• 2D Rotary Positional Embeddings (RoPE): A dual-coordinate embedding strategy that encodes both the segment index (capturing global progress) and the within-segment index (capturing local frame-wise order).

Experiments across the EgoExo4D, EgoProceL, and Assembly101 datasets demonstrate that P-JEPA significantly improves the linear separability of procedural segments. By reorganizing the feature space around procedural structure, the model achieves state-of-the-art results on fine-grained action classification for EgoExo4D. Notably, P-JEPA achieves these gains while using an order of magnitude fewer parameters than current VideoLLM-based methods and operates in real-time, proving that dense latent prediction is a highly efficient alternative for long-form temporal reasoning.