Through the PRISM: Preference Representation in Intermediate States of Video Diffusion Models

Traditional video reward models (VRMs) rely on pixel-level evaluation, which requires fully denoising and decoding video latents into the pixel space. This process is computationally expensive and often unreliable, as external models struggle to interpret the noisy, intermediate states of a diffusion process. The authors ask: Can a video diffusion model inherently discriminate human preferences directly from its own noisy latent states, thereby bypassing the need for pixel-level decoding?

To address this, the authors introduce PRISM (Preference Representation in Intermediate States of Diffusion Models). PRISM keeps the generative backbone frozen and adds a lightweight Query-based Aggregation head. This head acts as an information extractor, probing the backbone's intermediate spatio-temporal features to identify preference signals. By operating directly on noisy latents, PRISM avoids the overhead of VAE decoding and maintains high discriminative accuracy even at high noise levels. The model is trained using a Bradley-Terry model with Ties (BTT) loss, allowing it to learn from pairwise human preference data across the entire denoising trajectory.

PRISM achieves state-of-the-art preference accuracy while demonstrating significant noise-robustness. Unlike pixel-based models that collapse when evaluating noisy inputs, PRISM maintains consistent performance throughout the denoising process. This capability enables early-stage Best-of-N sampling, where suboptimal candidates are filtered out at the beginning of the generation process, drastically reducing computational costs. Furthermore, the study reveals a strong positive correlation between a backbone's generative quality and its inherent evaluative power, suggesting that larger, more capable models naturally make better internal evaluators.

This work shifts the paradigm of video evaluation from external, pixel-based assessment to internal, latent-space reasoning. By eliminating the VAE decoding bottleneck, PRISM makes inference-time scaling techniques like Best-of-N sampling computationally feasible for high-resolution video generation. It also paves the way for self-improving video backbones that can guide their own alignment without relying on external, structurally distinct reward models.