Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment

Video aesthetic assessment (VAA) is a challenging task due to the inherent subjectivity of human perception and the scarcity of large-scale, richly annotated datasets. Existing methods often treat VAA as a generic regression problem, failing to account for the cognitive mechanisms that shape how humans evaluate temporal experiences. This paper investigates whether incorporating psychological principles—specifically the peak-end rule—can lead to more accurate, interpretable, and generalizable aesthetic assessments.

The authors propose Peak-End-Net, a lightweight framework that models video aesthetics by focusing on key temporal moments. The approach consists of three main components:

1. Frame Aesthetic Perceiver (FAP): Using a frozen Vision Transformer (ViT) and an image aesthetic assessment (IAA) head pretrained on the AVA dataset, the model generates frame-wise aesthetic priors. This allows the system to identify aesthetically salient moments without requiring frame-level video annotations.
2. Peak-End Temporal Aggregation: Inspired by the psychological peak-end rule, the model assigns non-uniform weights to frames based on their aesthetic saliency (peaks and valleys) and their proximity to the end of the video. This creates a holistic video representation that mimics human memory of an experience.
3. Aesthetic Rhythm Encoding & Gated Fusion: To capture how aesthetics evolve over time, the model includes a rhythm encoder that models temporal variation. Finally, a dynamic gated fusion mechanism adaptively combines the VAA-predicted score with a global IAA estimate, improving robustness against cross-dataset distribution shifts.

Peak-End-Net achieves state-of-the-art performance on major VAA benchmarks, including VADB and DIVIDE-3K. The results demonstrate that explicitly modeling the peak-end rule and aesthetic rhythm patterns provides a more effective and interpretable way to aggregate temporal information than uniform averaging. By leveraging pretrained image-based knowledge, the framework remains parameter-efficient and generalizes well to new datasets, confirming that psychologically grounded modeling is a powerful strategy for subjective visual tasks.