video-SALMONN-R3: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

Video-LLMs often struggle with the trade-off between computational efficiency and the need for fine-grained visual detail. While many models use sparse frame sampling to manage memory, this often leads to missing critical information. The authors investigate whether a two-stage 're-watch' paradigm—where a model first performs a coarse global scan and then revisits specific segments at higher fidelity—can be implemented efficiently without the high cost and potential performance degradation associated with traditional chain-of-thought (CoT) supervised fine-tuning.

The authors introduce video-SALMONN-R3, an end-to-end model that learns to re-watch through reinforcement learning (RL) rather than CoT-based cold-start. The workflow consists of three key mechanisms:

• Re-watch: The model performs a coarse first pass to identify relevant segments, then re-processes those segments at higher temporal and spatial resolution.
• Re-answer: To prevent the reasoning-heavy re-watch process from interfering with the model's pretrained 'answer-first' tendencies, the model generates an initial answer during the first pass and then refines it after the second pass.
• Re-ask: The model re-injects the original query during the second pass, ensuring that the question tokens can directly attend to the high-fidelity visual and audio evidence observed during the re-watch.

The model is trained in three stages: audio alignment, audio-visual captioning via SFT, and finally, end-to-end RL using a dynamic sampling policy optimization (DAPO) objective with rule-based rewards.