BasketEvent: Understanding Who Did What and When in Basketball Videos

Traditional basketball video understanding often treats spatial perception (tracking) and semantic recognition (event classification) as isolated tasks. This paper addresses the need for a unified, player-centric approach that can answer three fundamental questions simultaneously: What event occurred, who was responsible for it, and when did the decisive evidence for that event appear?

The authors introduce BasketEvent, a new dataset curated from NBA broadcasts. Unlike previous benchmarks that provide only video-level labels, BasketEvent grounds event labels to specific players and includes a manually annotated subset of 1,000 samples with precise temporal boundaries for evidence localization.

To process this data, the authors propose PlayNet, a reasoning framework that:

1. Tracks and Identifies: Uses SAM3 and Qwen2.5-VL to track players and the ball, associating visual trajectories with specific player identities from official game rosters.
2. Encodes Trajectories: Extracts visual tokens guided by these trajectories to capture fine-grained player actions.
3. Models Interactions: Employs attention mechanisms to reason about player-player, player-ball, and global court interactions.
4. Aggregates Evidence: Uses a gated pooling module to identify the most discriminative temporal segments within a video clip, allowing the model to pinpoint exactly when an event occurs.

Experiments demonstrate that PlayNet significantly outperforms existing video-level and crop-based baselines. By explicitly modeling the relationships between players and the ball, the framework achieves superior performance in fine-grained event recognition. The results confirm that grounding events in both player identity and temporal intervals provides a more faithful and interpretable representation of basketball gameplay than traditional, non-grounded methods.