Miguel Monte e Freitas, Rui Henriques, Ricardo Rei, Pedro Henrique Martins
5 min
Abstract
Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised. We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies. Baseline results reveal that Gemini 3.1 Pro, Qwen3-VL and InternVL3.5 models perform only marginally above random chance, and although strategies such as incorporation of skeleton information, grounding instructions, reasoning structures and in-context learning lead to isolated gains, none is consistently effective. Analysis of prediction distributions uncovers two systematic biases: a tendency to predict correct execution regardless of visual evidence, and a sensitivity to superficial linguistic framing. Reformulating tasks contrastively to mitigate these biases yields minimal improvement, suggesting that the models' limitations go beyond these biases, pointing to a fundamental difficulty with fine-grained movement quality assessment. Our findings establish a rigorous baseline for future VLM-based AQA research and provide an actionable outline for failure modes requiring mitigation prior to reliable real-world deployment.
Sam: They reformulated tasks as comparisons: show two videos, ask which is better. This cuts biases—no absolute "good" judgment or phrasing tricks. Ground truth is the expert's higher score. Even top models like Gemini 3.1 Pro scored just over 50%—barely above random.
Alex: Okay, stripping away biases still leaves them struggling, especially on subtle errors.
Sam: Precisely. Checks of their "thinking" steps show guesses from general knowledge, like "squats usually have parallel trunk," over real analysis. This points to limits in understanding fine-grained movement.
Alex: What about preprocessing videos—like zooming on the body or highlighting joints—to help them focus?
Sam: Preprocessing prepares the input, like cropping to the body or adding stick-figure lines for joints. Or rendering just the skeleton—a wireframe of bones—so motion stands out without distractions like clothes. Across fitness, skating, and dives, it helped a little. Top models showed weak correlations, around 0.1 to 0.2, on score prediction—barely better than random.
Alex: Huh. So even joint diagrams don't capture timing or alignment.
Sam: The full evaluation used five datasets, tasks from visual question answering to scoring, and seven prompting strategies—including forcing step-by-step image checks. Gains were marginal, signaling an inherent issue with visual-motion reasoning.
Alex: Right, tweaks don't bridge the gap. That's a notable limit for coaching tools. But are there caveats—like what it leaves out?
Sam: Yes, it's zero-shot—models used off-the-shelf, without fine-tuning on AQA data. Fine-tuning might help by teaching motion specifics. It covers just three VLM families, so new ones could differ. These mean it highlights current weaknesses, not ruling out future progress.
Alex: Pulling it together, VLMs have broad skills but hit a wall on movement details—like missing a knee cave or poor dive. Tweaks to prompts, inputs, and comparisons barely beat guessing, pointing to a real gap in visual reasoning for action quality.
Sam: The setup across domains, tasks, inputs, and strategies isolates a fundamental bottleneck. For automated coaching, specialized models trained on motion data may be needed. This provides a clear benchmark, urging caution while pointing ahead.
Alex: That's a grounded look at AI's limits in judging movement. Thanks, Sam—thanks for listening to ResearchPod, everyone.
Sam: My pleasure.