Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised. We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies. Baseline results reveal that Gemini 3.1 Pro, Qwen3-VL and InternVL3.5 models perform only marginally above random chance, and although strategies such as incorporation of skeleton information, grounding instructions, reasoning structures and in-context learning lead to isolated gains, none is consistently effective. Analysis of prediction distributions uncovers two systematic biases: a tendency to predict correct execution regardless of visual evidence, and a sensitivity to superficial linguistic framing. Reformulating tasks contrastively to mitigate these biases yields minimal improvement, suggesting that the models' limitations go beyond these biases, pointing to a fundamental difficulty with fine-grained movement quality assessment. Our findings establish a rigorous baseline for future VLM-based AQA research and provide an actionable outline for failure modes requiring mitigation prior to reliable real-world deployment.
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at a paper called "Can Vision Language Models Judge Action Quality? An Empirical Evaluation," by researchers including Miguel Monte e Freitas from Sword Health. The central question is whether these huge AI systems—called Vision Language Models, or VLMs—that can describe pictures in detail can spot bad form in a workout video, like telling if your squat might wreck your knees.
Alex: So this is basically asking if VLMs, which seem great at everyday vision tasks, fall short when judging movement quality in things like squats or dives?
Sam: Exactly. These models handle both images and text, and they excel at broad tasks like captioning videos. But they struggle with Action Quality Assessment, or AQA—automatically checking if an exercise or athletic move is done right, like knee alignment in a squat for a physical therapist.
Alex: Huh. So even with training on massive data, they miss obvious flaws in form?
Sam: Yes. In fitness videos with clear errors—like knees caving inward, called knee valgus—the models often predict perfect form anyway. That's because they lean on what they "know" from text training about how exercises should look, ignoring the actual video. The paper checks this across gym exercises, figure skating, and diving.
Alex: Okay, that makes sense for therapy or coaching. But what exactly counts as judging action quality here?
Sam: It means watching a video of a move, like a squat, and scoring it or spotting errors based on rules—like keeping knees over toes. VLMs seem promising because you describe the rules in text alongside the video, and they might explain their thinking—no special training needed.
Alex: So they tested VLMs on real AQA tasks without special training—just videos and instructions. But didn't they try ways to make the models better?
Sam: They did, using prompt engineering—different ways to phrase instructions. One adds step-by-step reasoning: describe the pose, check rules, then score. Another shows a few good examples first, like in-context learning. Despite these, performance stayed only a bit above random guessing across fitness, skating, and diving.
Alex: Wait, so even examples or clearer steps didn't fix it much?
Sam: Right. The study found two main biases. First, a correctness bias: models assume moves are right, predicting perfect form even with clear knee cave-ins. Second, wording sways them too much—swap "knees over toes" to "avoid knees caving," and predictions flip, showing language overrides visuals.
Alex: So language tricks them more than the motion. How did they test if that's the real problem?
Sam: They reformulated tasks as comparisons: show two videos, ask which is better. This cuts biases—no absolute "good" judgment or phrasing tricks. Ground truth is the expert's higher score. Even top models like Gemini 3.1 Pro scored just over 50%—barely above random.
Alex: Okay, stripping away biases still leaves them struggling, especially on subtle errors.
Sam: Precisely. Checks of their "thinking" steps show guesses from general knowledge, like "squats usually have parallel trunk," over real analysis. This points to limits in understanding fine-grained movement.
Alex: What about preprocessing videos—like zooming on the body or highlighting joints—to help them focus?
Sam: Preprocessing prepares the input, like cropping to the body or adding stick-figure lines for joints. Or rendering just the skeleton—a wireframe of bones—so motion stands out without distractions like clothes. Across fitness, skating, and dives, it helped a little. Top models showed weak correlations, around 0.1 to 0.2, on score prediction—barely better than random.
Alex: Huh. So even joint diagrams don't capture timing or alignment.
Sam: The full evaluation used five datasets, tasks from visual question answering to scoring, and seven prompting strategies—including forcing step-by-step image checks. Gains were marginal, signaling an inherent issue with visual-motion reasoning.
Alex: Right, tweaks don't bridge the gap. That's a notable limit for coaching tools. But are there caveats—like what it leaves out?
Sam: Yes, it's zero-shot—models used off-the-shelf, without fine-tuning on AQA data. Fine-tuning might help by teaching motion specifics. It covers just three VLM families, so new ones could differ. These mean it highlights current weaknesses, not ruling out future progress.
Alex: Pulling it together, VLMs have broad skills but hit a wall on movement details—like missing a knee cave or poor dive. Tweaks to prompts, inputs, and comparisons barely beat guessing, pointing to a real gap in visual reasoning for action quality.
Sam: The setup across domains, tasks, inputs, and strategies isolates a fundamental bottleneck. For automated coaching, specialized models trained on motion data may be needed. This provides a clear benchmark, urging caution while pointing ahead.
Alex: That's a grounded look at AI's limits in judging movement. Thanks, Sam—thanks for listening to ResearchPod, everyone.
Sam: My pleasure.