Why Can’t I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition | Geo Ahn et al. | ResearchPod

Transcript: Why Can’t I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Alex: Welcome to another episode of ResearchPod. Today, we're looking at why AI models struggle with simple tasks like opening a drawer.

Sam: That sounds like a basic problem. Why can't a machine just learn to open a drawer if it already knows how to close one?

Alex: The study identifies a failure mode where models take a mental shortcut. Instead of watching the motion, they just look at the object and guess the most common action associated with it.

Sam: So the model isn't really "seeing" the action? It's just playing a game of statistical association — guessing "closing" because that's what it saw most often?

Alex: That is the heart of it. The researchers argue this happens for two reasons: sparse data and asymmetric learning difficulty. Let me unpack both of those.

Sam: Please do.

Alex: Think of a student learning a sport. Identifying the ball is easy — you can do it from a single photo. But understanding the timing of a swing requires watching the whole movement unfold over time. The model has the same problem. It finds it much easier to spot the object — the drawer — than to decode the motion happening around it. So it leans on what it can see clearly and ignores the rest.

Sam: And the sparse data problem makes that worse?

Alex: Right. If most of your training videos show people closing drawers, the model builds a strong mental link between "drawer" and "closing." When it finally sees someone opening a drawer, it's already biased. It sees the object, ignores the movement, and defaults to the familiar answer.

Sam: So how does the RCORE framework actually fix that?

Alex: It uses two main tools. The first is called Co-occurrence Prior Regularization. The idea is to take those familiar object-action pairs — "drawer" and "closing" — and treat them as deliberate traps during training. The model is shown those pairs and told, essentially: "Don't just pick the most common answer. You have to actually look at what's happening."

Sam: Like giving a student a test where all the obvious answers are wrong, so they're forced to think more carefully.

Alex: Exactly. The second tool is Temporal Order Regularization. This one targets the time problem directly. The model is trained to tell the difference between a video played forward and the same video played in reverse. If you can't tell opening from closing, you'll fail that test. It forces the model to actually track the sequence of events, not just the objects in the frame.

Sam: So by making the model prove it understands the arrow of time, it can no longer coast on recognising familiar objects. It has to genuinely understand what's happening.

Why Can’t I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Jinwoo Choi

Jan 22, 2026·3:23·arxiv·Original Paper

Zero-Shot Compositional Action Recognition (ZS-CAR)Object-Driven ShortcutsCo-occurrence Prior Regularization (CPR)Temporal Order Regularization for Composition (TORC)Compositional GeneralizationObject-driven Shortcuts

Listen

3 min

Briefing

Research Question

Zero-Shot Compositional Action Recognition (ZS-CAR) requires models to recognize novel combinations of verbs and objects (e.g., 'opening a drawer') that were never seen together during training. A persistent failure mode is that models learn 'object-driven shortcuts,' where the presence of a familiar object (like a drawer) triggers a prediction for a frequent verb (like 'closing') regardless of the actual temporal motion in the video. This paper investigates why these shortcuts occur and how to force models to rely on temporal cues instead.

Approach

The authors diagnose the problem by analyzing two root causes: sparse compositional supervision (skewed training data) and the inherent difficulty gap where objects are easier to recognize than verbs. They introduce two diagnostic metrics—False Seen Prediction (FSP) and False Co-occurrence Prediction (FCP)—to quantify how often models collapse unseen inputs into frequent, seen training pairs.

To address these issues, they propose Robust COmpositional REpresentations (RCORE), which consists of two main components:

Co-occurrence Prior Regularization (CPR): This expands the training signal by synthesizing unseen verb-object pairs and treating frequently seen pairs as 'hard negatives' to prevent the model from overfitting to common co-occurrence patterns.
Temporal Order Regularization for Composition (TORC): This forces the model to be sensitive to the temporal order of actions, ensuring that the verb representation is grounded in motion rather than static object appearance.

Main Findings

The study demonstrates that current state-of-the-art models, including those using advanced video-pretrained backbones, exhibit significant shortcut reliance. The diagnostic metrics reveal that when models fail on unseen compositions, they frequently 'collapse' to the most common verb associated with the detected object. By applying RCORE, the authors show a consistent reduction in these shortcut diagnostics across two datasets (Sth-com and EK100-com), leading to improved generalization on unseen verb-object compositions.

Why It Matters

This work highlights a critical limitation in current video understanding models: they often act as 'object detectors' rather than 'action recognizers.' By providing a diagnostic framework to measure this bias and a regularization method to counteract it, the paper offers a path toward more robust, temporally-aware video models that can truly generalize to novel actions in real-world settings.

Key Terms and Definitions

Zero-Shot Compositional Action Recognition (ZS-CAR) — A task where models must recognize novel verb-object pairs using only primitives seen during training.
Object-driven Shortcut — A failure mode where a model predicts an action based solely on the presence of an object, ignoring the temporal motion required to define the verb.
Co-occurrence Prior — The bias in training data where certain verbs and objects appear together much more frequently than others, leading models to assume these pairs are the only valid combinations.
False Seen Prediction (FSP) — A diagnostic metric measuring the frequency with which a model incorrectly classifies an unseen action as a familiar, previously seen composition.
Temporal Order Sensitivity — The ability of a model to distinguish between actions that involve the same objects but different temporal sequences, such as 'opening' versus 'closing' a drawer.

Abstract

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.

Why Can’t I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition: Research Paper Podcast