Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a specific failure mode in vision-language models — what the authors call over-compliance, where a model answers a compound query in full even when part of it should be refused.
Sam: So the problem isn't that the model is too restrictive — it's that it's too agreeable?
Alex: Exactly. And the failure is more subtle than it sounds. Take a query like: "What brand is that phone, and how can I hack into it?" A well-calibrated model should answer the first part and decline the second. What current models tend to do instead is treat the query as a single unit — either refusing everything or, more often, complying with everything. The authors argue this happens because models lack the capacity to decompose a compound request into its compliant and non-compliant components and handle each independently.
Sam: That's a meaningful distinction. It's not a safety failure in the sense of bypassing a guardrail — it's more like the model never parsed the query carefully enough to notice the guardrail was relevant to part of it.
Alex: Right. And that framing drives the whole paper. If the problem is decomposition, then the fix has to be at the level of training the model to reason about query structure — not just patching the output layer with a classifier.
Sam: So how do they operationalize that? What does the benchmark actually look like?
Alex: They introduce a benchmark called KoNA, which is built around paired queries. Each pair contains a simple version of a request — something entirely compliant — and a compound version that embeds a non-compliant trigger alongside a legitimate sub-question. The benchmark is designed to isolate whether a model's failure on the compound version is specifically due to the presence of the non-compliant component, rather than general capability gaps. That's a clean design choice — it lets you attribute the performance drop to the decomposition problem rather than, say, query complexity in general.
Sam: That's a sensible ablation structure baked into the benchmark itself. What does the baseline performance look like?
Alex: Poorly, across the board. Current vision-language models show a substantial drop in appropriate response behavior when moving from simple to compound queries. The gap is the central empirical target of the paper — and closing it is what the training method is designed to do.
Sam: Walk me through the training setup.
Alex: Two stages. First, supervised fine-tuning on examples where the correct behavior is demonstrated — the model sees compound queries and learns to produce responses that address the compliant component while declining the non-compliant one. That builds the basic capability. Then they apply GRPO — Group Relative Policy Optimization — as a reinforcement learning step on top.
Sam: What's GRPO doing that SFT alone doesn't handle?
Alex: The core issue with SFT is that it teaches the model to imitate correct outputs, but it doesn't directly optimize for the decision boundary between compliant and non-compliant handling. GRPO frames it as a reward signal: the model gets credit for correctly partitioning the query — declining what should be declined, answering what should be answered — and the reward is computed relative to a group of sampled responses rather than an absolute reference. That relative comparison is what makes GRPO more stable than standard policy gradient methods in this setting. Essentially, it's shaping the model's behavior around the decomposition judgment itself, not just the surface form of the output.
Sam: So the model is learning something closer to a deliberate reasoning step — checking each component of the query against its own compliance criteria — rather than just pattern-matching to a refusal template.
Alex: That's the intended mechanism, yes. And the main finding is that this two-stage training meaningfully narrows the performance gap between simple and compound queries that defines the baseline failure. That gap is the load-bearing result — if it closes, the core claim holds.
Sam: Does the training come at a cost to general capability? Over-refusal is the obvious failure mode on the other side.
Alex: The authors report that general capabilities remain stable, which is the expected claim in any alignment-adjacent paper. They do flag some instances of over-refusal in their qualitative analysis, which is worth taking seriously — it suggests the decision boundary isn't perfectly calibrated. What's not fully resolved is how sensitive that boundary is to query phrasing. If a legitimate request is worded in a way that superficially resembles a non-compliant trigger, does the model refuse it? The paper doesn't stress-test that systematically, and a careful reviewer would push on it.
Sam: That feels like the natural next experiment — adversarial compliant queries, things that look like they should be refused but aren't.
Alex: Exactly. And there's a related question about distribution shift — the KoNA benchmark defines what counts as a non-compliant trigger, but real-world compound queries are considerably messier. How the model generalizes beyond the benchmark's taxonomy is an open question the paper doesn't fully address.
Sam: So the contribution is real, but bounded. A training approach that demonstrably improves selective non-compliance on a well-structured benchmark, with some evidence that general capability is preserved, and open questions about calibration and generalization.
Alex: That's a fair summary. The decomposition framing is the conceptual contribution — it reframes over-compliance as a structural reasoning failure rather than a safety tuning problem, and that reframing motivates a training approach that's more principled than adding refusal examples to a fine-tuning set. Whether the approach scales cleanly to the full distribution of real queries is where the uncertainty lives.
Sam: Good place to sit with it. Thanks for walking through this one, Alex.
Alex: Thanks for listening to ResearchPod.