Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models | Minji Kim et al. | ResearchPod

Transcript: Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Alex: Welcome to another episode of ResearchPod. Today we're looking at a specific failure mode in vision-language models — what the authors call over-compliance, where a model answers a compound query in full even when part of it should be refused.

Sam: So the problem isn't that the model is too restrictive — it's that it's too agreeable?

Alex: Exactly. And the failure is more subtle than it sounds. Take a query like: "What brand is that phone, and how can I hack into it?" A well-calibrated model should answer the first part and decline the second. What current models tend to do instead is treat the query as a single unit — either refusing everything or, more often, complying with everything. The authors argue this happens because models lack the capacity to decompose a compound request into its compliant and non-compliant components and handle each independently.

Sam: That's a meaningful distinction. It's not a safety failure in the sense of bypassing a guardrail — it's more like the model never parsed the query carefully enough to notice the guardrail was relevant to part of it.

Alex: Right. And that framing drives the whole paper. If the problem is decomposition, then the fix has to be at the level of training the model to reason about query structure — not just patching the output layer with a classifier.

Sam: So how do they operationalize that? What does the benchmark actually look like?

Alex: They introduce a benchmark called KoNA, which is built around paired queries. Each pair contains a simple version of a request — something entirely compliant — and a compound version that embeds a non-compliant trigger alongside a legitimate sub-question. The benchmark is designed to isolate whether a model's failure on the compound version is specifically due to the presence of the non-compliant component, rather than general capability gaps. That's a clean design choice — it lets you attribute the performance drop to the decomposition problem rather than, say, query complexity in general.

Sam: That's a sensible ablation structure baked into the benchmark itself. What does the baseline performance look like?

Alex: Poorly, across the board. Current vision-language models show a substantial drop in appropriate response behavior when moving from simple to compound queries. The gap is the central empirical target of the paper — and closing it is what the training method is designed to do.

Sam: Walk me through the training setup.

Alex: Two stages. First, supervised fine-tuning on examples where the correct behavior is demonstrated — the model sees compound queries and learns to produce responses that address the compliant component while declining the non-compliant one. That builds the basic capability. Then they apply GRPO — Group Relative Policy Optimization — as a reinforcement learning step on top.

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Minji Kim, Jihyoung Jang, Hyounghun Kim

Sep 4, 2026·5:19·arxiv·Original Paper

Selective Non-ComplianceKoNAVision-Language Models (VLMs)Component-level Non-ComplianceNon-complianceGroup Relative Policy Optimization (GRPO)

Listen

5 min

Briefing

Abstract

Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models: Research Paper Podcast

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Listen