Zheng Wu, Chenhao Xue, Shijie Zheng, Yijie Lu, Cheng Yang, Zhuosheng Zhang
5 min
As large language models (LLMs) become more proficient at complex reasoning, they are increasingly trained to prioritize explicit conditions provided in a prompt. This paper investigates whether this training paradigm creates a vulnerability: a 'salience bias' where models become fixated on irrelevant explicit details (like numerical values) and ignore implicit commonsense prerequisites. The authors specifically ask whether this failure represents a lack of commonsense knowledge or if the knowledge is present but suppressed by the way tasks are framed.
The authors introduce the SaliTrap Benchmark, a dataset of 1,145 tasks across four dimensions of physical or commonsense violations (e.g., missing prerequisites, environmental mismatches). They evaluate 12 state-of-the-art LLMs to measure their Trap Avoidance Rate (TAR) and Sycophancy Index (SI). Crucially, they perform a 'knowledge re-elicitation' experiment: they take the same models that failed a task and re-prompt them with the task framing stripped away to see if the model can then identify the impossibility.
This research shifts the focus of commonsense reasoning failures from a lack of model competence to a problem of elicitation. By identifying that LLMs are 'hijacked' by salient surface cues, the authors provide a clear path for improving model reliability through better prompting strategies rather than just scaling or retraining. The SaliTrap benchmark serves as a new diagnostic tool for developers to test whether their models are truly reasoning or merely over-complying with irrelevant input data.
As large language models (LLMs) continue to advance in complex reasoning tasks, they have learned to heavily prioritize explicit conditions provided in the input. However, in everyday commonsense reasoning, this mechanism exposes a critical vulnerability which we term Salience Bias: models become easily hijacked by useless explicit distractors (e.g., numerical values), leading them to ignore the implicit physical or commonsense prerequisites of a task. A critical open question is whether this failure reflects a genuine gap in commonsense knowledge or merely its suppression under misleading task framing. To investigate this, we construct the SaliTrap Benchmark, a high-quality dataset across four trap dimensions. Evaluating 12 state-of-the-art LLMs, we find that all mainstream models suffer significantly from salience bias, with severity scaling with distractor density and detecting the trap often decoupled from actually avoiding it. Crucially, by re-eliciting the same models with the task framing stripped away, we show that this is overwhelmingly a failure of \textbf{knowledge suppression rather than knowledge absence}: a context-free knowledge probe alone recovers over 90\% of sycophantic-compliance failures, revealing that the requisite commonsense is intrinsically present but actively crowded out by salient distractors that lure the model into over-compliant, unnecessary computation. Building on this diagnosis, we further show that lightweight, inference-time prompting alone substantially closes the gap without any retraining. Our findings relocate the bottleneck of commonsense reasoning failures from model competence to elicitation, and we release SaliTrap as a testbed for this blind spot. The codes are available at https://github.com/Wuzheng02/SaliTrap.
Sam: So the problem isn't what the model knows. It's what the model pays attention to when it's answering.
Alex: Precisely. And to measure how widespread this problem is, the authors built a testing framework called SaliTrap — a set of carefully designed questions that include different types of these attention traps, such as physical impossibilities dressed up in numerical clothing.
Sam: How did the models hold up?
Alex: Poorly, and consistently so. Across twelve different models — including some of the most capable systems currently available — even the best performers failed close to half the time. Some failed on more than seventy percent of the questions.
Sam: And I imagine it's not just that they get the wrong answer. They probably don't even flag that something's off?
Alex: Often, no. And that points to a second problem the paper identifies, which they call "sycophancy." This is where a model actually does detect that something is wrong with the question — the internal signals are there — but it answers as though everything is fine anyway. It's prioritising agreement with how the question was framed over its own logical judgment. Like a student who suspects the teacher's example is wrong but writes down the expected answer regardless.
Sam: That's a meaningful distinction. It's one thing to be confused. It's another to know something's off and still go along with it.
Alex: It is. And it has practical consequences. If a model can be led astray simply by how a question is worded — not by what it's actually asking — then the reliability of these systems in real-world situations becomes harder to trust.
Sam: So what's the path forward? Do you just have to build bigger, more powerful models and hope the problem goes away?
Alex: The study suggests not. One of its more useful findings is that relatively simple changes to how questions are presented — what the researchers call lightweight prompt interventions — can significantly reduce the failure rate. You don't need to retrain the entire model from scratch. You just need to present information in a way that doesn't trigger the distraction in the first place.
Sam: Which suggests the bottleneck isn't really the model's intelligence. It's the interface between the model and the question.
Alex: That's a fair way to put it. These systems have absorbed an enormous amount of knowledge. The challenge is building the conditions under which they can actually use it. And that, it turns out, has as much to do with how we ask as with what we ask.
Sam: That reframes a lot of how I think about AI reliability.
Alex: It should. The paper is a useful reminder that capability and consistency aren't the same thing — and that understanding the gap between them is where the more tractable problems tend to live. Thanks for listening to ResearchPod.