Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez
4 min
Abstract
Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = +0.41), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.
Sam: So how did the researchers actually measure the gap between what the model believed and what the instructions said?
Alex: They developed a new measurement they call "Definition-Specific Familiarity," or DSF. Here's the key distinction: most ways of evaluating AI ask whether the model has seen certain text before—a kind of memorization check. DSF asks something different. It asks whether the model's internal concept of a word like "toxic" actually lines up with the specific definition you've provided.
Sam: So it's less about what the model has read, and more about whether its mental picture of the concept matches yours.
Alex: Exactly. And they found that this conceptual alignment—how closely the model's built-in understanding matches your definition—is a much better predictor of how well it will perform than simple memorization. If that alignment is low, better prompting alone won't reliably fix it.
Sam: That has real consequences for anyone using these models to moderate content or label data at scale. You might think you've set the rules clearly, but the model could be quietly running on its own rulebook the whole time.
Alex: The paper makes that point carefully. For annotation tasks especially, the model's pre-existing conceptual boundaries appear to shape its behavior in ways that prompt-based instructions can only partially override. It doesn't mean these tools are unusable—but it does mean that assuming a well-written prompt is sufficient may be an overestimate of what prompting can do.
Sam: It's a bit like the difference between telling someone what to do and actually changing how they think. Instructions can guide behavior up to a point, but they don't rewrite someone's instincts.
Alex: That's a fair way to frame it. The study suggests that for high-stakes annotation work, understanding what a model already believes about a concept may matter just as much as the instructions you give it. The prompt is only part of the picture.
Sam: That's going to make me think differently about what it actually means to "instruct" an AI. Thanks for walking through it.
Alex: Thanks for listening to ResearchPod.