Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = +0.41), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that asks a deceptively simple question: when you give an AI a specific set of rules, does it actually follow them?
Sam: That sounds like it should have an obvious answer. You give it instructions, it follows them. What's the complication?
Alex: The complication is that these AI systems—the kind that power modern chatbots—aren't blank slates. Before you ever type a single instruction, they've already been shaped by an enormous amount of text from the internet, books, forums, and more. That training leaves them with something like a built-in point of view.
Sam: So they arrive at the task with pre-existing ideas already baked in.
Alex: Exactly. The researchers call these "internalized priors." Think of it like hiring someone who has spent years developing their own strong opinions about a subject. You can hand them a rulebook, but their instincts don't just disappear.
Sam: And the study tested this with a specific kind of task?
Alex: Yes. The task was content annotation—specifically, flagging toxic comments in a gaming chat. It's a job that sounds straightforward: read a comment, decide if it's toxic, label it. But "toxic" is actually a slippery concept. What counts as toxic in a competitive gaming community might be completely different from what counts as toxic in a children's educational forum.
Sam: So the researchers gave the model their own specific definition of toxic, and then watched whether it used that definition or just defaulted to whatever it already believed?
Alex: That's precisely it. And what they found was a pattern they call "decision stickiness." Even when the model was given a clear, correct definition, it frequently stuck to its original judgment—the one formed during training—rather than applying the new rules.
Sam: So it's not just getting it wrong. It's getting it wrong with confidence?
Alex: That's one of the more notable findings. Even when the model was operating with a definition that didn't match the task, its confidence level didn't drop. It produced wrong answers while appearing completely certain. That's a meaningful problem, because high confidence is usually a signal we use to decide whether to trust a system.
Sam: It's like a student who memorized the wrong formula and then aces the test with completely wrong answers—and doesn't even flag that something might be off.
Alex: That's a useful way to put it. And it points to a deeper issue. We tend to assume that if we write a careful prompt—a clear, detailed set of instructions—the model will follow it. But this research suggests that assumption has real limits.
Sam: So how did the researchers actually measure the gap between what the model believed and what the instructions said?
Alex: They developed a new measurement they call "Definition-Specific Familiarity," or DSF. Here's the key distinction: most ways of evaluating AI ask whether the model has seen certain text before—a kind of memorization check. DSF asks something different. It asks whether the model's internal concept of a word like "toxic" actually lines up with the specific definition you've provided.
Sam: So it's less about what the model has read, and more about whether its mental picture of the concept matches yours.
Alex: Exactly. And they found that this conceptual alignment—how closely the model's built-in understanding matches your definition—is a much better predictor of how well it will perform than simple memorization. If that alignment is low, better prompting alone won't reliably fix it.
Sam: That has real consequences for anyone using these models to moderate content or label data at scale. You might think you've set the rules clearly, but the model could be quietly running on its own rulebook the whole time.
Alex: The paper makes that point carefully. For annotation tasks especially, the model's pre-existing conceptual boundaries appear to shape its behavior in ways that prompt-based instructions can only partially override. It doesn't mean these tools are unusable—but it does mean that assuming a well-written prompt is sufficient may be an overestimate of what prompting can do.
Sam: It's a bit like the difference between telling someone what to do and actually changing how they think. Instructions can guide behavior up to a point, but they don't rewrite someone's instincts.
Alex: That's a fair way to frame it. The study suggests that for high-stakes annotation work, understanding what a model already believes about a concept may matter just as much as the instructions you give it. The prompt is only part of the picture.
Sam: That's going to make me think differently about what it actually means to "instruct" an AI. Thanks for walking through it.
Alex: Thanks for listening to ResearchPod.