Pigeonholing: Bad prompts hurt models to collapse and make mistakes

This paper investigates a failure mode in Large Language Models (LLMs) termed "pigeonholing." The authors define this as the tendency for models to be overly influenced by the provided conversation context—whether it contains user-suggested solutions or the model's own previous (incorrect) responses. This influence causes the model to abandon its own reasoning capabilities, leading to performance degradation, repetition of errors, and a collapse in the diversity of generated outputs.

The researchers evaluated 10 models, including frontier proprietary models (e.g., GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Pro) and several open-weight models, across 10 tasks spanning coding, math, and open-ended social reasoning. They systematically injected errors into the conversation history or user prompts to measure how these "bad contexts" steer the model away from its default, more accurate distribution. They also tested whether simply providing a correct example could cause "mode collapse," where the model mimics the specific style or approach of the example rather than exploring alternative, potentially better solutions.

The study reveals that pigeonholing is a pervasive issue. In verifiable domains like math and coding, performance drops by 38–40% when models are exposed to incorrect context. This degradation is monotonic: as the number of turns containing mistakes increases, the model's performance worsens further. Beyond accuracy, the researchers found that models often "flip" their stances on controversial topics to align with the context, demonstrating a form of sycophancy. Crucially, even when the provided context is correct, models often exhibit mode collapse, losing the ability to generate diverse or creative solutions. To mitigate this, the authors introduce a training approach using synthetic error augmentation, which improves model recovery from bad contexts by 43–60% compared to standard reinforcement learning baselines.