The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity | Parshin Shojaee et al. | ResearchPod

Transcript: The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that examines the actual limits of what are called Large Reasoning Models, or LRMs.

Sam: These are the newer AI systems that visibly "think" before they answer, right? They show their working, like a student writing out steps on a math test. And the big question is whether that thinking is real, or just for show?

Alex: That's exactly the question. The assumption has been that if a model spends more time computing, it must be doing something more sophisticated. But this paper suggests there's a hard ceiling—a point where the model doesn't just struggle, it stops trying altogether.

Sam: That reminds me of a student who does fine on straightforward homework but completely shuts down when the exam gets genuinely difficult. Is that the central finding?

Alex: It is. The research shows these models increase their apparent effort as problems get harder—but only up to a point. Beyond that point, their effort actually drops, and their performance collapses. The harder the problem, the less they seem to engage with it.

Sam: How did the researchers test this without the models just recalling answers from their training data? If you give an AI a standard puzzle, hasn't it probably seen something like it before?

Alex: That's a key concern, and the researchers addressed it directly. They built what they call "controllable puzzle environments." Think of these as custom logic games—similar to the Tower of Hanoi, where you move discs between pegs in a specific order—where you can precisely dial up the difficulty by adding more pieces. The model hasn't memorised the answer, because the specific configuration is new every time.

Sam: So it's like a stress test. You keep adding weight to a bridge until it snaps, and that tells you exactly where the breaking point is.

Alex: Precisely. And what they found at that breaking point is telling. In simpler puzzles, the models would actually find the correct answer relatively early in their reasoning—but then keep going. They'd continue exploring wrong paths long after the solution was already in hand. It's a kind of compulsive overthinking.

Sam: So even when they get it right, they're not being efficient about it?

Alex: Right. And in moderately difficult puzzles, they'd work through some trial and error and eventually land on the right path. But once the complexity crossed a certain threshold, something different happened. Instead of thinking harder, they thought less. The reasoning traces—the written-out steps—actually got shorter as the problems got harder.

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar

Jul 21, 2026·4:11·user_upload

Large Reasoning Models (LRMs)Problem ComplexityReasoning TracePerformance RegimesScaling LimitLRMs (Large Reasoning Models)

Listen

4 min

Briefing

Research Overview

This paper investigates the actual reasoning capabilities and limitations of Large Reasoning Models (LRMs) like OpenAI's o1/o3, DeepSeek-R1, and Claude 3.7 Sonnet. While these models are marketed for their advanced 'thinking' processes, the authors argue that current evaluation benchmarks (like math and coding tests) are prone to data contamination and fail to reveal how these models handle varying levels of problem complexity. By using controlled, deterministic puzzle environments—such as the Tower of Hanoi and Blocks World—the researchers systematically manipulate problem difficulty to observe how these models perform and how their internal reasoning traces evolve.

The Three Regimes of Reasoning

The study identifies three distinct performance regimes based on problem complexity:

Low Complexity: Standard (non-thinking) LLMs often outperform LRMs, proving more efficient and accurate when the task is simple enough that extensive deliberation is unnecessary.
Medium Complexity: LRMs demonstrate a clear advantage, as their ability to generate long chains of thought allows them to navigate more intricate logical steps that standard models struggle to resolve.
High Complexity: Both LRMs and standard models experience a complete performance collapse. Notably, as tasks reach this high-complexity threshold, LRMs begin to reduce their reasoning effort (token usage), suggesting a fundamental scaling limit in their ability to allocate compute effectively when faced with insurmountable difficulty.

Insights into 'Thinking' Traces

By analyzing the internal reasoning traces, the authors found that LRMs often exhibit inefficient behaviors. In simpler tasks, they frequently engage in 'overthinking,' where they continue to explore unnecessary alternatives even after identifying the correct solution. In more complex tasks, they often fixate on early, incorrect paths and fail to self-correct, leading to a total failure to reach the target state. These findings suggest that current LRMs do not possess generalizable algorithmic reasoning but rather rely on patterns that break down when the compositional depth of a problem exceeds their training distribution.

Key Terms and Definitions

Large Reasoning Models (LRMs) — Specialized language models designed to generate extensive, step-by-step 'thinking' traces (Chain-of-Thought) before providing a final answer.
Compositional Complexity — The degree of difficulty in a task determined by the number of sequential, interdependent steps or elements required to reach a solution.
Inference Compute Budget — The total number of tokens a model is permitted to generate during its reasoning process before arriving at a final output.
Data Contamination — A common issue in AI benchmarks where test questions are inadvertently included in the model's training data, leading to inflated performance metrics.
Overthinking Phenomenon — A behavior where a model continues to generate redundant or unnecessary reasoning tokens even after it has already identified the correct solution.

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity: Research Paper Podcast