What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities

Recent research often attributes the inconsistent performance of Large Language Models (LLMs) on planning tasks to general task difficulty. This paper challenges that assumption, proposing that planning is not a single, monolithic ability. Instead, the authors investigate whether LLM planning performance reflects distinct latent competencies that respond differently to model scaling and reasoning strategies. Using ACPBench-Hard, the researchers evaluated multiple LLM families (Qwen, Gemma, and Granite) across three inference conditions: direct answering, Chain-of-Thought (CoT), and a scratchpad-based agent harness. They applied a Multidimensional Item Response Theory (MIRT) model to decompose performance into latent dimensions, testing whether these dimensions are compensatory (where strength in one offsets weakness in another) or non-compensatory (where success requires proficiency in both).

The analysis reveals that a two-dimensional, non-compensatory model best explains the observed performance patterns. The two identified dimensions are:

• Operational Reasoning: The ability to evaluate local action applicability and immediate state transitions. This competency improves significantly with model scaling and the use of Chain-of-Thought reasoning traces.
• Structural Enumeration: The ability to reason about goal reachability and landmark structure. Unlike operational reasoning, this competency remains a persistent bottleneck, showing little improvement even with larger models or more extensive reasoning budgets.

The non-compensatory nature of these dimensions is critical: because they do not substitute for one another, a model's failure in structural enumeration cannot be masked by its proficiency in operational reasoning. This explains why LLMs often remain brittle on planning tasks despite improvements in general reasoning capabilities.

This work shifts the focus of LLM evaluation from aggregate accuracy to competency-level analysis. By identifying that structural enumeration is a systematic blind spot, the authors provide a framework for diagnosing why current scaling laws and reasoning scaffolds fail to produce reliable planning agents. This suggests that future research should prioritize architectural or training interventions specifically targeting global state-space reasoning rather than relying solely on increased model size or standard CoT prompting.