Recent advancements in Large Language Models (LLMs) have given rise to a new generation known as Large Reasoning Models (LRMs). These models, exemplified by prominent names like OpenAI’s o-series, DeepSeek-R1, Claude Sonnet (3.7/4), and Gemini, are explicitly designed to generate detailed “thinking processes” or “reasoning traces” before providing their final answers. This sophisticated capability, often involving long Chain-of-Thought (CoT) with self-reflection, has demonstrated promising results on various reasoning benchmarks, leading some to propose them as significant steps towards more general artificial intelligence.
However, a recent study by Apple meticulously investigating these models using controlled puzzle environments reveals a more complex and, at times, counter-intuitive reality about their fundamental reasoning capabilities.
Beyond the Benchmarks
While LRMs have indeed shown improved performance on established mathematical and coding benchmarks, the traditional evaluation paradigms often suffer from issues like data contamination and do not offer deep insights into the actual structure and quality of the models’ internal reasoning. To overcome these limitations, researchers adopted controllable puzzle environments, such as Tower of Hanoi, Checker Jumping, River Crossing, and Blocks World. These environments allow for precise manipulation of problem complexity while maintaining consistent logical structures, enabling a systematic analysis of not just final answers, but also the rich “thinking” processes generated by these models.
Three Regimes of Reasoning
The study’s systematic comparison of LRMs with their standard LLM counterparts under equivalent inference compute unveiled three distinct performance regimes:
Low-complexity tasks
Surprisingly, standard (non-thinking) LLMs often outperform LRMs and demonstrate greater efficiency. They achieve comparable or even better accuracy with more token-efficient inference, suggesting that for simpler problems, the overhead of extensive “thinking” might be unnecessary or even detrimental.
Medium-complexity tasks
This is where the additional “thinking” mechanisms of LRMs, such as generating long Chain-of-Thought, manifest a clear advantage. The performance gap between LRMs and their non-thinking counterparts widens in favor of the reasoning models in these scenarios.
High-complexity tasks
Despite their advancements, for problems with high compositional depth, both LRMs and standard LLMs experience a complete collapse in performance. Their accuracy drops to zero, indicating fundamental limitations. While thinking models may delay this collapse compared to non-thinking counterparts, they ultimately encounter the same inherent barriers.
The “Pretense” of Thinking: Unveiling Core Limitations of Reasoning Models
Beyond these performance regimes, several findings from the study strongly suggest that the “thinking” demonstrated by current LRMs might be more akin to sophisticated pattern matching or an “illusion of thinking” rather than truly generalizable reasoning:
Counter-intuitive Scaling Limit in Reasoning Effort
One of the most striking findings is that LRMs exhibit a counter-intuitive scaling limit in their reasoning effort. Their reasoning effort, measured by the number of inference-time tokens generated during the “thinking” phase, initially increases with problem complexity. However, upon approaching a critical threshold—which closely aligns with their accuracy collapse point—models begin to reduce their reasoning effort despite increasing problem difficulty and having ample token budgets available. This suggests a fundamental inference-time scaling limitation in the reasoning capabilities of current LRMs relative to problem complexity; they seem to “give up” when problems become too hard, rather than applying more “thought”.
Failure to Benefit from Explicit Algorithms
Perhaps most surprisingly, even when LRMs were provided with the exact solution algorithm (e.g., pseudocode for Tower of Hanoi) directly in the prompt, their performance did not improve, and the accuracy collapse still occurred at roughly the same point. This is highly significant because finding and devising a solution should conceptually require substantially more computation than merely executing a given, prescribed algorithm. This observation strongly highlights limitations not just in discovering problem-solving strategies, but also in consistent logical verification and step execution. It raises critical questions about the symbolic manipulation capabilities of these models.
Inconsistent Reasoning Across Puzzle Types
LRMs demonstrate inconsistent reasoning capabilities across different puzzle types. For instance, Claude 3.7 Sonnet could perform up to 100 correct moves in the Tower of Hanoi (for N=10), achieving near-perfect accuracy for N=5 (31 moves). However, the same model failed to produce more than 5 correct moves in the River Crossing puzzle for N=3, which only requires 11 moves. This stark inconsistency may suggest a reliance on memorized instances or patterns from their training data rather than a truly flexible, generalizable reasoning ability, especially if certain puzzle types (like River Crossing with N>2) were scarce during training.
“Overthinking” Phenomenon
For simpler problems, LRMs frequently identify the correct solution early in their thinking process but then inefficiently continue exploring incorrect alternatives, leading to wasted compute. This “overthinking” is evidenced by solution accuracy tending to decrease or oscillate as thinking progresses in simpler problems. This points to fundamental inefficiencies in their internal reasoning processes, suggesting that their “thoughts” are not always perfectly logical or optimized.
Conclusion: Questions about True Reasoning
These comprehensive findings collectively challenge the prevailing assumptions about the true reasoning capabilities of current Large Reasoning Models. While their ability to generate extensive “thoughts” appears impressive, the study suggests that current approaches, even with sophisticated self-reflection mechanisms, may be encountering fundamental barriers to developing truly generalizable reasoning. The research underscores the critical need for more rigorous evaluation methods beyond traditional benchmarks, advocating for controlled environments that can systematically probe the strengths and, crucially, the inherent limitations of these complex AI systems. It prompts us to critically examine whether what we observe is genuine reasoning, or merely a sophisticated “illusion of thinking”.
References
“The Illusion of Thinking” by Parshin Shojaee et al.