Reasoning models hit a compute ceiling, but sparse architectures keep climbing

The researchers evaluated 19 open-source LLMs, from the 1-billion-parameter Gemma-3-1B up to 30-billion-parameter Qwen3 variants, on five reasoning benchmarks of escalating difficulty. GSM8K tests grade-school arithmetic. MATH500 and two competition-level sets (AIME 2025, HMMT February 2025) push into Olympiad-style territory. GPQA-Diamond asks graduate-level science questions designed to stump Google searches. Every model ran with full chain-of-thought decoding, a single pass per problem, and no majority-vote tricks.

The twist is in how they scored each model. Rather than reporting accuracy alone, the team estimated the floating-point operations (FLOPs) consumed by every reasoning trace, using a component-aware formula that accounts for grouped-query attention, gated feed-forward layers, and MoE routing. This let them plot each model's accuracy against its actual computational cost and identify the Pareto frontier, the set of models where you cannot improve accuracy without spending more compute.

The headline finding is architectural: Mixture-of-Experts models sit on the Pareto frontier far more often than dense models do. An MoE like Qwen3-30B-A3B activates only 3 billion of its 30 billion parameters per token, which lets it generate much longer reasoning chains within the same FLOP budget. On GPQA-Diamond, AIME 2025, and HMMT February 2025, MoE models consistently dominated the frontier, achieving accuracy that dense models could match only by burning orders of magnitude more compute.

This matters because chain-of-thought reasoning is token-hungry. Models that "think" by generating thousands of intermediate tokens pay a linear cost per token through the feed-forward layers and a quadratic cost through attention. By activating a sparse subset of experts, MoE architectures keep the linear cost low, which effectively decouples the model's total parameter knowledge from its per-token inference price. Sparsity, in other words, is the lever that makes extended reasoning affordable.

Every benchmark shows the same shape: a steep initial climb where more compute buys meaningful accuracy gains, followed by a flattening "knee" after which additional FLOPs return almost nothing. Where that knee sits depends on how hard the task is. On GSM8K, models plateau below 10¹⁶ FLOPs because even small models can largely solve grade-school maths. On AIME 2025 and GPQA-Diamond, the knee shifts to much higher compute regimes, yet even there the curve eventually flattens. No amount of thinking time can overcome a model's fundamental "deductive horizon."

Smaller dense models (1B to 8B parameters) exploit this dynamic in an interesting way. By generating substantially longer reasoning traces, they trade parameter capacity for inference-time compute and occasionally reach accuracy parity with 30B+ baselines. The catch is that this strategy has hard limits. Once a model's knowledge runs out, thinking longer produces only longer failures.

Perhaps the most practically useful finding: 97% of evaluated models spend more compute on incorrect answers than on correct ones. When a model lacks the capacity to solve a problem, it does not stop early. It loops, backtracks, hallucinates alternative paths, and frequently exhausts its context window in a futile attempt to converge. The paper calls this "trace length asymmetry," and it means that the hardest-to-solve problems are also the most expensive to fail at.

This creates a vicious efficiency trap for deployment. Problems outside a model's capability boundary are precisely the ones that consume the most resources, with no accuracy payoff. The authors frame this as an open algorithmic challenge: building early-stopping mechanisms that can distinguish between productive reasoning (which genuinely needs time) and unproductive spiralling (which should be terminated) remains unsolved.