The Expertise Trap: General365 and the Unsolved Challenge of General Reasoning

The contemporary narrative surrounding large language model capabilities has become fixated on Olympiad mathematics. We are shown charts where models achieve gold medal scores on the International Mathematical Olympiad (IMO) or near perfect accuracy on the American Invitational Mathematics Examination (AIME). These benchmarks suggest we are on the precipice of artificial general intelligence, or at least of systems that reason as well as human experts. Yet a closer inspection reveals a troubling disconnect. In the recent paper General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks, Liu et al. demonstrate that when we strip away the need for advanced domain knowledge and focus purely on reasoning within K-12 constraints, the performance of even the most sophisticated models collapses. The state of the art, achieved by Gemini-3-Pro, reaches only 62.8% accuracy, with the majority of evaluated models failing to meet a 60% passing threshold. This is not merely a measurement issue. It is evidence that we have been optimizing for expertise retrieval rather than general reasoning, and the gap between these two capabilities defines the actual frontier of artificial intelligence research.

The Architecture of a Pure Reasoning Benchmark

The methodological rigor of General365 reveals why previous benchmarks failed to expose this vulnerability. Prior general reasoning evaluations such as BBH (Big Bench Hard) and BBEH suffered from clustering effects in embedding space, indicating limited diversity and repetitive templates that allowed models to exploit surface patterns rather than engage in deep reasoning. Liu et al. address this through a dual constraint approach. First, they restrict background knowledge strictly to K-12 levels, explicitly decoupling reasoning from specialized expertise. Second, they construct 365 manually crafted seed problems expanded into 1,095 variants through controlled semantic and constraint alterations, ensuring that models cannot rely on memorized solution patterns.

The benchmark spans eight distinct categories including Constraint Satisfaction, Nested Logic, Semantic Interference, and Optimal Strategy selection. Each category targets specific failure modes in current architectures. Particularly notable is the hybrid scoring system combining rule-based verification with model-based evaluation, achieving 99.6% accuracy in automated grading. This technical detail matters because general reasoning tasks often involve open ended or multi step logical deductions where simple string matching fails. By establishing this rigorous scoring infrastructure, the authors ensure that the 62.8% ceiling represents a genuine reasoning limitation rather than a measurement artifact.

The Performance Collapse and Efficiency Paradoxes

When evaluated against 26 leading models including GPT-5 variants, Claude Sonnet 4.5, DeepSeek V3.2, and the Qwen3 family, the results reveal a stark hierarchy. Gemini-3-Pro leads at 62.8%, followed by Gemini-3-Flash at 60.8%, with most reasoning models clustering between 48% and 58%. This distribution is remarkable when contrasted with the saturation observed in mathematical reasoning benchmarks, where multiple models now exceed 90% on AIME. The authors identify "Semantic Interference" and "Optimal Strategy" as the primary bottlenecks, suggesting that current architectures struggle when logical requirements conflict with surface level linguistic cues or when they must select between multiple valid reasoning paths without explicit guidance.

Perhaps more revealing than the accuracy scores is the analysis of reasoning efficiency. Gemini-3-Pro achieves its leading performance with an average output of approximately 15,000 tokens, while comparably scoring models typically require 25,000 tokens. This discrepancy suggests that token efficiency serves as a proxy for reasoning quality. Models that truly understand a problem's logical structure can generate concise solution paths, whereas those engaged in pattern matching or exhaustive search produce verbose, meandering outputs. This finding has immediate practical implications for inference costs, but more importantly, it indicates that much of the "reasoning" we observe in current systems may be closer to stochastic parroting than structured logical deduction.

The Training Paradigm Problem

The General365 results force a reassessment of our training methodologies. Current reinforcement learning from human feedback (RLHF) and related alignment techniques have heavily emphasized mathematical and coding domains, where verification is automatic and reward signals are dense. This has created systems that excel at pattern matching within well defined formal systems but falter at the messy, constraint heavy reasoning required for everyday problem solving. The benchmark's design exposes what we might call the "expertise trap": by training models to accumulate vast domain specific knowledge, we have inadvertently created systems that conflate knowing with reasoning.

The Semantic Interference category particularly illuminates this issue. When models encounter problems where natural language semantics suggest one solution path but logical constraints demand another, they exhibit high failure rates. This indicates that current architectures lack robust mechanisms for decoupling surface form from underlying logical structure. Similarly, the Optimal Strategy category reveals deficits in metacognitive reasoning; models struggle to evaluate which of several valid approaches will most efficiently satisfy complex constraint sets. These are not knowledge gaps that can be solved by scaling pretraining data. They represent fundamental architectural limitations in how transformers compose and manipulate abstract logical structures.

Original Insights: Compositional Reasoning and the Diversity Crisis

Beyond the paper's explicit findings, several inferences deserve emphasis. First, the embedding visualization comparing General365 to BBH/BBEH suggests that our field has been suffering from a diversity crisis in evaluation. The clustering observed in existing benchmarks implies that we have been training and validating models on narrow distributions of reasoning problems, leading to overestimation of generalization. General365's broad distribution in semantic space indicates that true general reasoning requires handling combinatorial explosion in problem structures, something that cannot be captured by simply scaling up existing mathematical datasets.

Second, the token efficiency results suggest a path forward for model evaluation beyond accuracy metrics. We should be measuring the compressibility of reasoning traces. A model that requires 25,000 tokens to solve a problem that another solves in 15,000 is likely engaging in brute force search rather than structured reasoning. Future benchmarks should incorporate inference cost and token efficiency as first class metrics, not just afterthoughts.

Finally, the K-12 knowledge constraint highlights a counterintuitive possibility: we may have overindexed on knowledge intensive reasoning at the expense of pure logical capability. The fact that models perform worse on General365 than on physics Olympiad problems suggests that domain knowledge acts as a crutch, allowing models to mask reasoning deficits with retrieved facts and patterns. This implies that the next generation of training data should emphasize compositional problems with minimal background knowledge requirements, forcing architectures to develop more robust logical processing rather than sophisticated retrieval mechanisms.

Conclusion: Toward General Reasoning

General365 serves as a necessary corrective to the field's current trajectory. It demonstrates that our progress on mathematical benchmarks, while technically impressive, does not translate to the general reasoning capabilities required for real world application. The 62.8% performance ceiling leaves substantial room for improvement, and the identification of Semantic Interference and Optimal Strategy as key challenges provides concrete targets for future research.

Moving forward, the field must grapple with how to train for general reasoning directly rather than hoping it emerges as a side effect of domain expertise. This may require new architectures that explicitly model logical constraints, training objectives that reward compositional generalization over pattern completion, and evaluation protocols that prioritize diversity and efficiency alongside accuracy. The question is no longer whether models can solve Olympiad problems, but whether they can navigate the mundane yet complex reasoning tasks that define human cognition. General365 shows us that on this metric, we are still in the early days.