The 36% Ceiling: What Generated Games Reveal About the Limits of LLM Reasoning

Introduction

The numbers are stark, and they tell an uncomfortable story. On the newly proposed gg-bench evaluation, GPT-4o and Claude 3.7 Sonnet achieve win rates of merely 7% to 9% against reinforcement learning agents. Reasoning models such as o1, o3-mini, and DeepSeek-R1 perform significantly better, reaching 31% to 36%. While the fourfold improvement between base and reasoning models has garnered attention, the real revelation lies in what these models cannot do. Even the most sophisticated systems available today fail to win two-thirds of matches against agents trained from scratch on games neither player has seen before.

This performance ceiling emerges from "Measuring General Intelligence with Generated Games," a paper introducing a dynamic evaluation framework designed to test strategic generalization rather than knowledge retrieval. Unlike static benchmarks that suffer from training data contamination, gg-bench generates novel game environments algorithmically, creating an effectively infinite stream of unseen evaluation tasks. The results suggest that despite impressive advances in scale and training compute, large language models remain fundamentally limited in their ability to adapt to novel rule systems and strategic contexts.

The Methodological Innovation: Dynamic Evaluation

Traditional benchmarks for reasoning, such as GSM8K or MATH, rely on fixed datasets. As models train on increasingly large corpora of internet text, the risk of data contamination grows; a model may perform well not through genuine reasoning, but through memorization of solutions encountered during training. The authors of the gg-bench paper address this limitation through a three-stage generative pipeline that ensures every evaluation instance is truly novel.

First, a large language model generates natural language descriptions of previously nonexistent games, specifying rules, win conditions, and state representations. Second, the same model translates these descriptions into executable Python code conforming to the OpenAI Gym interface, creating functional game environments. Third, reinforcement learning agents undergo training via self-play exclusively on these generated games, developing specialized policies optimized for the specific rule structures.

This methodology creates a moving target for evaluation. Because the games are synthesized rather than curated from existing sources, no amount of pretraining on internet data can confer direct advantage. The RL agents serve as a baseline representing specialized competence; they master the specific game through thousands of training episodes, while the tested language models must interpret rules and select moves using only in-context learning from the textual description and current board state.

The metric is straightforward yet demanding: win rate in direct competition. Success requires not merely understanding syntax or following instructions, but constructing an internal model of game dynamics, anticipating opponent responses, and identifying winning strategies in combinatorial spaces. That state-of-the-art reasoning models peak at approximately 36% against these specialized agents indicates a profound gap between human-like strategic adaptation and current artificial intelligence capabilities.

Interpreting the Performance Gap

The divergence between base models and reasoning models warrants careful analysis. The improvement from roughly 8% to 36% suggests that inference-time computation and chain-of-thought reasoning significantly enhance performance on novel strategic tasks. Models like o1 and DeepSeek-R1 apparently leverage additional test-time compute to simulate game dynamics or evaluate move consequences more effectively than their base counterparts.

However, the ceiling itself demands scrutiny. These are not superhuman opponents. The RL agents are trained from scratch on single game instances, often with relatively simple architectures and limited training budgets compared to the massive pretraining of the LLMs they defeat. In principle, a system with genuine general intelligence should rapidly adapt to novel rule sets, potentially matching or exceeding specialized agents through superior transfer learning and abstraction capabilities. That even the best systems lose nearly two-thirds of matches suggests that current architectures lack the capacity for sample-efficient strategic adaptation.

The distinction between knowledge retrieval and strategic generalization becomes crucial here. Most LLM capabilities demonstrated to date involve retrieving and synthesizing patterns present in training data. When presented with coding problems, translation tasks, or scientific questions, models excel by drawing upon vast corpora of human knowledge. gg-bench strips away this crutch. The games are freshly generated, never before encountered by any human or machine. Performance therefore depends entirely on the ability to manipulate symbolic representations of rules and consequences in real time.

Original Insights: The Strategic Generalization Bottleneck

These results illuminate a fundamental tension in current AI development. We have constructed systems of remarkable breadth, capable of discussing philosophy, writing poetry, and debugging code, yet these same systems struggle to master simple board games described moments earlier. This suggests that the scaling laws we have relied upon, increasing model parameters and training data, primarily improve knowledge compression and retrieval rather than the core cognitive capability of adapting to novel environments.

The 36% ceiling resembles performance patterns observed in human cognitive development. Children can learn new games with minimal examples, transferring understanding of concepts like turn-taking, resource management, or spatial reasoning across domains. Current LLMs appear to lack this fluid intelligence. When confronted with gg-bench tasks, they may engage in shallow pattern matching, confusing superficial similarities with structural equivalence, or fail to maintain consistent mental models of game state across moves.

This limitation has implications for the path toward artificial general intelligence. If strategic generalization requires architectural innovations beyond transformer-based next-token prediction, current research trajectories may be approaching diminishing returns. The benchmark suggests that we need mechanisms for explicit world-model construction, perhaps through neuro-symbolic integration, differentiable planning algorithms, or meta-learning approaches that enable rapid adaptation to new rule structures.

Furthermore, the comparison between reasoning models and base models indicates that simply increasing inference-time computation provides only marginal gains in this domain. The jump from 8% to 36% is significant, but the plateau near 36% across multiple reasoning architectures suggests that test-time scaling hits a ceiling when the underlying model lacks the representational capacity for certain types of structured reasoning.

Limitations and Future Directions

Several caveats temper these conclusions. First, gg-bench evaluates only one dimension of intelligence: competitive game playing. While games provide clean win conditions and strategic depth, they may not fully represent the complexity of real-world reasoning tasks involving partial information, stochastic environments, or social dynamics.

Second, the RL baseline, while reasonable, may exploit specific inductive biases in the generated games. If the LLM-generated games contain hidden structural regularities, the RL agents might capitalize on these in ways that do not generalize to broader intelligence. The paper does not extensively analyze the diversity or complexity distribution of the generated games, leaving open the possibility that the benchmark tests a narrower capability than intended.

Third, the text-based interface may disadvantage certain models. While the authors standardized input formats to include game descriptions, board states, and valid move lists, different architectures might process spatial or structural information more effectively through alternative modalities or representations.

Conclusion

The gg-bench evaluation serves as a necessary corrective to the narrative of inexorable progress in artificial intelligence. By demonstrating that even our most capable reasoning models win only a third of matches against specialized agents in novel games, the authors of "Measuring General Intelligence with Generated Games" highlight the persistent gap between pattern recognition and genuine strategic adaptation.

As the field moves forward, this benchmark suggests several priorities. We require architectures that can construct and manipulate explicit models of environmental dynamics, not merely predict likely next tokens. We need training regimes that emphasize few-shot adaptation to novel rule systems. Most importantly, we must recognize that scale alone may not bridge the gap between retrieval and reasoning.

The 36% ceiling stands as a challenge. Until systems can reliably master newly invented games with the facility of a human child, claims of approaching general intelligence remain premature. The path forward likely requires fundamental innovations in how machines represent and manipulate abstract rules, moving beyond the memorization of human knowledge toward the fluid construction of understanding.