The Illusion of Strategic Competence: Counterfactual Evaluation Exposes Memorization in Large Language Models

Recent evaluations of Large Language Models (LLMs) in competitive and cooperative environments have revealed a troubling discrepancy. While these models can often articulate optimal strategies and equilibrium concepts when prompted, their actual performance in interactive game theoretic settings frequently deviates from rational predictions. In their paper "Evaluating Counterfactual Strategic Reasoning in Large Language Models," Georgousis and colleagues from the National Technical University of Athens provide compelling evidence that this gap between verbalization and execution stems from a fundamental limitation: LLMs rely heavily on memorized patterns and lexical associations rather than genuine reasoning about incentive structures.

The research introduces a rigorous framework for distinguishing statistical pattern matching from structural reasoning by testing models on counterfactual variants of canonical games. By systematically perturbing payoff matrices and permuting action labels, the authors expose how quickly strategic performance deteriorates when surface level cues no longer align with pre training distributions. This work carries significant implications for the deployment of autonomous agents in multi agent systems, where adaptability to novel incentive structures is not merely desirable but essential for reliable operation.

Counterfactual Methodology and Game Theoretic Testbeds

Georgousis et al. select two complementary experimental settings to probe different dimensions of strategic reasoning. The Prisoner’s Dilemma (PD) examines cooperation under conflicting incentives, testing whether models can sustain mutual cooperation in repeated interactions or whether they default to defection regardless of temporal dynamics. The Rock Paper Scissors (RPS) framework assesses stochastic reasoning, evaluating whether models can implement mixed strategy equilibria or whether they fall into exploitable deterministic patterns.

The critical innovation lies in the construction of counterfactual game pairs. In the default PD, mutual cooperation yields higher collective payoffs than mutual defection, yet defection dominates in single shot interactions. The counterfactual variants alter this structure by permuting payoff values or swapping action labels. For instance, the action labeled "Cooperate" might be mapped to the payoff structure traditionally associated with defection, or the entire payoff matrix might be inverted. Similarly, in RPS, the cyclical dominance relations (Rock beats Scissors, Scissors beats Paper, Paper beats Rock) are preserved but the labels are permuted, so that what was previously called "Rock" now beats "Paper" instead.

This experimental design is elegant in its simplicity. If LLMs genuinely compute equilibria based on incentive structures, performance should remain invariant under these isomorphic transformations. However, if models rely on memorized associations between words like "Cooperate" and prosocial outcomes, or between "Rock" and a particular strategic context, then counterfactual permutations should induce significant performance degradation.

Empirical Findings: Lexical Bias and Strategic Brittleness

The results demonstrate that LLMs exhibit substantial sensitivity to these counterfactual manipulations. When action labels are swapped or payoff matrices are inverted, models show marked declines in strategic performance. In the Prisoner’s Dilemma, models tend to associate the word "Cooperate" with positive social outcomes regardless of the actual underlying payoffs. When the label "Cooperate" is attached to the dominated strategy (the one yielding lower payoffs), models continue to select it at rates inconsistent with rational choice, suggesting they are responding to lexical valence rather than computing expected utilities.

In Rock Paper Scissors, the models fail to implement the Nash equilibrium mixed strategy of uniform randomization. Instead, they display predictable behavioral patterns that sophisticated opponents could exploit. When action labels are permuted in counterfactual RPS variants, models do not successfully map the new labels to the underlying cyclical dominance structure. Rather, they appear to carry over response patterns from the standard labeling, indicating that their strategic "knowledge" is bound to specific surface form representations rather than abstract game theoretic properties.

The evaluation framework employs multiple behavioral metrics to capture these dynamics, including measures of consistency, exploitability, and temporal adaptation. The authors track how strategies evolve over repeated interactions, revealing that models often show inductive biases toward short term outcomes and exhibit path dependencies that rational agents would avoid. This brittleness persists despite the models' ability to correctly describe Nash equilibria when asked directly, highlighting a dissociation between explicit strategic knowledge and embodied decision making.

Structural Generalization and the Limits of Pattern Matching

These findings illuminate a broader challenge in artificial intelligence research: the distinction between statistical interpolation and causal reasoning. Current LLMs appear to operate as sophisticated pattern completion engines that map game descriptions to behavioral scripts encountered during training. When the mapping between linguistic description and strategic structure aligns with pre training data, performance appears competent. When this mapping is perturbed, the lack of underlying causal models of incentives becomes apparent.

This limitation has particular urgency for multi agent deployment scenarios. Real strategic environments rarely present themselves in canonical forms. Negotiation contexts feature shifting payoff structures, novel action spaces, and redefined social preferences. If an LLM cannot generalize when "Cooperation" becomes the selfish option or when standard dominance relations are inverted, its reliability in autonomous negotiation or economic decision making is questionable. The model is not computing equilibria; it is retrieving statistical patterns that happen to correlate with rational behavior in common contexts.

The paper also contextualizes these findings within the broader literature on counterfactual reasoning in LLMs. Prior work has shown that models struggle when hypothetical premises contradict real world knowledge, and that scaling model size does not necessarily improve counterfactual robustness. Georgousis et al. extend this observation to interactive, sequential settings, demonstrating that the brittleness persists even when stakes are clearly defined and feedback is immediate. Notably, the authors observe that chain of thought prompting and instruction tuning, which often improve performance on static reasoning tasks, fail to resolve these strategic inconsistencies, suggesting that the limitations are architectural rather than superficial.

Limitations and Trajectories for Future Research

While this work provides crucial diagnostic tools, certain limitations warrant acknowledgment. The experimental framework focuses on two player, perfect information games with discrete action spaces and well defined payoff matrices. Real world strategic interactions typically involve incomplete information, continuous strategy spaces, coalition dynamics, and evolving social contexts. Whether the observed brittleness in PD and RPS generalizes to these more complex environments remains an open empirical question.

Furthermore, the study primarily evaluates models through their textual outputs in prompted interactions. The extent to which these findings apply to LLMs fine tuned specifically for reinforcement learning from human feedback, or to models augmented with external symbolic reasoning modules, requires additional investigation. The counterfactual evaluation framework itself, however, provides a template that could readily extend to more complex game theoretic environments.

Conclusion

The research by Georgousis and colleagues offers a sobering assessment of current LLM capabilities in strategic domains. By demonstrating that performance collapses under counterfactual permutations that preserve underlying game structures while altering surface representations, they provide strong evidence that these models lack genuine strategic reasoning. Instead, LLMs appear to rely on fragile associations between lexical items and behavioral templates.

For the field to progress toward autonomous agents capable of reliable negotiation and cooperation, we must develop architectures that compute equilibria based on explicit representations of incentives rather than retrieving patterns from training data. Future research should investigate hybrid approaches that combine neural language models with explicit game theoretic solvers, or training regimes that explicitly penalize reliance on surface level heuristics. Until such developments materialize, we should remain cautious about deploying LLMs in high stakes strategic environments where counterfactual reasoning is not a theoretical curiosity but a practical necessity. The path from pattern matching to structural understanding remains one of the central challenges in building truly intelligent artificial agents.