Nonstandard Errors in AI Agents: When Foundation Models Disagree on Empirical Facts

Introduction

The replication crisis in empirical research has taught us that science is far more path dependent than we once assumed. When twenty nine teams analyzed identical soccer data to test for racial bias in refereeing decisions, they produced twenty nine distinct analytical paths and often contradictory conclusions. When seventy neuroimaging teams tested the same hypotheses on identical fMRI datasets, they generated a constellation of results that defied simple synthesis. Menkveld et al. (2024) recently quantified this phenomenon as nonstandard errors (NSE), the excess variation in empirical findings that arises not from sampling noise but from researcher degrees of freedom: the choice of measures, specifications, and estimation paradigms that remain underspecified even when data and research questions are held constant.

As AI agents increasingly automate the evidence generating process, from data cleaning to hypothesis testing, a critical question emerges. Do foundation models eliminate this variability through algorithmic consistency, or do they replicate, or even amplify, the garden of forking paths that characterizes human research? In their recent working paper, "Nonstandard Errors in AI Agents," Ruijiang Gao and Steven Chong Xiao provide a rigorous empirical answer. By deploying 150 autonomous Claude Code agents to independently analyze identical NYSE TAQ data, they demonstrate that AI agents exhibit substantial nonstandard errors, with distinct "empirical styles" emerging across model families and convergence properties that challenge our intuitions about automated knowledge production.

The Experimental Architecture

Gao and Xiao replicate the Menkveld et al. (2024) experimental design with a fully automated twist. They task 150 autonomous agents (100 Claude Sonnet 4.6 and 50 Claude Opus 4.6) with testing six hypotheses regarding market quality trends in SPY ETF data from 2015 to 2024. Each agent operates as a complete empirical researcher: reading instructions, exploring the 66 gigabyte dataset containing seven billion rows of millisecond level trades and quotes, constructing measures, estimating trends, and producing 2,000 to 4,000 word research reports with accompanying code and figures. The entire pipeline runs without human intervention.

The results reveal alarming dispersion. For the market efficiency hypothesis (H1), estimates of the annual trend in efficiency range from negative 0.74% per year to positive 1.7% per year, with an interquartile range (IQR) of 2.43% per year. For trading volume trends (H4), the IQR reaches 10.69% per year, driven largely by a binary split between agents measuring dollar volume (approximately positive 6% per year) versus share volume (approximately negative 5% per year). Price impact estimates (H6) show similar divergence, with an IQR of 10.34% per year depending on whether agents compute trade level price impact or the Amihud illiquidity ratio.

The Structure of AI Variation

Crucially, the structure of AI nonstandard errors differs fundamentally from human variation. Human researchers in the Menkveld et al. study diverged across numerous estimation paradigms: level specifications versus log specifications versus relative changes (period over period ratios), each introducing distinct biases through Jensen's inequality. The AI agents, by contrast, show remarkable uniformity in estimation approach. All 150 agents independently select regression with a linear time trend, splitting primarily between level OLS (56%) and log OLS (44%). No agent employs relative changes or first differences, specifications that require explicit reasoning about growth rates and compounding.

Instead, AI NSE concentrates almost entirely in measure choice forks. The agents face discrete decisions about which construct to operationalize, and these choices dominate the variation. For market efficiency, Sonnet 4.6 agents prefer autocorrelation measures (87% adoption), level specifications, and daily frequency. Opus 4.6 agents universally select variance ratio measures (100%), prefer log OLS, and use monthly frequency 28% of the time. These preferences constitute stable "empirical styles" embedded in the foundation models' training, consistent with recent evidence that large language models encode systematic methodological biases in code generation.

This finding suggests that AI agents do not sample randomly from the space of analytical choices. Rather, they exhibit persistent, model specific heuristics that shape empirical output. The stochasticity of temperature sampling and prompt variation manifests not as uniform exploration of the methodological frontier, but as clustering around distinct attractor states defined by model architecture.

Peer Review versus Exemplar Exposure

Gao and Xiao implement a three stage feedback protocol to test mechanisms for convergence. Stage 1 establishes the baseline independent analysis. Stage 2 introduces AI peer review, where agents receive written critiques from other AI agents (both Sonnet and Opus evaluators). Stage 3 exposes agents to the five highest rated exemplar papers from Stage 1.

The results challenge conventional wisdom about scientific correction mechanisms. AI peer review has minimal effect on dispersion. The interquartile range remains essentially unchanged from Stage 1 to Stage 2, indicating that written feedback, even when generated by capable language models, fails to coordinate agents toward methodological consensus. The critiques appear to produce undirected movement, with agents adjusting their analyses in idiosyncratic directions that do not systematically reduce variance.

In stark contrast, exposure to top rated exemplar papers in Stage 3 triggers dramatic convergence. For measure families that converge, the IQR collapses by 80% to 99% between Stage 1 and Stage 3. This convergence operates through two channels: within family estimation tightening, where agents refine their implementation of a specific measure toward the exemplar standard, and between family switching, where agents abandon their initial measure choice entirely to adopt the exemplar's approach.

The asymmetry between deliberation (peer review) and imitation (exemplar exposure) carries profound implications. It suggests that AI agents, like human researchers, require concrete demonstrations of methodological norms rather than abstract critique to coordinate on analytical standards. The "show, don't tell" principle applies with surprising force to automated research systems.

Broader Implications and Limitations

These findings complicate the vision of fully automated policy evaluation systems, such as the Autonomous Policy Evaluation (APE) initiative, which promise to generate end to end economic research without human labor. If different AI agents produce meaningfully different conclusions from identical data, then automated research pipelines require explicit coordination mechanisms to ensure replicability. The choice of foundation model itself becomes a researcher degree of freedom that could drive policy relevant conclusions.

The study also highlights an underappreciated dimension of model alignment. While much attention focuses on value alignment and safety, methodological alignment, the coordination of empirical approaches across agent instances, may prove equally critical for scientific applications. The observation that Opus and Sonnet exhibit distinct empirical styles suggests that model selection is not merely a cost performance tradeoff but an epistemological commitment that shapes findings.

Several limitations warrant consideration. The study focuses on a specific domain (market microstructure) with well established measures and relatively structured data. Whether AI agents exhibit similar divergence in less codified fields, such as qualitative political science or experimental psychology, remains an open question. Additionally, the agents operated with specific prompting constraints and tool access; variations in the autonomous research framework might alter the NSE profile.

Conclusion

The work of Gao and Xiao demonstrates that nonstandard errors persist in the age of artificial intelligence. AI agents are not methodological monoliths that converge on objective truth when given identical inputs. Instead, they replicate the pluralism, and the associated uncertainty, of human research communities, complete with stable stylistic differences across model families and resistance to correction through critique alone.

As we integrate AI agents into the production of scientific knowledge and policy evidence, we must confront the reality that automation does not eliminate the garden of forking paths. It merely transplants it into silicon. The 80% to 99% convergence achieved through exemplar exposure offers a potential path forward, suggesting that curated methodological standards may prove more valuable than unrestricted agent autonomy. Future research must investigate whether these coordination mechanisms scale to complex, multi disciplinary questions, and whether the empirical styles we observe in current foundation models represent fixed properties or trainable dimensions of agent behavior. The challenge ahead lies not in removing human judgment from empirical research, but in understanding how algorithmic judgment reconfigures, and perhaps amplifies, the uncertainties that have always haunted the scientific enterprise.