The Thought Experiment
If you give the same data and the same questions to 150 identical AI agents, will they reach the same conclusions? Intuition says yes—same model, same algorithm, same input. The real-world answer: no. They disagree.
Experiment Design: An "Independent Survey" of Market Quality
In early 2026, researchers collected ten years of NYSE trading data for SPY (the S&P 500 ETF) and posed six market-quality hypotheses:
1. Has market liquidity changed over the decade? 2. What is the trend in volatility? 3. Has price discovery efficiency improved? 4. Are trading costs rising or falling? 5. How has market depth changed? 6. Has information asymmetry improved?
They deployed 150 independent Claude Code agents (AI coding assistants based on Claude 3.7), each independently testing the same six hypotheses. Same data, same questions, same model—the only difference: the agents ran in isolation, with no communication between them.
Shocking Finding: AIs Also "Freestyle"
The 150 agents reached drastically different conclusions. Some found liquidity significantly declined, some found no significant change, and some found it increased. Same question, opposite answers.
Divergence in Measurement Methods
Analyzing the agents' workflows, researchers found a key issue: they chose different measurement methods. For liquidity, some used bid-ask spread, some price impact, some order book depth, and others effective spread, realized spread, or quoted-slope measures. Each is defensible, but each measures "liquidity" differently.
> It's like asking "is this city wealthy?" You can look at GDP per capita, income, luxury car density, or housing prices—each indicator reflects wealth, but they can yield different answers.
Divergence in Units
Some agents measured volume in dollars, others in shares. If a stock's price doubles from $100 to $200 with unchanged volume: dollar volume doubles ("market more active!"), while share volume is flat ("activity stable"). One phenomenon, two readings.
Divergence in Time Windows
Some agents analyzed daily data, others weekly or monthly averages; some used sliding windows. Time granularity strongly affects trend judgments.
"Empirical Style": Different Model Families, Different Tastes
Comparing Claude Sonnet 4.6 and Claude Opus 4.6, researchers found that different model families have stable "empirical styles"—systematic, non-random preferences for certain types of methods, akin to how human statisticians prefer OLS, Bayesian, or non-parametric approaches. These preferences likely stem from training data: models that "read" more papers using a given method tend to inherit that methodological taste.
What Are "Nonstandard Errors"?
In traditional statistics, standard error quantifies uncertainty from random sampling. But this study found that even with the complete dataset (infinite effective sample size), huge uncertainty remains. This variance comes not from insufficient data but from the analyst's choices—method, units, time window. This is nonstandard errors (NSE): variation arising from analysts' subjective choices rather than data randomness. The concept was first used for human researchers: 2019 work showed different teams analyzing the same data reach very different results—not through cheating, but through different "reasonable" choices. This study extends NSE to AI.
What AI's Nonstandard Errors Mean
1. AI Is Not "Objective"
We often assume AI is objective—same data should yield the same result. This research shows AIs also disagree. They don't cheat, but they accumulate "reasonable but different" choices that compound into divergent conclusions.2. Risks of Automated Research
If AI does research independently, you may get 150 different answers. Perhaps all are right from some angle; perhaps all are wrong. The essence of empirical research: there is no single correct answer, only better or worse choices.3. The Value of Peer Review
The experiment used a three-stage protocol:- Stage 1: Agents work independently, producing 150 divergent results.
- Stage 2: Agents peer-review each other's papers. Result: AI peer review barely reduced dispersion. Knowing other methods and conclusions exist rarely changed their own approach.
- Stage 3: Agents read "high-scoring exemplars." This was the key: after exposure to high-quality examples, dispersion dropped 80–99% (within converging method families).
Humans vs. AI: Who Disagrees More?
In 2019, 29 human research teams analyzing the same data produced 29 different results. Now, 150 AI agents produced 150 different results. AI is no more "objective" than humans. Why? The space of discretionary choices is enormous—from data cleaning to variable definitions to model selection to presentation. These are judgment calls, not math, and AI inherits them from human training data.
A Paradox: Certainty vs. Diversity
We want research to be deterministic (same data → same conclusion) yet also pluralistic (different perspectives reveal different facets). If all AIs used the same methods, we'd lose diversity; if all used different ones, we'd lose comparability. The researchers offer no balance point, but propose transparency: if every AI documents its analytical choices—what method, why, and what alternatives were considered—we can understand and evaluate differences rather than blindly chasing a single correct answer.
Future: Collaborative AI Research
The authors propose collaborative multi-agent research: one agent proposes hypotheses, another challenges them, a third tries different methods, a fourth synthesizes. This could yield meta-level insights—not about "is market quality declining," but "why do we disagree about this question?" That is the essence of science: answering not just "what is" but "how do we know."
Conclusion: Machines as a Mirror
Why did 150 AIs with the same data and model reach different conclusions? Because research is never purely algorithmic. Even the most "objective" data analysis is filled with judgment—inherited, in this case, from humans. The study's value lies in quantifying the disagreement, tracing its sources, and revealing a previously unrecognized space of choices. AI's greatest value in science may be as a mirror, exposing hidden assumptions, unconscious choices, and our self-assured "objectivity." When 150 AIs give 150 answers, the right question isn't "which AI is right?" but:
"Why does this question have so many different answers?"
References
1. Nonstandard Errors in AI Agents (2026). arXiv preprint. The core paper of this study. 2. Camerer, C. F., et al. (2016). "Evaluating replicability of laboratory experiments in economics." *Science*. 3. Silberzahn, R., et al. (2018). "Many analysts, one data set: Making transparent how variations in analytic choices affect results." *Advances in Methods and Practices in Psychological Science*. 4. Botvinik-Nezer, R., et al. (2020). "Variability in the analysis of a single neuroimaging dataset by many teams." *Nature*. 5. Breznau, N., et al. (2022). "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty." *PNAS*.
*"The value of science lies not in eliminating uncertainty, but in understanding its sources."*