Nothing Deceives Like Success: Social Learning and the Illusion of Understanding in Science
> Source: "Nothing Deceives Like Success: Social Learning and the Illusion of Understanding in Science" > Authors: Avery W. Louis (Department of Symbolic Systems, Stanford University), Marina Dubova (Santa Fe Institute) > Preprint: arXiv:2604.27188 [physics.soc-ph], April 2026 > Method: 2,301 agent-based simulations of collective theory building
The Core Problem: Science Is Blind Men Touching an Elephant
Science is cumulative and social—but scientists can never directly know how good their theories are. They evaluate theories only through proxy indicators: predictive accuracy, fit to existing data, parsimony, and internal consistency. History offers cautionary examples, from the early-20th-century "N-rays" reported by French physicists to the Alzheimer's amyloid plaque hypothesis, which dominated research for decades while clinical trials repeatedly failed.
Louis and Dubova call this the illusion of understanding: a systematic gap between a community's real-time judgment of theory quality and those theories' long-term true performance.
The Simulation Design
Each agent in the model behaves like a miniature scientist:
- Data collection: actively sampling observations from the environment
- Theory building: compressing data into representations (theories)
- Social learning: exchanging data and selectively adopting peers' theories
- Perceived success: how well a theory fits the agent's own sampled data
- Actual success: how well it fits the full ground truth
- N-rays (1903): French physicist Prosper-René Blondlot and dozens of colleagues "confirmed" a new form of radiation—until Robert Wood removed a key prism and Blondlot still saw it. The fraud was sincere self-deception.
- Amyloid plaque hypothesis: Since the 1990s, Alzheimer's research bet nearly everything on β-amyloid. Trial after trial failed—not because the direction was entirely wrong, but because the community converged prematurely on a local optimum.
- For individual researchers: confidence in one's own theories may be the biggest cognitive trap. Genuine humility means clear-eyed awareness of one's limits.
- For science policy: metrics like citations, h-index, and impact factors are proxies for "perceived success" that are systematically distorted under success bias. A system rewarding only apparent success quietly destroys the possibility of the next breakthrough.
- For scientific communities: protect space for heresy. Nearly every paradigm revolution was first seen as deviant. Premature convergence turns science into an echo chamber.
- Dunning, D. & Kruger, J. (1999). "Unskilled and Unaware of It." *Journal of Personality and Social Psychology*.
- Merton, R. K. (1968). "The Matthew Effect in Science." *Science*.
- Sloman, S. & Fernbach, P. (2017). *The Knowledge Illusion: Why We Never Think Alone*.
- Louis, A. W. & Dubova, M. (2026). "Nothing Deceives Like Success." arXiv:2604.27188.
The model distinguishes:
The gap between the two quantifies the illusion. Two social learning strategies were compared: success bias (interacting with peers whose theories look most successful) and community bias (interacting within one's own community).
Five Key Findings
1. The illusion is everywhere
Even without social learning, agents systematically overestimate their theories—a scientific version of the Dunning-Kruger effect. The illusion worsens sharply with problem complexity (correlation ρ=0.817, p<0.001).
2. Success bias amplifies the illusion
Under strong success bias, agents rapidly converge on a few "star theories," collapsing diversity. Perceived-vs-actual success gaps are significantly larger in high-bias conditions (Mann-Whitney U=143,688, p<0.001). Success bias does not create the illusion—it amplifies and entrenches it.
3. Success bias can only lift the floor, not raise the ceiling
Regression analysis shows success bias significantly helps the worst-performing theories (bottom 25%), has weak effects on middle-tier theories, and is negative for the best theories (top 25%). It filters out bad theories but cannot discover new, better ones—which requires diversity, not convergence.
4. Rationally optimizing perceived success degrades real performance
When agents freely tune their strategies to maximize perceived success, they settle on high success bias plus strong community clustering—the combination most damaging to actual success. The resulting knowledge inequality strikingly resembles citation distributions in real scientific communities, suggesting the Matthew Effect may partly be a natural product of amplified cognitive illusions in social learning.
5. Evaluation accuracy only matters under high success bias
The community's ability to correctly identify good theories (measured by CrPC—Centrality and Relative-Performance Correlation) is irrelevant without success bias, but critical when it is strong: a single misjudged star theory gets magnified through mass imitation. This creates a vicious cycle—stronger success bias increases dependence on evaluation accuracy while simultaneously undermining it.
Historical Echoes
In both cases, the problem was not stupidity or insufficient data, but a community that *believed it understood when it did not*.
Implications
The authors conclude that progress may depend less on choosing the right theory and more on maintaining the diversity of explanations worth remembering—in a world richer, stranger, and less perfect than any single theory can capture.