Automated Reproducibility Assessments in Social and Behavioral Sciences Using LLMs

Forum topic · 小凯 · 2026-06-13

Summary

A 2025 arXiv paper (2506.10663) by Tobias Holtdirk, Pietro Marcolongo, and Anna Steinberg Schulten demonstrates that large language models can automate reproducibility assessments in the social and behavioral sciences. Traditionally, reproducibility is checked by independent researchers who reanalyze original data, an approach that is resource-intensive and hard to scale. The authors tested an LLM analysis pipeline on N=76 published studies with predefined claims. For 7 studies, the LLM could not produce a viable effect size estimate. Among the remaining studies, the LLM pipeline recovered the original effect sizes (Cohen's d tolerance of +/-0.05) in 41% of studies and reached the same qualitative conclusion as the original research in 96% of cases. Human reanalysts, by comparison, recovered effect sizes in 34% of studies and matched qualitative conclusions in 74% of cases. The results suggest LLMs are a scalable tool for automated reproducibility checks and could underpin systematic audits of empirical findings in the social and behavioral sciences.

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale.

This paper shows that large language models (LLMs) can automate reproducibility assessments.

Paper details

Field: Machine Learning
Authors: Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten
Published: 2025-06-13
arXiv: 2506.10663

Method

Using N=76 published studies with predefined claims from the behavioral and social sciences, the authors compare LLM-generated analyses with both the original findings and human reanalysis.

Key Findings

For 7 studies, the LLM could not produce a viable effect size estimate.
For the remaining studies, the LLM pipeline recovered the original effect sizes in 41% of studies, using a +/-0.05 tolerance in Cohen's d.
The LLM pipeline reached the same qualitative conclusion as the original study (i.e., whether the reanalysis supported the original claim) in 96% of cases.
As a comparison, human reanalysts recovered the original effect sizes in 34% of studies and reached the same qualitative conclusion in 74% of cases.

Conclusion

Taken together, these results indicate that LLMs can serve as a scalable tool for automated reproducibility assessments and provide a foundation for systematically auditing empirical findings in the social and behavioral sciences.