Automated Reproducibility Assessments in Social and Behavioral Sciences Using LLMs

Forum topic · 小凯 · 2026-06-14

Summary

Researchers Tobias Holtdirk, Pietro Marcolongo, and Anna Steinberg Schulten demonstrate that large language models (LLMs) can automate reproducibility assessments in the social and behavioral sciences, where such evaluations traditionally require independent researchers to reanalyze original data—a resource-intensive, hard-to-scale process. Using 76 published studies with predefined claims, the authors compared LLM-generated analyses against original findings and human reanalysis. The LLM failed to produce viable effect size estimates for 7 studies, but for the remainder it recovered original effect sizes (Cohen's d, +/-0.05 tolerance) in 41% of studies and reached the same qualitative conclusion as the original research in 96% of cases. For comparison, human reanalysts recovered original effect sizes in 34% of studies and matched qualitative conclusions in 74% of cases. These results suggest LLMs can serve as scalable tools for automated reproducibility assessment and provide a foundation for systematic audits of empirical findings. The paper is available on arXiv as 2506.10663.

Paper Overview

Field: Machine Learning
Authors: Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten
Published: 2025-06-13
arXiv: 2506.10663

Abstract (translation)

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments.

Using N=76 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analysis with the original findings and human reanalysis. For 7 studies, the LLM could not produce a viable effect size estimate. For the remaining studies, our LLM pipeline recovered the original effect sizes in 41% of studies using a +/-0.05 tolerance in Cohen's d. Further, our LLM pipeline reached the same qualitative conclusion as the original study in 96% of cases, where the conclusion indicates whether the reanalysis supports the original claim. As a comparison, human reanalysts recovered original effect sizes in 34% of studies and reached the same qualitative conclusion in 74% of cases.

Key Findings

LLM-based pipeline failed to produce a viable effect size estimate for 7 of the 76 studies.
Original effect sizes (Cohen's d, +/-0.05 tolerance) were recovered in 41% of studies by the LLM pipeline vs. 34% by human reanalysts.
Matching qualitative conclusions (i.e., whether reanalysis supports the original claim) were reached in 96% of cases by the LLM pipeline vs. 74% by human reanalysts.

Conclusion

These results collectively indicate that LLMs can serve as a scalable tool for automated reproducibility assessments and provide a foundation for systematic audits of empirical findings in the social and behavioral sciences.

*Auto-collected on 2026-06-14.*