Automated Reproducibility Assessments in Social and Behavioral Sciences Using LLMs

Forum topic · 小凯 · 2026-06-15

Summary

A new arXiv paper (2606.13670) by Tobias Holtdirk and colleagues from LMU Munich and ETH Zurich shows that large language models (LLMs) can automate reproducibility assessments in the social and behavioral sciences. Traditional reproducibility checks require independent researchers to reanalyze original data, which is resource-intensive and hard to scale. The team tested an LLM pipeline on 76 published studies with predefined claims. For 7 studies the LLM could not produce a viable effect size estimate. Among the rest, the pipeline recovered original effect sizes in 41% of studies (Cohen's d within a +/-0.05 tolerance) and reached the same qualitative conclusion as the original study in 96% of cases. Human reanalysts, by comparison, recovered effect sizes in 34% of studies and agreed on qualitative conclusions in 74% of cases. The results suggest LLMs are a scalable tool for systematic auditing of empirical findings.

Overview

Field: Machine Learning / Computational Social Science
Authors: Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, Stefan Feuerriegel
arXiv: 2606.13670

Abstract (translated from the post)

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. This paper demonstrates that large language models (LLMs) can automate reproducibility assessments.

Using N=76 published studies with predefined claims from the behavioral and social sciences, the authors compare LLM-generated analyses with the original findings and with human reanalysis.

Key Findings

For 7 studies, the LLM could not produce a viable effect size estimate.
For the remaining studies, the LLM pipeline recovered the original effect sizes in 41% of studies, using a +/-0.05 tolerance in Cohen's d.
The LLM pipeline reached the same qualitative conclusion as the original study (i.e., whether the reanalysis supported the original claim) in 96% of cases.
In comparison, human reanalysts recovered the original effect size in 34% of studies and reached the same qualitative conclusion in 74% of cases.

Conclusion

The results suggest that LLMs can serve as a scalable tool for automated reproducibility assessments, providing a foundation for systematically auditing empirical findings in the social and behavioral sciences.