AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science
| Autoři |
BRODEUR Abel VALENTA David MARCOCI Alexandru APARICIO Juan P. MIKOLA Derek BARBARIOLI Bruno ALEXANDER Rohan DEER Lachlan STAFFORD Tom VILHUBER Lars BENSCH Gunther MOTOKI Fabio ABDELHADY Mohamed ABDELMOULA Yousra BAKI Ghina Abdul AGUIRRE Tomás AIYER Sriraj AKHTAR Shumi AKHTAR Farida ALBADA Melle R. ALTMAN Micah ANGENENDT David ARJMANDI LARI Zahra DE LEÓN TEJADA Jorge Armando ARANA David Rodriguez ASANOV Igor NOHA Anastasiya-Mariya ASHONG Rebecca AUER Tobias BAHAMONDE-BIRKE Francisco J. BAKER Bradley J. BARTRAM Söhnke M BAO Dongqi BATINOVIC Lucija BATISTONI Tommaso BEEDER Monica BELAND Louis-Philippe GERO BIENZ Carsten ARYANTO Christ Billy BOLIBAUGH Cylcia BONANDER Carl BRAVO Ramiro BRONNIKOV Egor BRUNS Stephan BULISKERIA Nino CAICEDO-SILVA Sara CALEF Andrea SEBASTIAN CANO ARIAS Juan A CASTILLO ALVAREZ Gustavo CAULKER Solomon CEPENAS Simonas CHATTON Arthur CHEN Zirou CHIOMA EWURUM Ngozi CIOCÎRLAN Anda-Bianca CLOUTH Felix J. COLLINS Jason COOK Nikolai CORNEJO Cesar CRAVEIRO Joao CRÉCHET Jonathan CUI Jing CHALIL VAYALABRON Niveditha CZYMARA Christian BERMÚDEZ JARAMILLO Carlos Daniel DATTA Hannes DENOO Lien DHALIWAL Arshia DHAMEJA Nency DJEMAI Elodie DUJEANCOURT Erwan DÜNDAR Ugurcan DUPREY Thibaut EISSA Yasmine EL FASSI Youssef EL FASSI Ismail ELLIS Keaton ELMINEJAD Ali ELSHERIF Mahmoud EMIRMAHMUTOGLU Aysil ETINGIN-FRATI Giulian EZE Emeka DOLLBAUM Jan Fabian FELD Jan FELIPE RENGIFO JARAMILLO Andres FENIG Guidon FERNANDES Victoria FIALA Lenka FINK Lukas FIROUZJAEIANGALOUGAH Mojtaba FISH Sara FITZGERALD Jack FORSHAW Rachel FORTIER-CHOUINARD Alexandre FRÉGET Louis FRESE Joris GABANI Jacopo GALLEGOS Sebastian GAMILL Max C. GÁSPÁR Attila GAURIOT Romain GAVRILOVA Evelina GERALDES Diogo CANTONE Giulio Giacomo GIBSON Grant GOLDSCHMITT Dirk GOURDON-KANHUKAMWE Amélie GREGOR DE VARDA Andrea GRIGORYEVA Idaliya GUGUSHVILI Alexi FLETCHER Aaron H A HABERMANN Florian HABLICSEK Márton HADDAD Joanne HALL Jonathan D. HAMMAR Olle HASSOUNEH Malek HAUSLADEN Carina I. HENDRIKSE Sophie C F HEPPLEWHITE Matthew HO Anson T Y HOGAN-HENNESSY Senan HOWLEY Elliot HUANG Gaoyang HULSTAERT Héloise ILCHOVSKA Zlatomira G. JAIMES SANTAMARIA Paola JAKOBSSON Niklas JANSSON Joakim JAROSZ Ewa JEBELI Hossein JIANG Yanchen JUNAID Hiba KALLURAYA Rohan KARIM Sunny KELLY Edmund KIMEL Eva KINGSUWANKUL Sorravich KLOTZBÜCHER Valentin KRÄHMER Daniel KRUMINAS Pijus KRUUS Nicholas KUJANSUU Essi KURZ Christoph F. KÜSTER Stephan LEE-WHITING Blake LEWANDOWSKI Felix LI Tongzhe LI Ruoxi LIU Dan LIU Jiacheng LO Helix LOTER Katharina MACEDO DIAS Felipe MADAN Christopher R. MÄDER Nicolas MANDAS Marco MANTILLA Cesar MARCUS Jan MARINO FAGES Diego MARTIN Xavier MCWAY Ryan MEDINA-GASPAR Daniel MENG Sisi MENG Lingyu MERZ Simon MILLER Alex P. MIRABEL Thibault MISHRA Dibya Deepta MISHRA Sumit MOGES Belay W. MOHANDES MOJARRAD Morteza MOHNEN Myra MORIN Louis-Philippe MUEHLENBACHS Lucija MULLIN Gastón MUSULAN Andreea MUZZI Sara MYERS James A C NEUBAUER Florian NGUYEN Tuan NIAZI Ali NORDSTROM Ardyn NOWAK Bartłomiej O'HABIB Daneal ÖLKERS Tim ONG Justin OROZCO CASTIBLANCO Valeria ÖZAK Ömer OZKES Ali I. PAASO Mikael PANDEY Shubham PAPAZOGLOU Varvara PENHEIRO Romeo PHAM Linh PHIELER Ulrike PÜTZ Peter QI Quan QIU Jingyi REIN Manuel T. REINSTEIN David A. REPO Juuso RUDOLF Nicolas SAHA Shree SAKA Orkun SAPONARO Chiara SATOR Georg SCHOENMAKERS Martijn SERI Raffaello SHAH Meet SIBILLE Paul SIEMROTH Christoph SKAVYSH Vladimir SLATER Ben SONG Wenting STAUBLI Stefan STEINDL Tobias WAONGO Nomwendé Steven STOTT Paul STROBEL Stephenson SUDHAHARAN Roshini SUN Pu SWAIN Scott D. TALAVERA Oleksandr TANTIANGCO Hanz M. TARASENKO Georgy TARLINTON Boyd TARRAF Mariam TEOH Ken THÉRIAULT Rémi THOMPSON Bethan TIAN Tonghui TIAN Wenjie TOLANI Emmanuel BORGEN Nicolai TOPSTAD BORGEN Solveig TORRALBA Javier VELEZ-OSPINA Carolina MAK Man Wai WALLRICH Lukas WANG Zeyang WARD Leah WEBB Matthew D. WEBB Duncan WEBER Bryan S. WEBER Christoph WENG Wei-Chien WESTHEIDE Christian WILKINSON Tom WONG Kwong-Yu WROŃSKI Marcin WU Zhuangchen WU Qixia WU Victor Y. XIAO Bohan XU Feihong XU Cong YADAV Pranav YANG CHOU Yu YAP Luther YAZBECK Myra YAO Bo ZAGRODZKA Zuzanna ZAHRA Tahreen ZANEVA Mirela ZHANG Xiaomeng ZHAO Ziwei ZHONG Han ZIRGULIS Aras ZOU Jiacheng ZOUTMAN Floris ZOZOUNGBO Christelle |
|---|---|
| Rok publikování | 2026 |
| Druh | Recenzovaný odborný článek |
| Časopis / Zdroj | Proceedings of the National Academy of Sciences of the United States of America |
| Fakulta / Pracoviště MU | |
| Citace | |
| www | https://www.pnas.org/doi/10.1073/pnas.2524747123 |
| Doi | https://doi.org/10.1073/pnas.2524747123 |
| Klíčová slova | AI; large language models; reproducibility |
| Přiložené soubory | |
| Popis | Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification. |