Examining the generalizability of research findings from archival data

Andrew Delios; Elena Giulia Clemente; Tao Wu; Hongbin Tan; Yong Wang; Michael Gordon; Domenico Viganola; Z Chen; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Eric Luis Uhlmann; Al-Aziz AM Abd; Ajay T. Abraham; Jais Trojan; Matus Adamkovic; Elena Agadullina; Jungsoo Ahn; Cinla Akinci; Handan Akkas; David Albrecht; Shilaan Alzahawi; Marcio Amaral-Baptista; Rahul Anand; Kevin Francis U. Ang; Frederik Anseel; John Jamir Benzon R. Aruta; Mujeeba Ashraf; Bradley J. Baker; Xueqi Bao; Ernest Baskin; Hanoku Bathula; Christopher W. Bauman; Jozef Bavolar; Secil Bayraktar; Stephanie E. Beckman; Aaron S. Benjamin; Stephanie E.V. Brown; Jeffrey Buckley; Ricardo E. Buitrago; Jefferson L. Bution; Nick Byrd; Clara Carrera; Eugene M. Caruso; M Chen; L Chen; Eyyub Ensari Cicerali; Eric D. Cohen; Marcus Crede; J Cummins; L Dahlander; DP Daniels; LL Daskalo; IGJ Dawson; MV Day; E Dietl; A Domurat; J Dsilva; CD Plessis; DI Dubrov; S Edris; CT Elbaek; MM Elsherif; TR Evans; MR Fellenz; S Fiedler; M Firat; R Freitag; RA Furrer; R Gautam; DK Gautam; B Gearin; S Gerschewski; O Ghasemi; Z Ghasemi; A Ghosh; C Giani; MH Goldberg; M Goswami; L Graf-Vlachy; JA Griffith; D Grigoryev; J Gu; R H; AL Hadida; AC Hafenbrack; S Hafenbrädl; JJ Hammersley; H Han; JL Harman; A Hartanto; Jan K. Woike

doi:10.1073/pnas.2120377119

Examining the generalizability of research findings from archival data

Andrew Delios^*
, Elena Giulia Clemente
, Tao Wu
, Hongbin Tan
, Yong Wang
, Michael Gordon
, Domenico Viganola
, Z Chen
, Anna Dreber
, Magnus Johannesson
, Thomas Pfeiffer
, Eric Luis Uhlmann^*
, Al-Aziz AM Abd
, Ajay T. Abraham
, Jais Trojan
, Matus Adamkovic
, Elena Agadullina
, Jungsoo Ahn
, Cinla Akinci
, Handan Akkas

David Albrecht, Shilaan Alzahawi, Marcio Amaral-Baptista, Rahul Anand, Kevin Francis U. Ang, Frederik Anseel, John Jamir Benzon R. Aruta, Mujeeba Ashraf, Bradley J. Baker, Xueqi Bao, Ernest Baskin, Hanoku Bathula, Christopher W. Bauman, Jozef Bavolar, Secil Bayraktar, Stephanie E. Beckman, Aaron S. Benjamin, Stephanie E.V. Brown, Jeffrey Buckley, Ricardo E. Buitrago, Jefferson L. Bution, Nick Byrd, Clara Carrera, Eugene M. Caruso, M Chen, L Chen, Eyyub Ensari Cicerali, Eric D. Cohen, Marcus Crede, J Cummins, L Dahlander, DP Daniels, LL Daskalo, IGJ Dawson, MV Day, E Dietl, A Domurat, J Dsilva, CD Plessis, DI Dubrov, S Edris, CT Elbaek, MM Elsherif, TR Evans, MR Fellenz, S Fiedler, M Firat, R Freitag, RA Furrer, R Gautam, DK Gautam, B Gearin, S Gerschewski, O Ghasemi, Z Ghasemi, A Ghosh, C Giani, MH Goldberg, M Goswami, L Graf-Vlachy, JA Griffith, D Grigoryev, J Gu, R H, AL Hadida, AC Hafenbrack, S Hafenbrädl, JJ Hammersley, H Han, JL Harman, A Hartanto, Jan K. Woike

^*Corresponding author for this work

National University of Singapore
Stockholm School of Economics
The Chinese University of Hong Kong, Shenzhen
Tongji University
Xi'an Jiaotong University
Massey University
World Bank
University of Innsbruck
INSEAD
Seattle University
Keele University
University of Presov in Presov
Higher School of Economics
Western University
University of St Andrews
Ankara Science University
Maastricht University
Stanford University
University Institute of Lisbon
Aarhus BSS
Value Care Health Systems
University of New South Wales
De La Salle University-Manila
University of the Punjab
Temple University
Saint Josephs University
The University of Auckland
University of California at Irvine
P. J. Safarik University
TBS Business School
Madison College
University of Illinois at Urbana-Champaign
Texas A&M University
KTH Royal Institute of Technology
Universidad del Rosario
Universidade de São Paulo
Stevens Institute of Technology
University of California at Los Angeles
Nisantasi Universitesi
Universidade Estadual de Campinas
Iowa State University

Research output: Contribution to journal › Article › peer-review

21 Downloads (Pure)

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Original language	English
Number of pages	0
Journal	Proceedings of the National Academy of Sciences
Volume	119
Issue number	30
Early online date	19 Jul 2022
DOIs	https://doi.org/10.1073/pnas.2120377119
Publication status	Published - 19 Jul 2022

Access to Document

10.1073/pnas.2120377119

PnasFinal published version, 843 KB

Cite this

Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., Pfeiffer, T., Uhlmann, E. L., Abd, A.-A. AM., Abraham, A. T., Trojan, J., Adamkovic, M., Agadullina, E., Ahn, J., Akinci, C., ... Woike, J. K. (2022). Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences, 119(30). https://doi.org/10.1073/pnas.2120377119

@article{63c65e28e35d49b2afa004375723a569,

title = "Examining the generalizability of research findings from archival data",

abstract = "This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45\% of the reproductions returned results matching the original reports together with 55\% of tests in different spans of years and 40\% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84\% emerged in other available time periods and 57\% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.",

author = "Andrew Delios and Clemente, \{Elena Giulia\} and Tao Wu and Hongbin Tan and Yong Wang and Michael Gordon and Domenico Viganola and Z Chen and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and Uhlmann, \{Eric Luis\} and Abd, \{Al-Aziz AM\} and Abraham, \{Ajay T.\} and Jais Trojan and Matus Adamkovic and Elena Agadullina and Jungsoo Ahn and Cinla Akinci and Handan Akkas and David Albrecht and Shilaan Alzahawi and Marcio Amaral-Baptista and Rahul Anand and Ang, \{Kevin Francis U.\} and Frederik Anseel and Aruta, \{John Jamir Benzon R.\} and Mujeeba Ashraf and Baker, \{Bradley J.\} and Xueqi Bao and Ernest Baskin and Hanoku Bathula and Bauman, \{Christopher W.\} and Jozef Bavolar and Secil Bayraktar and Beckman, \{Stephanie E.\} and Benjamin, \{Aaron S.\} and Brown, \{Stephanie E.V.\} and Jeffrey Buckley and Buitrago, \{Ricardo E.\} and Bution, \{Jefferson L.\} and Nick Byrd and Clara Carrera and Caruso, \{Eugene M.\} and M Chen and L Chen and Cicerali, \{Eyyub Ensari\} and Cohen, \{Eric D.\} and Marcus Crede and J Cummins and L Dahlander and DP Daniels and LL Daskalo and IGJ Dawson and MV Day and E Dietl and A Domurat and J Dsilva and CD Plessis and DI Dubrov and S Edris and CT Elbaek and MM Elsherif and TR Evans and MR Fellenz and S Fiedler and M Firat and R Freitag and RA Furrer and R Gautam and DK Gautam and B Gearin and S Gerschewski and O Ghasemi and Z Ghasemi and A Ghosh and C Giani and MH Goldberg and M Goswami and L Graf-Vlachy and JA Griffith and D Grigoryev and J Gu and R H and AL Hadida and AC Hafenbrack and S Hafenbr{\"a}dl and JJ Hammersley and H Han and JL Harman and A Hartanto and Woike, \{Jan K.\}",

year = "2022",

month = jul,

day = "19",

doi = "10.1073/pnas.2120377119",

language = "English",

volume = "119",

journal = "Proceedings of the National Academy of Sciences",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "30",

}

Delios, A, Clemente, EG, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Uhlmann, EL, Abd, A-AAM, Abraham, AT, Trojan, J, Adamkovic, M, Agadullina, E, Ahn, J, Akinci, C, Akkas, H, Albrecht, D, Alzahawi, S, Amaral-Baptista, M, Anand, R, Ang, KFU, Anseel, F, Aruta, JJBR, Ashraf, M, Baker, BJ, Bao, X, Baskin, E, Bathula, H, Bauman, CW, Bavolar, J, Bayraktar, S, Beckman, SE, Benjamin, AS, Brown, SEV, Buckley, J, Buitrago, RE, Bution, JL, Byrd, N, Carrera, C, Caruso, EM, Chen, M, Chen, L, Cicerali, EE, Cohen, ED, Crede, M, Cummins, J, Dahlander, L, Daniels, DP, Daskalo, LL, Dawson, IGJ, Day, MV, Dietl, E, Domurat, A, Dsilva, J, Plessis, CD, Dubrov, DI, Edris, S, Elbaek, CT, Elsherif, MM, Evans, TR, Fellenz, MR, Fiedler, S, Firat, M, Freitag, R, Furrer, RA, Gautam, R, Gautam, DK, Gearin, B, Gerschewski, S, Ghasemi, O, Ghasemi, Z, Ghosh, A, Giani, C, Goldberg, MH, Goswami, M, Graf-Vlachy, L, Griffith, JA, Grigoryev, D, Gu, J, H, R, Hadida, AL, Hafenbrack, AC, Hafenbrädl, S, Hammersley, JJ, Han, H, Harman, JL, Hartanto, A & Woike, JK 2022, 'Examining the generalizability of research findings from archival data', Proceedings of the National Academy of Sciences, vol. 119, no. 30. https://doi.org/10.1073/pnas.2120377119

TY - JOUR

T1 - Examining the generalizability of research findings from archival data

AU - Delios, Andrew

AU - Clemente, Elena Giulia

AU - Wu, Tao

AU - Tan, Hongbin

AU - Wang, Yong

AU - Gordon, Michael

AU - Viganola, Domenico

AU - Chen, Z

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Uhlmann, Eric Luis

AU - Abd, Al-Aziz AM

AU - Abraham, Ajay T.

AU - Trojan, Jais

AU - Adamkovic, Matus

AU - Agadullina, Elena

AU - Ahn, Jungsoo

AU - Akinci, Cinla

AU - Akkas, Handan

AU - Albrecht, David

AU - Alzahawi, Shilaan

AU - Amaral-Baptista, Marcio

AU - Anand, Rahul

AU - Ang, Kevin Francis U.

AU - Anseel, Frederik

AU - Aruta, John Jamir Benzon R.

AU - Ashraf, Mujeeba

AU - Baker, Bradley J.

AU - Bao, Xueqi

AU - Baskin, Ernest

AU - Bathula, Hanoku

AU - Bauman, Christopher W.

AU - Bavolar, Jozef

AU - Bayraktar, Secil

AU - Beckman, Stephanie E.

AU - Benjamin, Aaron S.

AU - Brown, Stephanie E.V.

AU - Buckley, Jeffrey

AU - Buitrago, Ricardo E.

AU - Bution, Jefferson L.

AU - Byrd, Nick

AU - Carrera, Clara

AU - Caruso, Eugene M.

AU - Chen, M

AU - Chen, L

AU - Cicerali, Eyyub Ensari

AU - Cohen, Eric D.

AU - Crede, Marcus

AU - Cummins, J

AU - Dahlander, L

AU - Daniels, DP

AU - Daskalo, LL

AU - Dawson, IGJ

AU - Day, MV

AU - Dietl, E

AU - Domurat, A

AU - Dsilva, J

AU - Plessis, CD

AU - Dubrov, DI

AU - Edris, S

AU - Elbaek, CT

AU - Elsherif, MM

AU - Evans, TR

AU - Fellenz, MR

AU - Fiedler, S

AU - Firat, M

AU - Freitag, R

AU - Furrer, RA

AU - Gautam, R

AU - Gautam, DK

AU - Gearin, B

AU - Gerschewski, S

AU - Ghasemi, O

AU - Ghasemi, Z

AU - Ghosh, A

AU - Giani, C

AU - Goldberg, MH

AU - Goswami, M

AU - Graf-Vlachy, L

AU - Griffith, JA

AU - Grigoryev, D

AU - Gu, J

AU - H, R

AU - Hadida, AL

AU - Hafenbrack, AC

AU - Hafenbrädl, S

AU - Hammersley, JJ

AU - Han, H

AU - Harman, JL

AU - Hartanto, A

AU - Woike, Jan K.

PY - 2022/7/19

Y1 - 2022/7/19

N2 - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

AB - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

UR - https://pearl.plymouth.ac.uk/psy-research/1007/

U2 - 10.1073/pnas.2120377119

DO - 10.1073/pnas.2120377119

M3 - Article

SN - 0027-8424

VL - 119

JO - Proceedings of the National Academy of Sciences

JF - Proceedings of the National Academy of Sciences

IS - 30

ER -

Examining the generalizability of research findings from archival data

Abstract

Access to Document

Other files and links

Fingerprint

Cite this