A Survey on the Attitudes Towards and Perception of Reproducibility and Replicability in Sports and Exercise Science
Main Article Content
Abstract
There are formal calls for increased reproducibility and replicability in sports and exercise science, yet there is minimal information on the overall knowledge of these concepts at a field-wide level. Therefore, we conducted a survey on the attitudes and perceptions of sports and exercise science researchers towards reproducibility and replicability. Descriptive statistics (e.g., proportion of responses), and thematic analysis, were utilized to characterize the responses. Of the 511 respondents, 42% (n = 217) believe there is a significant crisis of reproducibility or replicability in sports and exercise science while 36% (n = 182) believe there is a slight crisis. 3% (n = 15) of respondents believe there is no crisis while 19% (n = 95) did not know. Four themes were generated in the thematic analysis: the research and publishing culture, educational barriers to research integrity, research responsibility to ensure reproducibility and replicability, and current practices facilitating reproducibility and replicability. Researchers believe that engaging in open science can be detrimental to career opportunities due to lack of incentives. They also feel journals are a barrier to reproducible and replicable research due to high publication charges and a focus on novelty. Statistical expertise was identified as a key factor for improving reproducibility and replicability in the future, particularly, a better understanding of study design and different statistical techniques. Statistical education should be prioritised for early career researchers which could positively affect publication and peer review. Researchers must accept responsibility for reproducibility and replicability with thorough project design, appropriate planning of analyses, and transparent reporting practices.
Metrics
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain the copyright of their manuscripts, and all Open Access articles are distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided that the original work is properly cited.
References
Abt, G. et al. (2021) ‘Registered Reports in the Journal of Sports Sciences’, Journal of Sports Sciences, pp. 1789–1790. Available at: https://doi.org/10.1080/02640414.2021.1950974.
Allen, C. and Mehler, D.M.A. (2019) ‘Open science challenges, benefits and tips in early career and beyond’, PLoS Biology, 17(5), p. e3000246. Available at: https://doi.org/10.1371/journal.pbio.3000246.
Aschwanden, C. and Nguyen, M. (2018) How Shoddy Statistics Found A Home In Sports Research | FiveThirtyEight, FiveThirtyEight. Available at: https://fivethirtyeight.com/features/how-shoddy-statistics-found-a-home-in-sports-research/ (Accessed: 24 May 2022).
Atkinson, G., Batterham, A. and Drust, B. (2008) ‘Is it time for sports performance researchers to adopt a clinical-type research framework?’, International Journal of Sports Medicine, 29(9), pp. 703–705. Available at: https://doi.org/10.1055/S-2008-1038545/ID/15.
Baker, M. and Penny, D. (2016) ‘Is there a reproducibility crisis?’, Nature, 533(7604), pp. 452–454. Available at: https://doi.org/10.1038/533452A.
Bernards, J. et al. (2017) ‘Current research and statistical practices in sport science and a need for change’, Sports, 5(87). Available at: https://doi.org/10.3390/sports5040087.
Blincoe, S. and Buchert, S. (2020) ‘Research preregistration as a teaching and learning tool in undergraduate psychology courses’, Psychology Learning and Teaching, 19(1), pp. 107–115. Available at: https://doi.org/10.1177/1475725719875844.
Borg, D.N. et al. (2020) ‘Comment on: “Moving sport and exercise science forward: a call for the adoption of more transparent research practices”’, Sports Medicine, 50(8), pp. 1551–1553. Available at: https://doi.org/10.1007/s40279-020-01298-5.
Borg, D.N., Lohse, K.R. and Sainani, K.L. (2020) ‘Ten common statistical errors from all phases of research, and their fixes’, PM&R, 12(6), pp. 610–614. Available at: https://doi.org/10.1002/pmrj.12395.
Boulbes, D.R. et al. (2018) ‘A survey on data reproducibility and the effect of publication process on the ethical reporting of laboratory research’, Clinical Cancer Research, 24(14). Available at: https://doi.org/10.1158/1078-0432.CCR-18-0227.
Braun, V. and Clarke, V. (2006) ‘Using thematic analysis in psychology’, Qualitative Research in Psychology, 3(2), pp. 77–101. Available at: https://doi.org/10.1057/978-1-137-35913-1.
Braun, V. and Clarke, V. (2019) ‘Reflecting on reflexive thematic analysis’, Qualitative Research in Sport, Exercise and Health, 8 August, pp. 589–597. Available at: https://doi.org/10.1080/2159676X.2019.1628806.
Buttliere, B. and Wicherts, J. (2018) ‘Opinions on the value of direct replication: A survey of 2,000 psychologists’. Available at: https://doi.org/10.31234/osf.io/z9kx6.
Büttner, F. et al. (2020) ‘Are questionable research practices facilitating new discoveries in sport and exercise medicine? The proportion of supported hypotheses is implausibly high’, British Journal of Sports Medicine, pp. 1–7. Available at: https://doi.org/10.1136/bjsports-2019-101863.
Caldwell, A.R. et al. (2020) ‘Moving sport and exercise science forward: A call for the adoption of more transparent research practices’, Sports Medicine, 50(3), pp. 449–459. Available at: https://doi.org/10.1007/s40279-019-01227-1.
Camerer, C.F. et al. (2016) ‘Evaluating replicability of laboratory experiments in economics’, Science, 351(6280), pp. 1433–1436. Available at: https://doi.org/10.1126/science.aaf0918.
Camerer, C.F. et al. (2018) ‘Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015’, Nature Human Behaviour, 2(9), pp. 637–644. Available at: https://doi.org/10.1038/s41562-018-0399-z.
Chalmers, S. et al. (2018) ‘Asymmetry during Functional Movement Screening and injury risk in junior football players: A replication study’, Scandinavian Journal of Medicine and Science in Sports, 28(3), pp. 1281–1287. Available at: https://doi.org/10.1111/sms.13021.
Chambers, C.D. et al. (2014) ‘Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond’, AIMS Neuroscience, 1(1), pp. 4–17. Available at: https://doi.org/10.3934/Neuroscience.2014.1.4.
Derksen, M. and Field, S. (2022) ‘The tone debate: Knowledge, self, and social order’, Review of General Psychology, 26(2), pp. 172–183. Available at: https://doi.org/10.1177/10892680211015636.
Errington, T.M. et al. (2021) ‘Investigating the replicability of preclinical cancer biology.’, eLife, 10, pp. 1–30. Available at: https://doi.org/10.7554/eLife.71601.
Fanelli, D., Costas, R. and Ioannidis, J.P.A. (2017) ‘Meta-assessment of bias in science’, Proceedings of the National Academy of Sciences of the United States of America, 114(14), pp. 3714–3719. Available at: https://doi.org/10.1073/pnas.1618569114.
Gabelica, M., Bojčić, R. and Puljak, L. (2022) ‘Many researchers were not compliant with their published data sharing statement: a mixed-methods study’, Journal of Clinical Epidemiology, 150, pp. 33–41. Available at: https://doi.org/10.1016/j.jclinepi.2022.05.019.
Guttinger, S. (2020) ‘The limits of replicability’, European Journal for Philosophy of Science, 10(2), pp. 1–17. Available at: https://doi.org/10.1007/s13194-019-0269-1.
Halperin, I., Pyne, D.B. and Martin, D.T. (2015) ‘Threats to internal validity in exercise science: A review of overlooked confounding variables’, International Journal of Sports Physiology and Performance, 10(7), pp. 823–829. Available at: https://doi.org/10.1123/IJSPP.2014-0566.
Hansford, H.J. et al. (2022) ‘Open and transparent sports science research: the role of journals to move the field forward’, Knee Surgery, Sports Traumatology, Arthroscopy, pp. 1–3. Available at: https://doi.org/10.1007/s00167-022-06893-9.
Heathers, J.A.J. and Grimes, D.R. (2022) ‘The Mechanics Behind A Precipitious Rise in Impact Factor: A Case Study From the British Journal of Sports Medicine’. Available at: https://doi.org/10.17605/OSF.IO/4C6XA.
Heneghan, C. et al. (2012) ‘Forty years of sports performance research and little insight gained.’, BMJ, 345. Available at: https://doi.org/10.1136/bmj.e4797.
Impellizzeri, F.M., McCall, A. and Meyer, T. (2019) ‘Registered reports coming soon: our contribution to better science in football research’, Science and Medicine in Football, 3(2), pp. 87–88. Available at: https://doi.org/10.1080/24733938.2019.1603659.
Janz, N. and Freese, J. (2020) ‘Replicate others as you would like to be replicated yourself’, PS: Political Science and Politics, 54(2), pp. 1–4. Available at: https://doi.org/10.1017/S1049096520000943.
Jekel, M. et al. (2020) ‘How to teach open science principles in the undergraduate curriculum—The Hagen cumulative science project’, Psychology Learning and Teaching, 19(1), pp. 91–106. Available at: https://doi.org/10.1177/1475725719868149.
John, L.K., Loewenstein, G. and Prelec, D. (2012) ‘Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling’, Psychological Science, 23(5), pp. 524–532. Available at: https://doi.org/10.1177/0956797611430953.
Kathawalla, U.-K., Silverstein, P. and Syed, M. (2020) ‘Easing into open science: A guide for graduate students and their advisors’, Collabra: Psychology, 7(1), pp. 1–38. Available at: https://doi.org/10.1525/collabra.18684.
Kent, B.A. et al. (2022) ‘Recommendations for empowering early career researchers to improve research culture and practice’, PLOS Biology, 20(7), p. e3001680. Available at: https://doi.org/10.1371/journal.pbio.3001680.
Klein, R.A. et al. (2014) ‘Investigating variation in replicability: A “many labs” replication project’, Social Psychology, 45(3), pp. 142–152. Available at: https://doi.org/10.1027/1864-9335/a000178.
Klein, R.A. et al. (2018) ‘Many Labs 2: Investigating Variation in Replicability Across Samples and Settings’, Advances in Methods and Practices in Psychological Science, 1(4), pp. 443–490. Available at: https://doi.org/10.1177/2515245918810225.
Knudson, D. (2005) ‘Statistical and reporting errors in applied biomechanics research’, in International Society of Biomechanics in Sport Conference. Beijing, China, pp. 811–814.
Knudson, D. (2017) ‘Confidence crisis of results in biomechanics research’, Sports Biomechanics, 16(4), pp. 425–433. Available at: https://doi.org/10.1080/14763141.2016.1246603.
Makel, M.C. and Plucker, J.A. (2014) ‘Facts are more important than Nnvelty: Replication in the education sciences’, Educational Researcher, 43(6), pp. 304–316. Available at: https://doi.org/10.3102/0013189X14545513.
Maxwell, S.E., Lau, M.Y. and Howard, G.S. (2015) ‘Is psychology suffering from a replication crisis?: What does “failure to replicate” really mean?’, American Psychologist, 70(6), pp. 487–498. Available at: https://doi.org/10.1037/a0039400.
Mellor, D.T. (2021) ‘Improving norms in research culture to incentivize transparency and rigor’, Educational Psychologist, 56(2), pp. 122–131. Available at: https://doi.org/10.1080/00461520.2021.1902329.
Mesquida, C. et al. (2022) ‘Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field’, Royal Society Open Science, 9(220946). Available at: https://doi.org/https://doi.org/10.1098/rsos.220946.
Morey, R.D. et al. (2015) ‘The Peer Reviewers’ Openness Initiative: incentivizing open research practices through peer review’, Royal Society Open Science, 3(1). Available at: https://doi.org/10.1098/RSOS.150547.
Morin, J.B. et al. (2019) ‘A simple method for computing sprint acceleration kinetics from running velocity data: Replication study with improved design’, Journal of Biomechanics, 94, pp. 82–87. Available at: https://doi.org/10.1016/j.jbiomech.2019.07.020.
Munafò, M.R. et al. (2017) ‘A manifesto for reproducible science’, Nature Human Behaviour. Nature Publishing Group. Available at: https://doi.org/10.1038/s41562-016-0021.
Murphy, J. et al. (2022) Selection Protocol for Replication in Sports and Exercise Science.
Nielsen, R.O. et al. (2018) ‘Seven sins when interpreting statistics in sports injury science’, British Journal of Sports Medicine, 52(22), pp. 1410–1412. Available at: https://doi.org/10.1136/bjsports-2017-098524.
Nosek, B.A. et al. (2015) ‘Promoting an open research culture’, Science, 348(6242), pp. 1422–1425. Available at: https://doi.org/10.1126/science.aab2374.
Nosek, B.A. et al. (2022) ‘Replicability, robustness, and reproducibility in psychological science’, Annual Review of Psychology, pp. 719–748. Available at: https://doi.org/10.1146/annurev-psych-020821-114157.
Nosek, B.A. and Errington, T.M. (2020) ‘What is replication?’, PLoS Biology, 18(3), pp. 1–8. Available at: https://doi.org/10.1371/journal.pbio.3000691.
Nosek, B.A., Spies, J.R. and Motyl, M. (2012) ‘Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability’, Perspectives on Psychological Science, 7(6), pp. 615–631. Available at: https://doi.org/10.1177/1745691612459058.
Nuijten, M.B. et al. (2016) ‘The prevalence of statistical reporting errors in psychology (1985–2013)’, Behavior Research Methods, 48(4), pp. 1205–1226. Available at: https://doi.org/10.3758/s13428-015-0664-2.
Open Science Collaboration (2015) ‘Estimating the reproducibility of psychological science’, Science, 349(6251). Available at: https://doi.org/10.1126/science.aac4716.
Ostaszewski, M. (2014) ‘Analysis of the attitude within academic and research communities towards open science – a quantitative survey’, pp. 1–16. Available at: https://doi.org/depot.ceon.pl/handle/123456789/3719.
Pashler, H. and Wagenmakers, E.J. (2012) ‘Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?’, Perspectives on Psychological Science, 7(6), pp. 528–530. Available at: https://doi.org/10.1177/1745691612465253.
Pitsch, W. and Emrich, E. (2012) ‘The frequency of doping in elite sport: Results of a replication study’, International Review for the Sociology of Sport, 47(5), pp. 559–580. Available at: https://doi.org/10.1177/1012690211413969.
Piwowar, H.A. and Vision, T.J. (2013) ‘Data reuse and the open data citation advantage’, PeerJ, 2013(1), pp. 1–25. Available at: https://doi.org/10.7717/peerj.175.
Powell, K. (2015) ‘The future of the postdoc’, Nature, pp. 144–147. Available at: https://doi.org/10.4324/9781315674827-14.
Pownall, M. et al. (2022) ‘The impact of open and reproducible scholarship on students’ scientific literacy, engagement, and attitudes towards science: A review and synthesis of the evidence’, MetaArXiv Preprints, pp. 1–56. Available at: https://doi.org/10.31222/OSF.IO/9E526.
Prieto, D. (2017) ‘Make databases language-proof’, Nature, 550(332), p. 29. Available at: https://doi.org/10.1038/d41586-018-05847-x.
R Core Team (2022) ‘A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/’. Vienna, Austria: R Foundation for Statistical Computing.
Rosenthal, R. (1979) ‘The “file drawer problem” and tolerance for null results’, Psychological Bulletin, 86(3), pp. 638–641.
Ross-Hellauer, T., Deppe, A. and Schmidt, B. (2017) ‘Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers’, PLoS ONE, 12(12), pp. 1–28. Available at: https://doi.org/10.1371/journal.pone.0189311.
Sainani, K. and Chamari, K. (2022) ‘Wish list for improving the quality of statistics in sport science’, International Journal of Sports Physiology and Performance, 17(5), pp. 673–674. Available at: https://doi.org/10.1123/ijspp.2020-0934.
Sainani, K.L. et al. (2019) ‘Magnitude-based Inference is not Bayesian and is not a valid method of inference’, Scandinavian Journal of Medicine and Science in Sports, 29(9), pp. 1428–1436. Available at: https://doi.org/10.1111/sms.13491.
Sainani, K.L. et al. (2021) ‘Call to increase statistical collaboration in sports science, sport and exercise medicine and sports physiotherapy’, British Journal of Sports Medicine, 55(2), pp. 118–122. Available at: https://doi.org/10.1136/bjsports-2020-102607.
Scheel, A.M. et al. (2020) ‘Why hypothesis testers should spend less time testing hypotheses’, Perspectives on Psychological Science, 16(4), pp. 744–755. Available at: https://doi.org/10.1177/1745691620966795.
Schmidt, S. (2009) ‘Shall we really do it again? The powerful concept of replication is neglected in the social sciences’, Review of General Psychology, 13(2), pp. 90–100. Available at: https://doi.org/10.1037/a0015108.
Smaldino, P.E. and McElreath, R. (2016) ‘The natural selection of bad science’, Royal Society Open Science, 3(9). Available at: https://doi.org/10.1098/rsos.160384.
Smith, L.M., Yu, F. and Schmid, K.K. (2021) ‘Role of replication research in biostatistics graduate education’, 29(1), pp. 95–104. Available at: https://doi.org/10.1080/10691898.2020.1844105.
Stojmenovska, D., Bol, T. and Leopold, T. (2019) ‘Teaching Replication to Graduate Students’, Teaching Sociology, 47(4), pp. 303–313. Available at: https://doi.org/10.1177/0092055X19867996/ASSET/IMAGES/LARGE/10.1177_0092055X19867996-FIG1.JPEG.
Stroebe, W. and Strack, F. (2014) ‘The Alleged Crisis and the Illusion of Exact Replication’, Perspectives on Psychological Science, 9(1), pp. 59–71. Available at: https://doi.org/10.1177/1745691613514450.
Turner, L. et al. (2012) ‘Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane reviewa’, Systematic Reviews, 1(1), pp. 1–7. Available at: https://doi.org/10.1186/2046-4053-1-60.
Twomey, R. et al. (2021) ‘The nature of our literature: A registered report on the positive result rate and reporting practices in kinesiology’, Communications in Kinesiology, 1(3), pp. 1–14. Available at: https://doi.org/10.51224/cik.v1i3.43.
Vines, T.H. et al. (2014) ‘The availability of research data declines rapidly with article age’, Current Biology, 24(1), pp. 94–97. Available at: https://doi.org/10.1016/j.cub.2013.11.014.
Vinkers, C.H., Tijdink, J.K. and Otte, W.M. (2015) ‘Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis’, BMJ, 351. Available at: https://doi.org/10.1136/BMJ.H6467.
de Vrieze, J. (2021) ‘Large survey finds questionable research practices are common Dutch study finds 8% of scientists have committed fraud’, Science, 373(6552), p. 265. Available at: https://doi.org/10.1126/science.373.6552.265.
Wagge, J.R. et al. (2019) ‘Publishing research with undergraduate students via replication work: The collaborative replications and education project’, Frontiers in Psychology, 10(FEB), pp. 1–4. Available at: https://doi.org/10.3389/fpsyg.2019.00247.