| Question code | Quotes |
|---|---|
| Sub-Theme: Incentives for undertaking replication research | |
| Factors that could improve reproducibility and replicability | “If the culture changes that you should every now and then replicate a study just like you should review papers, then you might get a more actively reproducing/replicating community. For now, there simply is no individual benefit and in fact, you'll probably 'get behind' in your own publications so it may even be detrimental for your career.” |
| Failure to replicate or reproduce findings is a major problem | “The problem is that it is almost impossible to publish replication studies in high quality (e.g., Q1) journals - if we can't publish replication studies, there is limited incentive to conduct them as researchers” |
| Existing journal efforts and why they help or not | “Most academics are incentivised by what will get them promoted. We need to include Open Science practices in promotion criteria. For example, has the candidate submitted a Registered Report in the last x years? how many of their studies have been pre-registered? how many of their studies have shared code/data? etc. Only then will many academics take Open Science/replication seriously. It's sad that academics have to be externally motivated like this, but unfortunately that is what it will take.” |
| Factors that could improve reproducibility and replicability | “Pressure to publish/get funding which then means replicability studies are not as valued by employer” |
| Barriers to implementing changes | “The cultural inertia of previous practices has been somewhat of a barrier. It's hard and uncomfortable for people to acknowledge that the work they may have done in the past is not of the best quality and changing practices is an explicit acknowledgement of that.” |
| Barriers to implementing changes | “We'd need to see structural changes within universities where studies with larger sample sizes, requiring longer data collections, and therefore fewer publications was rewarded (e.g., considered for tenure track, promotion, hiring, ranking, for funding etc). But currently, academics are rewarded for being prolific with less emphasis on quality. I think journals requiring/rewarding replication and/or reproduction would also go a long way.” |
| Factors that could improve reproducibility and replicability | “Convincing journals of the need to change is the most difficult, because there's very little incentive for the editors and/or the publisher to change.” |
| Existing journal efforts and why they help or not | “I don't think journals do enough to help provide a platform for better replication but like with peer-review and the fact that they drive a model of work that is largely underpinned by volunteers providing content and volunteers reviewing content, there is nothing to force them to change.” |
| Failure to replicate or reproduce findings is a major problem | “Scientific replication isn't 'sexy,' or well-funded (as far as I know) so researchers don't have much incentive to replicate studies. Funding is given for new research.” |
| Factors that could improve reproducibility and replicability | “I'd like to emphasize that lack of incentive (funding/time, but also the added benefit for your career) is an important reason for the low effort put in reproducing or replicating the work.” |
| Barriers to implementing changes | “Almost all of the strategies (for improving the replication crisis) listed above come at an increased labour/logistics cost. This increased science labour/logistics must come with a commensurate increase in resources.” |
| Factors that could improve reproducibility and replicability | “It comes down to university-based metrics. In sports science, reproducibility-based studies do not attract funding or citations. We have to go out of our way to do this research. While important, unless it is recognized and rewarded by the university, it is very difficult to do.” |
| Barriers to implementing changes | “Time. It takes longer to do things 'properly'” |
| Barriers to implementing changes | “Already science occurs on a tight budget. Scientist's altruism is already exploited (in terms of salary for young scientists). You want to end the replication crisis: then establish the protocols and allocate resources commensurate to the increased labour/logistics.” |
| Factors that could improve reproducibility and replicability | “Build the issue into funding, publications and importantly university appraisal/targets etc. if I have to double my time in an experiment because I always need to do a specific replicability study my university needs to realise, I may produce less volume overall” |
| Factors that could improve reproducibility and replicability | More robust design is somewhat linked to professional incentives in the sense that robust research designs are invariably more expensive to implement, and thereby require funding bodies to recognize that one study with 100 subjects may well be worth more than 3 studies with 30. |
| Sub-Theme: Priority of novel research | |
| Failure to replicate or reproduce findings is a major problem | “We are told from early on in our careers that your research must be 'novel' so I don't know of anyone reproducing or replicating studies - I am not sure they would be published. I think then that may lead to results from single studies being taken as 'true' and you also end up with lots of review articles/meta-analyses trying to make sense out of a lot of studies that are all different.” |
| Level of replication in my field compared to other fields | “There is such a high focus on publishing "new" results that we do not sufficiently consider the accuracy and generalizability of prior results. Even with good intentions, so much existing work is very software intensive --- so, mistakes happen. And many mistakes are just not found” |
| Failure to replicate or reproduce findings is a major problem | “As before, research rewards accrue to those doing novel studies.” |
| Barriers to implementing changes | “As noted previously, most journals only want to publish "new" methods. I'm not aware of ANY journals in my field that would welcome a reproducibility or replication study. It would be rejected outright as "not novel."” |
| Existing journal efforts and why they help or not | “I have had papers rejected on the basis that 'the results weren't 'positive' or 'significant'. We all have. Journals perpetuate the problem by prioritizing novel findings.” |
| Failure to replicate or reproduce findings is a major problem | “Replication studies are not favoured in science currently. It’s all about the next new and best thing.” |
| Sub-Theme: The business model of publishing | |
| Existing journal efforts and why they help or not | “As with the funding, we know that reviewers are seeking novelty in the work, and I would expect to be criticised if I submitted a replication study.” |
| Existing journal efforts and why they help or not | “As I have previously stated, replication studies are discouraged by journal editors and frequently rejected without being reviewed. This fact leads funding entities and labs to avoid the reproducibility of existing research, mainly because they do not consider it innovative and susceptible of scientific breakthroughs. Sadly, the vicious circle in Sports Sciences is not favourable for reproducibility.” |
| Existing journal efforts and why they help or not | “Having experienced several rejections of studies that were similar to previous works, it seems that "impact factor" is driving most journals. In addition, the increase in the number of journals with publication charges is turning the scientific world into the business world. Some of these page changes are astronomical and well beyond the means of typical researchers in the field of exercise science.” |
| Existing journal efforts and why they help or not | “Publishers are leeches, who care nothing more than making a profit. Token gestures of encouraging open access and data deposition are hollow at best. They do not help” |
| Existing journal efforts and why they help or not | “Generally, rigorous peer review and editorial handling goes a long way. However scientific publishing has become a billion-dollar business with way, way too much financial dependence and consequently a flood of low-quality and predatory journals publishing poor science.” |
| Factors that could improve reproducibility and replicability | “Take the politics out of science.” |
| Existing journal efforts and why they help or not | “All journals want to do is increase impact ratings” |
| Barriers to implementing changes | “As mentioned, before I believe that academia pushes for greater scientific output at the cost of its quality” |
| Existing journal efforts and why they help or not | “Way too much nepotism in review process. Poorly designed/described studies are often published purely because of a well-known co-author (who likely had very little to do with the study.” |
Supplementary Tables
A Survey on the Attitudes Towards and Perception of Reproducibility and Replicability in Sports and Exercise Science
DOI: https://doi.org/10.51224/cik.2023.53
Table 1
Table 2
| Question code | Quotes |
|---|---|
| Sub-Theme: Quality of peer review | |
| Existing journal efforts and why they help or not | "The peer review process is only as good as the peer reviewers. I've read many studies with missing details." |
| Existing journal efforts and why they help or not | "Methods are reviewed at a level that is deemed "peer review". However, given my personal experience of peer review and papers that have been sent to me by journals, many papers sent by so called "top journals" fall outside specialist areas and deemed "expertise". This is before we consider the lack of general understanding for statistics within the field of exercise sciences. Which open up levels of bias, poor analysis, lack of controls...... The list is endless here as to why replication or repeating findings would be an issue." |
| Factors contributing towards a failure to reproduce or replicate | "Sometimes the research is so badly written that it is hard to understand important parts of the research/test/experiment. This could go under insufficient peer review, but often conference papers (which are still indexed) are lazily peer-reviewed." |
| Existing journal efforts and why they help or not | "I believe it comes down to the reviewer. Many reviewers miss issues within methodologies and therefore this issue continues." |
| Factors contributing towards a failure to reproduce or replicate | "Authors and reviewers pretending they know the technical procedures. They make wrong interpretations of the phenom and bring low contribution to science" |
| Existing journal efforts and why they help or not | "I also encounter editors and reviewers insisting that hypotheses are added after submission if not present. Reviewers also influence authors to adopt their (reviewers') conventions, style, rules, etc. which leads to a slow evolution of arbitrary practices." |
| Existing journal efforts and why they help or not | "It’s all well and good having checklists but editors need to listen to reviewers (like me) who flag up dodgy studies rather than ignore and publish them just because they are sexy" |
| Sub-Theme: Statistical expertise and knowledge of researchers | |
| Factors contributing towards a failure to reproduce or replicate | "I think many times researchers believe that they know more about research than they do, making serious errors in methodology, using the wrong statistical tests, or not having clear objectives that they know how to accomplish." |
| Factors contributing towards a failure to reproduce or replicate | "I guess many researchers simply underestimate (just like I did for a long time) the role of chance for obtaining seemingly significant results, particularly when you combine low power and researcher degrees of freedom. Stuff becomes significant by chance, and then of course you cannot replicate it" |
| Factors that could improve reproducibility and replicability | "I believe that improving research education is the key, improving statistics education is vital, and above all improving research ethics, since there are researchers who think that the data should say what they want and that is why they review and modify them until they get what they want. In those cases, replicability is impossible." |
| Factors contributing towards a failure to reproduce or replicate | "Lack of understanding of statistical methods to analyse data." |
| Factors contributing towards a failure to reproduce or replicate | "I think that most students, and therefore advisors, rarely explore their data adequately before thesis and publishing due to pressure to publish and complete. I think many blunders would be avoided, especially failures to detect differences, and insights into the nature of the data would better inform the approaches for analysis. Perhaps a data scrubbing to data exploration module could be produced. Also, I have witnessed many cases of research assistants not using the actual protocol in clinical RCT sport science studies resulting in lots of variance in the data." |
| Factors contributing towards a failure to replicate | "Investigator/researcher laziness or sloppiness/short cuts" |
| Factors contributing towards a failure to replicate | "There are many but ability to recruit larger numbers of participants who fit study criteria, human biases in a number of aspects of the research, poorly reported methodology in the literature which we cannot replicate, poorly performed or incorrectly reported statistical analysis that we cannot replicate etc" |
| Factors that could improve reproducibility and replicability | "Errors in data management (cleaning, accounting for missingness, coding variation between statistical software) and important differences in the data are both potential issues. Actual variation in reality is always a contributor. Measurement and misclassification." |
| Factors contributing towards a failure to reproduce or replicate | "There is a need to educate existing researchers - perhaps by holding workshops at conferences on methodology, rather than just on results, and also, encouraging journals to publish papers or perspectives on clinical trial methods. for example, encouraging journals to consider really well-designed pilot studies as "real" research. Having time or money to replicate findings or mentor students won’t work if you are not using the right methods in the first place." |
| Level of replication in my field compared to other fields | "Typically, we publish small sample size research, but most researchers are too statistically innumerate to analyse and interpret data accordingly." |
Table 3
| Question code | Quotes |
|---|---|
| Sub-Theme: Journal responsibility | |
| Existing journal efforts and why they help or not | "Reporting requirements do not seem to be consistent across journals." |
| Existing journal efforts and why they help or not | "I think that by the time a journal imposes guidelines related to reproducibility/replicability, it is too late, because the project has already been done and the manuscript written. These directions should come from funding agencies and research institutions." |
| Existing journal efforts and why they help or not | "I think the journal publishers are using standards/checklists developed by the scientific community, because the community is demanding more transparent reporting. I don’t think it is the journals responsibility. I think the researchers should own and drive it." |
| Existing journal efforts and why they help or not | "The journals wield a double-edged sword when it comes to replication and reproducibility. On the one hand, reporting guidelines in most journals I have experience with seem to overall have a positive effect on reproducibility and replicability. However, journals seem to reject papers that disseminate the replication of a study, thus preventing an objective test of the replicability of any study." |
| Existing journal efforts and why they help or not | "As per previous answer, journals are gatekeepers to much of what we publish as scientists. We are bound by their rules (preprints being the exception). I believe journals should make much more effort to improve transparency and openness of research published in their journals." |
| Existing journal efforts | "I'm not sure this is something a journal publisher should be responsible for. I think this should be core to the scientific community." |
| Existing journal efforts and why they help or not | "Reporting requirements do not seem to be consistent across journals." |
| Existing journal efforts and why they help or not | "The bigger issue is article length. So much effort goes into writing 'objective' papers with brief method sections that the nuance about what, when, and why certain decisions made can't fit into the paper, which fuels the crisis." |
| Level of replication in my field | "My concern is more related to the level of detail provided in the methods section. Exercise can be highly variable, and authors (and reviewers) aren't doing a great job of ensuring that enough methodological detail is provided so that studies can be replicated. You can't replicate a study if you aren't positive what is being done. This is also similar for reporting of participant characteristics or handling of blood/tissue samples." |
| Existing journal efforts and why they help or not | "While it is important to enforce reproducibility and replicability, I struggle to see how journals can enforce this." |
| Existing journal efforts and why they help or not | "Journals are a barrier to reproducible research, actively promoting file drawer problems and having statistically naive editors." |
| Existing journal efforts and why they help or not | "Journal can certainly facilitate good open science practices among academics via their author guidelines, expectations, types of publications, etc. I think the sport and exercise science journals are still playing catch-up to journals in other fields though (e.g., psychology)." |
| Existing journal efforts and why they help or not | "Academic journals have an opportunity to be a leader in the space of reproducibility and replicability by implementing policies for authors to abide by in submitting their work." |
| Existing journal efforts | "As with funding agencies, I think this is looking in the wrong place for a solution." |
| Sub-Theme: Senior researcher responsibility | |
| Barriers to implementing changes | "Some older "traditionalist" colleagues prefer not to change the ways of assessment, conducting and writing studies." |
| Barriers to implementing changes | "Other faculty members and students are not always responsive" |
| Barriers to implementing changes | "The PI [principal investigator]. In North America, all "trainees" working under a PI can only do as much as the PI supports. There is a huge power imbalance that can be very difficult to navigate for more junior colleagues if the PI is not interested, dismissive, and in some cases hostile to such practices. This has been my experience (but I also know of several PIs who are supportive). " |
| Factors that could improve reproducibility and replicability | "The PI is often the source of the problem. Who mentors them?" |
| Barriers to implementing changes | "A big one is collaborating with colleagues who don't have the same values. We either have to not collaborate with certain people, try to convince them of the benefits of publishing fewer studies per year, or agree then we go outside the lab group people have different research norms." |
| Barriers to implementing changes | "Poor acceptance from laboratory heads on the importance of such work." |
| Factors that could improve reproducibility and replicability | "*Explicitly* encouraging reproducibility and replicability would make a difference, rather than teaching students that all their work must be new and novel" |
| Factors that could improve reproducibility and replicability | "Mentoring of students, graduate students, post-docs in all aspect of quality research." |
Table 4
| Question code | Quotes |
|---|---|
| Sub-Theme: Data sharing and availability | |
| Existing journal efforts and why they help or not | "While some journals explicitly advise authors to do x or y, often they do not enforce, which means that authors ignore the recommendations." |
| Existing journal efforts and why they help or not | "Making data sharing compulsory would be a major step forward. Many journals state this is a requirement but do not enforce this." |
| Existing journal efforts and why they help or not | "Depositing datasets in data repositories (I mostly use our own institutional repository) and making them accessible in publications has been helpful. However, the act of creating these datasets in ways suitable for sharing is very time consuming and challenging when time/funding are limited." |
| Existing journal efforts and why they help or not | "The only effort I have encountered is a requirement to provide data open access on acceptance, which I think might have some role in deterring people from actively making up data. I think we can’t ignore the potential career disadvantages in forcing all data and code to be shared: Particularly for smaller/less well-funded groups, having to ‘give away’ work that they would otherwise be able to leverage to get a head start on future publications to bigger (and hence faster-moving) groups is a real problem (which gets shouted down when we are banging the drum for ‘open science’)." |
| Existing journal efforts and why they help or not | "Collecting the data is hard work and expensive. Immediately giving away those data can deter a lab’s ability to be successful, if other labs end up publishing new analysis of those data before your own lab gets the chance." |
| Existing journal efforts and why they help or not | "The “open data” concept also claims to be for the public good. But beware: information curation platforms will capture these commons. Just look at Facebook/Google/Etc. On the scale of a civilization, an entity that controls access to information can manipulate the data without owning the data." |
| Existing journal efforts and why they help or not | "I have been pleased to discover opportunities to submit registered reports, receive pre-registration badges, and share data. I am unsure if these opportunities are having a positive effect, and I think journal publishers should do more to encourage reproducibility and replicability because I still read articles that seem to describe questionable research practices." |
| Sub-Theme: Checklists | |
| Existing journal efforts and why they help or not | "Journal checklists are overly generic which impedes their utility. Making them more extensive is not useful and would drive me crazy, especially for a desk-reject." |
| Existing journal efforts and why they help or not | "More work needs to be done in a fostering manner rather than a policing manner. Checklists are inadequate to deal with the issue." |
| Existing journal efforts and why they help or not | "Checklists are useless. So are requirements to use e.g., non-parametric statics or report an effect size, which I have seen. People just google a non-parametric test, run, and interpret it just as blindly as they did any other. Same for the effect size. Same goes for reviewers." |
| Existing journal efforts and why they help or not | "I think that the checklists are not enough and most times not mandatory. It would be better to be more rigorous in the methods section revision and ask the authors to share more detailed information on how the study was carried." |
| Existing journal efforts and why they help or not | "The implementation of and adherence to checklists and standards is very haphazard." |
| Existing journal efforts and why they help or not | "Methods checklists, sources for research materials, and the requirement to have all raw data in a public repository or as supplemental files are extremely useful. It does need to be enforced better, and standardization is currently lacking." |
| Existing journal efforts and why they help or not | "Some journals attempt to enforce standards around sample sizes, reporting, and analysis procedures which does help in terms of planning an appropriately sized/powered study, which in term will help with replicability. However, this does need to be more consistent across journals, and also needs to be accompanied by a change in culture (collaboration, time, less pressure to publish) in order to be successful." |
| Existing journal efforts and why they help or not | "Mandatory open data and open code, statements regarding researcher degrees of freedom, justification of sample sizes (and others) “force” authors to consider these things." |
Table 5
| Question | Quotes |
|---|---|
| What barriers would prevent you from volunteering in a large reproducibility or replication project? | "Some of these projects come off as “witch hunts” unless proper safeguards are in place. There are many biases in our field. One group could make an effort to single out another group. I would hope this wouldn’t happen, but that is why I would carefully evaluate the effort before agreeing to participate [in a replication project]." |
| Factors that could improve reproducibility and replicability | "Reduce the negative stigma of having a result that is not replicable, and emphasize the opportunity to sort out what is going on." |
| What barriers would prevent you from volunteering in a large reproducibility or replication project? | "Time and effort versus the benefit. Sports science isn’t cancer biology if the findings of a study are questionable, they can simply be ignored, they don’t have to be proved wrong – it’s not life or death!" |