The Registered Reports Process
The cornerstone of the Registered Reports format is that a significant part of the manuscript is assessed prior to data collection, with the highest quality submissions accepted in advance. Initial submissions include a description of the key research question and background literature, hypotheses, experimental procedures, analysis pipeline, a statistical power analysis (or Bayesian equivalent), and pilot data (where applicable).
Initial submissions will be triaged by an editorial team for suitability. Those that pass triage will then be sent for in-depth peer review (Stage 1). Following review, the article will be either rejected or accepted in principle for publication. Following in principle acceptance (IPA), the authors will proceed to conduct the study, adhering exactly to the peer-reviewed procedures. When the study is complete the authors will submit their finalised manuscript for re-review (Stage 2) and will upload their raw data, digital study materials, and laboratory log to a publicly accessible file-sharing service. Pending quality checks and a sensible interpretation of the findings, the manuscript will be published regardless of the results.
Registered Reports in Kinesiology publishes only registered reports. When authors submit a Registered Report, the introduction and methodology are reviewed before data are collected. This allows peer reviewers to address methodological concerns before time and resources are used to collect data. If the introduction and methodology are approved prior to data collection, and authors follow the approved methodology in good faith, then Registered Reports in Kinesiology guarantees publication of the results, regardless of the outcome.
This process is designed to reduce publication bias as well as questionable and unfortunate research practices, including p-hacking and HARKing. This process is also designed to reduce future additions to the File Drawer. Registered Reports in Kinesiology values all quality and replicable research, regardless of outcome. Finally, this process is designed to streamline the publication process and give researchers a feeling of peace and security, because they know that their work will be published, regardless of outcome. Please read this preprint for more information: https://osf.io/preprints/sportrxiv/fxe7a/
Initial manuscript submission and review Stage 1 submissions should include the manuscript (details below) and a brief cover letter. Please note that the editorial board will not agree to send manuscripts for in-depth review until all steps of a Stage 1 submission are completed. The Stage 1 cover letter should be addressed to the managing editors and include: A brief scientific case for consideration. Authors should explain how the submission fits within the Aims and Scope of Registered Reports in Kinesiology. Authors are encouraged to refer to the likely replication value of the research. Replication studies and novel studies are equally welcome. A statement indicating the preferred section for review:
-Coaching and Sports Pedagogy
-Exercise and Sport Psychology
-Physical Activity, Health, and Disease
-Physiology and Nutrition
-Sports Medicine and Rehabilitation
-Training and Performance Analysis
A statement confirming that all necessary support (e.g. funding, facilities) and approvals (e.g. ethics) are or will be in place for the proposed research. Note that manuscripts will be generally considered for studies that are able to commence immediately; however, authors with alternative plans are encouraged to contact the journal office for advice. Authors should note if they are seeking in-principle acceptance to strengthen a funding application. An anticipated timeline for completing the study if the initial submission is accepted. A statement of the preferred embargo period (if any) of the In-Principle Acceptance, Stage 1 manuscript. Please specify if you would like to keep your Stage 1 manuscript private, and if so, for how long. You may keep stage 1 manuscripts private for up to four years, or until the full Stage 2, the full manuscript is published, whichever is sooner. A statement confirming that the authors agree to share their raw, de-identified data, any digital study materials, and analysis code as appropriate. A statement confirming that, following Stage 1 in-principle acceptance, the authors agree to have their approved protocol registered on the Open Science Framework (https://osf.io/). Accepted protocols will be registered by Registered Reports in Kinesiology, and authors will be assigned a digital object identifier (DOI) to ensure that they receive proper credit for their work. A statement confirming that if the authors later withdraw their paper, they agree to Registered Reports in Kinesiology publishing a short summary of the pre-registered study under a section Withdrawn Registrations. A statement declaring any conflicts of interests or explicitly stating that there are no conflicts of interest. A statement confirming the names and affiliations of all authors involved with the project and their respective contributions.
Manuscript preparation guidelines – Stage 1
Initial Stage 1 submissions should include the following sections:
Introduction. A review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that following IPA, the Introduction section cannot be altered apart from the correction of factual errors, typographic errors and altering of tense from future to past (see below).
Methods. Authors are encouraged to submit a screenshot, syntax, and/or output of their sample size estimation and justification as part of the cover letter materials, to be shared with the reviewers. As the absence of significant is no evidence of the absence of effect, authors are encouraged to statistically evaluate null-results using equivalence tests, Bayesian estimation, or Bayes factors. Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g. outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced. A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information.
These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected. Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below). Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Since publication bias overinflates published estimates of effect size, power analysis must be based on the lowest available or meaningful estimate of the effect size. For frequentist analysis plans, the a priori power should be 0.8 or higher for all proposed hypothesis tests. Occasionally, the planned number of participants cannot be recruited for practical reasons. Deviations from this suggestion should be justified and explicitly mentioned. Explanations for deviations should be discussed to assist future researchers. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.
Methods involving Bayesian hypothesis testing are also encouraged. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified. For example, will authors use a uniform up to some specified maximum, or a normal/half-normal to represent a likely effect size or a JZS/Cauchy with a specified scaling constant? For inference by Bayes factors, authors must be able to guarantee data collection until the Bayes factor is at least 6 times in favor of the experimental hypothesis over the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however to be eligible for advance acceptance this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field. For further advice on Bayes factors or Bayesian sampling methods, prospective authors are encouraged to read this key article by Schönbrodt and Wagenmakers. Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.
Timeline for completion of the study and proposed resubmission date if Stage 1 review is successful. Extensions to this deadline can be negotiated with the section editor. Any description of prospective methods or analysis plans should be written in future tense. Pilot DataOptional. Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot experiments will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered experiment(s). Secondary RegistrationsRegistered Reports in Kinesiology welcomes submissions proposing secondary analyses of existing data sets, provided authors can supply sufficient evidence (e.g. self-certification; letter from independent gatekeeper) to confirm that they have had no prior access to the data in question. For advice on the eligibility of specific scenarios, authors are welcome to contact the editorial office at email@example.com.
Stage 1 submissions that are judged by the editorial board to be of sufficient quality and within journal scope will be sent for in-depth peer review. In considering papers at the registration stage, reviewers will be asked to assess:
-The importance of the research question(s).
-The logic, rationale, and plausibility of the proposed hypotheses.
-The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
-Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
-Whether the authors provide a sufficiently clear and detailed description of the methods to prevent undisclosed flexibility in the experimental procedures or analysis pipeline.
-Whether the authors have pre-specified sufficient analyses for ensuring that the results obtained can test the stated hypotheses.
Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or accepted. Proposals that exceed the highest standards of importance and scientific rigor will be issued an in-principle acceptance (IPA), indicating that the article will be published pending completion of the approved methods and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results. Stage 1 protocols are not published by the journal following IPA. Instead, they are registered by the journal and then integrated into a single completed article following approval of the final Stage 2 manuscript. Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the editorial board immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted per editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the experimental procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see Results & Discussion section below).
Once the study is complete, authors prepare and resubmit their manuscript for full review, with the following additions: Cover letter. The Stage 2 cover letter must confirm that: The manuscript includes a link to the public archive containing anonymized study data, digital materials/code, and the laboratory log. The cover letter should state the page number in the manuscript that lists the URL. The authors obtained ethical approval to conduct the described research if required. The manuscript contains a link to the approved Stage 1 protocol on the Open Science Framework or other recognized repository. The cover letter should state the page number in the manuscript that lists the URL. For primary Registered Reports, no data for any pre-registered study (other than pilot data included at Stage 1) was collected prior to the date of IPA. For secondary Registered Reports, authors should confirm that no data (other than pilot data included at Stage 1) was subjected to the pre-registered analyses prior to IPA. All data exclusions (if any), all manipulations, and all measures used in the study are reported. The authors, affiliations, and contributions are current or have been updated (e.g., additions, removals, changes in order).
Background, Rationale, and Methods
Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 submission, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g. correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion. Authors should also include a statement indicating that ethical approval was obtained, where appropriate.
Results & Discussion
The outcome of all registered analyses must be reported in the manuscript, except in rare instances, where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases, the analysis would still be mentioned in the Methods but omitted with justification from the Results. It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses. Authors reporting null hypothesis significance tests are required to report exact p values and effect sizes for all inferential analyses. The resubmission will most likely be considered by the same reviewers as in Stage 1 but could also be assessed by new reviewers. In considering papers at Stage 2, reviewers will be asked to decide:
-Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls).
-Whether the Introduction, rationale, and stated hypotheses are the same as the approved Stage 1 submission (required).
-Whether the authors adhered precisely to the registered experimental procedures
-Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative.
-Whether the authors’ conclusions are justified given the data. Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results.
Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.
Manuscript withdrawal and Withdrawn Registrations
Authors with IPA may wish to withdraw their manuscript following or during data collection. Possible reasons could include major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, manuscripts can, of course, be withdrawn at the authors’ discretion. However, the journal will publicly record each case in a section called Withdrawn Registrations. This section will include the authors, proposed title, the abstract from the approved Stage 1 submission, and brief reason(s) for the failure to complete the study. Partial withdrawals are not possible; i.e. authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such cases must lead to the withdrawal of the entire paper. Studies that are not completed by the agreed Stage 2 submission deadline (which can be extended in negotiation with the editorial office) will be considered withdrawn and will be subject to a Withdrawn Registration.
Authors may add experiments to approved submissions. In such cases, the approved Stage 2 manuscript will be accepted for publication, and authors can propose additional experiments for Stage 1 consideration. Where these experiments extend the approved submission (as opposed to being part of new submissions), the editorial team will seek to fast-track the review process. This option may be particularly appropriate where an initial experiment reveals a major serendipitous finding that warrants follow-up within the same paper. In cases where an incremented submission is rejected (at either Stage 1 or 2), authors will retain the option of publishing the most recently approved version of the manuscript. For further advice on specific scenarios for incremental registration, authors are invited to contact the editorial office at firstname.lastname@example.org.Following Stage 2, full-manuscript publication, the public will have an opportunity to comment on research on the Society’s website (http://storkinesiology.org/ or the journal's OSF page).