Abstract

Recently, a group of editors from physiotherapy journals wrote a joint editorial on the use of statistics in their journals. Like many editorials before them, the editors, who were not statistical experts themselves, put forth numerous recommendations to physiotherapy researchers on how to analyze and report their statistical analyses. This editorial unfortunately suffers from numerous mischaracterizations or outright falsehoods regarding statistics. After a thorough review, two major issues appear throughout the editorial. First, the editors incorrectly state that the use of confidence intervals (CI) would alleviate some of the issues with significance testing. Second, the editors incorrectly assume “smallest worthwhile change” statistics are immutable facts related to some ground truth of treatment effects. In this critical review, we briefly outline some of the problematic statements made by the editors, point out why it is too premature to adopt an estimation approach relying on a minimal clinically relevant difference, and offer some simple alternatives that we believe are statistically sound and easy for the average physiotherapy researcher to implement.

Misleading Statements about Statistics

At a foundational level, the goal of NHST is to make inferences with an eye towards error control and the goal of Estimation, whether Frequentist or Bayesian, is to quantify the magnitude of an effect and the uncertainty of the estimate. As Elkins et al (2022) also points out (page 2, paragraph 6), there is a mathematical relationship between the p-values calculated through NHST and confidence intervals around model estimates (Altman & Bland, 2011). For this reason, it is surprising the number of misstatements made within the Editorial (2022) regarding NHST and CIs. For example, Table 1 in the Editorial (2022) states “Statistically significant findings are not very replicable”; however, when exactly reproducing a study repeatedly in the same population with different samples, one would have the exact same replication characteristics for both p-values and CIs. This appears to be a misinterpretation of Boos & Stefanski (2011) which was primarily focused on the reported precision of p-values (i.e., reporting p = 0.0123 vs. p < 0.025) and how the average study is underpowered (~67% power) so an exact replication is unlikely to yield a significant result¹. This is a point directly addressed by Lakens (2022) in his review of the Editorial, but was seemingly misinterpreted again in their response (2022). The authors also seem to forget that a move to CIs would suffer from these exact same issues and would not magically solve the problem of replicability (Hoekstra et al., 2014; Morey et al., 2015).

Table 1 also makes some very peculiar assumptions about interventions in clinical trials by stating without evidence that “Almost all interventions would be expected to have some effect, even if that effect was trivially small”.² It is possible this is inelegant wording, and the intention was to state that, within a given trial, it is highly unlikely for a measured construct to be exactly nil. It could be that the the Editors (2022) are making a vague allusion to Lindley’s paradox (Lindley, 1957) and, that given a large enough sample size (i.e., high statistical power), NHST will yield a significant effect even when the difference is itself of no practical value (Rouder et al., 2009). In fact, in situations of very high statistical power, a p-value close to the significance threshold (e.g., p = 0.045) would be more likely under the null hypothesis than the alternative hypothesis (Maier & Lakens, 2022). All of this is technically true, but it ignores that the alpha does not need to be fixed at 5%. The statement by the Editors (2022) ignores the Neyman-Pearson approach of balancing type 1 and type 2 errors. The alpha level could be lowered in situations where negligible effects could be detected (thereby balancing the type 1 and type 2 error rates) (Maier & Lakens, 2022). Even if the null were never exactly true (which we believe is a unjustified claim), secondary equivalence testing could be utilized to prevent small effects from being declared as “significant” when they are practically equivalent (Campbell & Gustafson, 2018). The related statement in the Editorial (2022) that “All trials should therefore identify an effect” (Table 1), is simply not justifiable in any case that we can envision. It is often unclear if the Editorial (2022) is talking about an effect measured by a statistical model/test (which can always be wrong for a variety of reasons) or a “real” effect which can never be truly known in empirical work.

Finally, there is a bit of irony in that while the Editorial (2022) states, “it is possible to put a confidence interval around any statistic, regardless of its use, including mean difference, risk, odds, relative risk, odds ratio, hazard ratio, correlation, proportion, absolute risk reduction, relative risk reduction, number needed to treat, sensitivity, specificity, likelihood ratios, diagnostic odds ratios, and difference in medians.” They omit the fact that SWE or Minimal Clinically Important Difference (MCID) metrics can and should also be reported with confidence intervals. These estimates of “clinical relevance” are subject to the same sampling errors as an estimate of treatment effect.

Smallest Worthwhile Effect and Minimal Clinically Important Difference Values Are Estimates with Uncertainty

A failure to recognize the empirical ambiguity in the SWE/MCID metric is a fatal flaw in the Editorial (2022) as the primary thesis and remediation for supposed ills of NHST are to examine the overlap between effect estimates and the SWE/MCID. While SWE/MCID can be useful concept, it is not an immutable ground truth. We have no problem with establishing some SWE/MCID threshold for an individual study, but care has to be taken in the usage and interpretation of such thresholds. Many researchers, including one author of this manuscript (Tenan et al., 2020), have noted that there are a multitude of issues with SWE/MCID measures reported in the literature. Additionally, evidence from previous attempts to abandon NHST (i.e., “magnitude based inference” or “MBI”) indicates that researchers are more likely to adopt standard thresholds³ rather than develop empirically based MCIDs (Lohse et al., 2020). All of these issues with MCIDs/SWEs preclude the immediate use of the “estimation” method suggested in the Editorial (2022), and should give some pause when establishing or utilizing an SWE/MCID.

Alternative Hypothesis Tests

The Editorial (2022) sets out to tell researchers that p-values should not be used but the only method which makes their proposed NHST alternative statistically valid is, in fact, a p-value. If one can detach themselves from the some of misleading statements in the Editorial (2022), the concept that researchers should think more critically about their research questions and analyses is an excellent suggestion. In fact, if we are willing to accept that SWE/MCIDs are not immutable facts but rather “reasonably good thresholds in certain circumstances”, similar to an alpha level of 0.05, there exists a NHST-based framework that seems to approximate the goal of comparing a sample to a “clinically meaningful bound” against sample population estimates: superiority, equivalence, non-inferiority, and minimal effects hypothesis tests (Caldwell & Cheuvront, 2019; Mazzolari et al., 2022). Therefore, many of the goals outlined in the the Editorial (2022) could very well be accomplished with NHST and p-values.

Conclusions

We are sad to see yet another example of scientists making claims about statistics beyond their expertise (Sainani et al., 2020). The unfortunate reality is that authoritative papers such as the Editorial (2022) can do real damage to the field of physiotherapy. First, the incorrect information provided in the Editorial (2022) will undoubtedly mislead physiotherapy researchers towards worse statistical practices by providing misinformed beliefs about NHST and continuing the trend of believing an MCID/SWE is an immutable threshold. We fear that the mindless implementation of another threshold, much like the perfunctory use of the 0.05 significance threshold (Hopewell et al., 2009), will fail to improve the quality of research and only create a new form of publication bias. Similar frameworks, such as MBI (Lohse et al., 2020), did not improve statistical practice among sport scientists, and when broadly implemented can cause more harm than good (e.g., adopting a new threshold of 0.2 standard deviations as SWE/MCID). Second, editors stating their preferred statistical methods implicitly coerces authors submitting to those journals into performing and reporting statistical analyses they do not find useful. Misguided commentaries from editorial boards are nothing new within academic publishing (Mayo, 2021). We would caution all non-statisticians to avoid making such sweeping statements about proper statistical practice without the involvement of a variety of statisticians. Even statisticians have diverse viewpoints on how statistics should be applied to the analysis of data (e.g., Frequentist versus Bayesian schools of thought), and editorial commentaries should not be the place for picking philosophical sides. Instead, editorial commentaries should be focused on improving the reporting of statistics within their journals to ensure whatever analytical approach is used is appropriately reported for public consumption.⁷

Additional Information

Appendix 1: Vignette Analysis with TOSTER

We can use the tsum_TOST function within the TOSTER R package to perform the required statistical tests. We should note that both authors would prefer to use an ANCOVA to analyze results from a pre-post study.

library(TOSTER)
test1 = tsum_TOST(
  m1 = 55.8, m2 = 37, # Means
  sd1 = 53.8, sd2 = 30.7, # SD
  n1 = 78,  n2 = 78, # Sample Sizes
  hypothesis = "EQU", low_eqbound = -12, high_eqbound = 12)
test1$decision$ttest

## [1] "The null hypothesis test was significant, t(122.34) = 2.680, p = 8.37e-03"

test1$decision$TOST

## [1] "The equivalence test was non-significant, t(122.34) = 0.970, p = 8.33e-01"

test1$decision$combined

## [1] "NHST: reject null significance hypothesis that the effect is equal to zero \nTOST: don't reject null equivalence hypothesis"

We can also provide a plot of the estimates with multiple confidence intervals.

plot(test1)

A visualization of the cumulative distribution function with 4 levels of confidence being displayed for the standardized mean difference (top panel) and the mean difference (bottom panel)

The interpretation provided above takes a Neyman-Pearson perspective. Both the NHST and TOST tests have an alpha-level of 0.05 and one reached significance will the other did not. Therefore, an author using this approach would have to conclude that one null hypothesis is rejected regarding GLU while other other null hypothesis is rejected.

However, those who wish to use an estimation approach may have a different interpretation. Under the approach outlined by Rafi & Greenland (2020), we could instead look at the data and see how “compatible” the data is with each competing hypothesis (i.e., NHST versus TOST). From this perspective, the interpretation is much more fluid, and one could conclude that the data is more incompatible with “no effect” than “equivalence” (p-values of 0.008 and 0.83, respectively).

Both perspectives are valid and it is up to researchers to decide how they plan to tests or estimate their effects. Again, it our belief that researchers, not editorial boards, are usually the better judges of what statistical framework is best for their research questions. However, researchers should be consistent with whatever language/framework (e.g., estimation or NHST) within each study/manuscript.