Can we trust “Magnitude-based inference”?

Abstract

Since the times and works of William Sealy Gosset (1876-1937) and Ronald Aylmer Fisher (1890-1962), imperfections of conventional null-hypothesis significance testing and in particular, use of P-values to evaluate such testing (invariably referred to as inferential statistics), have been well recognised (Wilkinson, 1999; Wasserstein and Lazar, 2016). Attempts have been made to identify alternatives. For example, Cohen's effect sizes (Cohen 1988) and region of practical equivalence procedure (ROPE) (Kruschke, 2014). A more recent alternative is magnitude-based inference (MBI) (Hopkins and Baterham, 2016) although unlike others, MBI has created considerable controversy when reporting the results of studies (almost exclusively used in the field of sport and exercise science). Instead of defining research effects as “significant” based on P-values (using traditional hypothesis testing), MBI uses terms such as “implementable” and “substantial” based on two constraints called the “risk of harm” and the “chance of benefit”. However, concerns have been raised about the MBI approach. Stanford statistician Kristin Sainani was so concerned about the consequences of using MBI that she wrote a formal analysis of the MBI method. Published in MSSE (Sainani, 2018) her paper showed that, depending on sample size and thresholds for harm/benefit, MBI produces false positive rates that can be two to six times greater than those using traditional hypothesis testing. A finding, she claims, that makes MBI less reliable.