Year: 2025
Type: article
Abstract: We read the article by Huber with great interest and appreciation [1]. It effectively underscores the importance of moving beyond the ritualistic reliance on statistical significance and adopting measures that provide a more intuitive interpretation than the p value. We would like to highlight additional essential considerations concerning the use of the S value for this purpose. While it is entirely appropriate to adopt the S value as a measure of statistical evidence against a test hypothesis, typically the null hypothesis of no difference, it must be specified that this interpretation is conditional on the statistical test employed. Different analytical choices (e.g. using a Student's t-test vs. a Welch's t-test) can yield markedly different S values [2-4]. Indeed, the S value provides information against the entire set of statistical model assumptions, not just the hypothesis targeted by the test. As with the p value, all other model assumptions are typically treated as correct for mere simplicity, so the S value provides information only against the targeted hypothesis. However, we emphasise that such simplification is risky, as the S value is not an inherent property of the data or underlying phenomenon, but rather a function of the chosen statistical model and its foundational assumptions. Accordingly, any inference about the targeted hypothesis ultimately depends on the validity of that model. Compared with the p value, one key advantage of the S value lies in its ability to aid the interpretation of interval estimates [2-6]. For instance, a 95%CI – which includes all effect sizes with p > 0.05 under the chosen model – can be viewed equivalently as including all effect sizes associated with S values < 4.3 bits [4-6]. In practical terms, these correspond to hypotheses no more surprising than observing four heads in four fair coin tosses compared with the data, as assessed by the selected statistical test. In this context, ‘fairness’ requires that the probability of observing heads equals that of observing tails – namely, 50% – and that each toss is independent of the others. Consider, for example, a meta-analysis comparing goal-directed haemodynamic therapy with standard treatment, which reports an odds ratio (OR) of 0.70 for postoperative pneumonia, with a 95%CI of (0.49–1.01). According to a misleading interpretation based solely on ‘statistical significance’, this result would be deemed ‘non-significant’ simply because the 95%CI includes the null value OR 1.00 (or, equivalently, because the p value for the null hypothesis is above 0.05). However, such an interpretation overlooks the fact that the data are more compatible with alternative hypotheses. Indeed, the 95%CI includes values indicating a substantial treatment benefit (up to OR 0.49), and the point estimate – i.e. the hypothesis most compatible with the data according to the test – is OR 0.70, and not OR 1.00. We stress that focusing solely on relative measures can be misleading; absolute measures should also be considered when evaluating treatment effects. Thus, all values ranging from meaningful benefit to no effect remain reasonably compatible with the data, as they are less surprising than getting four heads in four fair coin tosses under the data and the model. At most, this reflects considerable uncertainty around the effect and not a lack of clinical relevance. The assessment of the clinical importance of an effect depends on contextual information about the costs and benefits of the therapy [2, 4, 5, 7]. Treatments with severe adverse effects must show substantial beneficial effects, whereas less burdensome therapies may be advantageous even with smaller effects. On this basis, it has been suggested that confidence intervals be renamed compatibility intervals, as they more accurately represent the range of effect sizes most compatible with the data, given the statistical method used to generate the interval [2-7]. In addition, the term ‘confidence’ may imply misleadingly that we should trust the interval, a presumption rarely warranted in biomedical contexts [2-6]. Indeed, even ‘precise’ estimates (e.g. narrow 95%CIs) remain compatible with alternative explanations, such as violated statistical assumptions, protocol deviations, residual confounding or problems in generalisability and transportability [2-6]. Accordingly, confidence in estimates should derive not from statistical results alone, but from a critical appraisal of study design and transparency about study limitations [2, 4, 7], Greenland S, preprint, https://arxiv.org/abs/2508.10168.
Source: Anaesthesia
DOI: 10.1111/anae.16759
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4413990712