Pick your prior: scepticism about sceptical prior beliefs

It is with initial curiosity followed by admiration but then significant confusion that we studied the recent secondary Bayesian analysis of the COVID STEROID 2 trial comparing dexamethasone 12 mg versus 6 mg for patients with coronavirus disease 2019 (COVID-19) and severe hypoxemia, as well as the accompanying optimistic tweets by the journal and the authors [1].

The findings of high probabilities of benefit on all outcomes—whereas the primary analysis reported no statistically significant difference—is fascinating and may drive immediate change of clinical practice. We fully agree with the authors that the field deserves a more nuanced interpretation of trials than only ‘positive’ or ‘negative’ [2]. Intensivists are natural Bayesians, as we are used to continuously updating our existing beliefs with information from the bedside or scientific literature.

But even though the approach deserves praise, its execution is confusing and may lead to arbitrariness in the interpretation of trials. To illustrate this let us first agree that, in light of the very many therapies found ineffective in critical care medicine, scepticism about treatment benefits seems the most appropriate attitude with which to interpret trial results. We were therefore initially happy to see that the investigators used a so-called sceptical prior distribution and defined a so-called minimum clinically important benefit. For mortality, the clinically important benefit was defined as an absolute risk difference of 2 percentage points or better. The sceptical prior was defined on the log odds ratio scale as a normal distribution with mean 0 (no effect) and a standard deviation (SD) of 0.15.

We see two important problems. Firstly, it takes quite some calculations to understand how the sceptical prior distribution (on the log odds ratio scale) translates to the minimum clinically important difference (on the absolute risk difference scale). With a control-group 90-day mortality rate of 38%, a 2-percentage point absolute risk reduction is equal to an odds ratio of 0.92 ([0.36/0.64]/[0.38/0.62]), equal to a log odds ratio of -0.086, which is equal to a cumulative probability of 0.28 on the normal (0, 0.15) distribution. In plain language: The prespecified prior distribution entailed that there was a 28% prior probability that mortality was 2 percentage points lower in the 12 mg group. We believe that it is unreasonable to expect practicing clinicians to deduce this information from the paper.

Secondly, many readers may feel that a 28% probability of clinically important benefit is not actually very sceptical at all. The problem lies, of course, in the subjectivity of those opinions and in the arbitrariness of the chosen prior distribution. The SD of the prior distribution greatly influences the prior probability and, thereby, the posterior probability of clinically important benefit. Between a prior distribution SD of 0.05 and 0.15, the posterior probability of clinically important mortality benefit ranges from 11 to 68%, given the trial results. Why should the sceptical prior distribution SD be set at 0.15, and not 0.05 or 0.5? In Fig. 1, we show how the prior probability of benefit (defined by the distribution SD) is related to the posterior probability of benefit, given the COVID STEROID 2 trial results.

Fig. 1

The relationship between the prior probability and the posterior probability of clinically important 90-day mortality benefit for the COVID STEROID 2 trial. The prior distribution was prespecified as a normal distribution with mean 0 and a standard deviation (SD) of 0.15 on the log odds scale. This corresponds to a 28% prior probability of ‘clinically significant benefit’, defined as an absolute risk reduction of 2% points or better. With this prior distribution and the trial results (without adjusting for stratification variables), the posterior probability of clinically important benefit is 68%. If the sceptical prior distribution is defined so that there is a 10% prior probability of clinically important benefit (by changing the SD of the sceptical prior to 0.068), the posterior probability of clinically important benefit decreases to 26%

We propose that a sceptical prior distribution should be internally consistent with the specified minimum clinically important benefit. Being sceptical about a treatment could mean, as a rule of thumb, a prior probability of 10% that the true treatment effect is equal or larger than the prespecified clinically important benefit. In the case of the COVID STEROID 2 trial, this would require a normal prior distribution with a mean of 0 and an SD of 0.068 for the 90-day mortality effect. Using this prior and the trial data, the posterior probability of significant mortality benefit is only 26% (Fig. 1). It is with that knowledge that optimistic intensivist tweeps might want to update their prior beliefs and consider withdrawing some of their tweets on this secondary analysis.

Technical note: As we do not have access to the trial data, we used as an example the patient-oriented outcome of 90-day mortality (for which the relevant data can be surmised from the publication) rather than the trial’s primary endpoint of days alive without life support at day 28 (for which the original data would be needed to calculate the posterior distributions). Our posterior probabilities differ marginally from those reported because we do not have the stratification variable data that were used to adjust the estimates.