Background

Hydroxychloroquine (HCQ) is not an effective treatment for established coronavirus disease 2019 (COVID-19) [1, 2], but it is unclear whether HCQ can prevent symptomatic COVID-19. Early in the SARS-CoV-2 pandemic, about 30 randomized trials were designed to study HCQ as prophylaxis for COVID-19 [3]. After the findings from two of these trials were reported in the Summer of 2020 [4, 5], HCQ was generally viewed by the medical community as ineffective for COVID-19 prophylaxis. The emergence of that consensus was surprising because both trials found a lower risk of COVID-19 in the HCQ group, though they were too small to rule out either benefit or harm of HCQ.

A timely completion of the remaining trials would have generated precise estimates of the potential effectiveness of HCQ to prevent COVID-19 among those at high risk of infection or complications. However, the widespread conviction about HCQ’s lack of effectiveness dramatically slowed down the recruitment into ongoing trials of HCQ prophylaxis (one of them carried out by the authors of this report) [6]. As a result, key decisions were made based on insufficient evidence during the pre-vaccine period of the pandemic.

We conducted a systematic review and meta-analysis of randomized trials that study the effectiveness of HCQ to prevent COVID-19 either before known exposure to an infected individual (pre-exposure prophylaxis) or after known exposure to an infected individual (post-exposure prophylaxis).

Methods

Studies were eligible for inclusion if they were randomized clinical trials comparing hydroxychloroquine as prophylaxis (pre-exposure or post-exposure) for COVID-19 with a non-active control, included individuals who had a polymerase chain reaction (PCR) negative test for SARS-CoV-2 at the time of treatment assignment, and had the full text published in a peer-reviewed journal or as a pre-print. Given the relatively small number of trials, we identified candidate studies through expert consultation and confirmed the findings with a search of PubMed, medRxiv and Clinicaltrials.gov as of March 20, 2022, using the search strategies described in the Appendix (first reported as part of a preprint on September 29, 2020 [7]). Two authors (XGA and MAH) independently reviewed the full text of the identified studies and extracted the data. Disagreements were resolved by consulting other co-authors (JdA, RP). Incomplete or unpublished data were requested from the investigators of each trial. The risk of bias of the included studies was assessed independently by 2 authors (XGA, MAH) using the “Rob 2” tool by the Cochrane Bias Methods Group [8].

For each of the identified trials, we obtained or calculated the risk ratio of COVID-19 for assignment to HCQ versus no HCQ (either placebo or usual care) among test-negative individuals at baseline. Our primary analysis is based on the definition of COVID-19 reported in the primary analysis of each study. In addition, we tried to harmonize the definition of COVID-19 across studies by conducting a separate meta-analysis for laboratory-confirmed symptomatic COVID-19.

We calculated, separately for pre- and for post-exposure prophylaxis studies, the pooled risk ratio estimate and its 95% confidence or compatibility interval (CI) using a fixed (or common) effect approach and two types of random effects approaches [9, 10]. Because the standard random effects method may yield an anticonservative (that is, too narrow) 95% CI [9], we also used the Hartung-Knapp random effects method [10], which has been shown to generally outperform the standard random effects method [11]. However, this latter method results in an even more anticonservative 95% CI (narrower than the 95% CI from the standard method) when, as in our meta-analysis, the estimated between-study heterogeneity is small (tau-squared near zero) [12, 13]. We therefore used the Hartung-Knapp method with an ad hoc modification [14] designed to ensure that its 95% CI remains wider than that of the standard method, even though the resulting 95% CI is expected to be conservative (too wide) when, as in our meta-analysis, the number of studies is small [15]. We also computed prediction intervals [16]. The protocol of the review was not registered. All analyses were conducted in Stata (version 16.1; Stata Corporation, College Station, Texas, USA).

Results

Eighty-eight records were identified, of which 16 were duplicates. The screening excluded fifty-nine records because of lack of randomization, use of HCQ as treatment for COVID-19 or, in the case of records from clinicaltrials.gov, the absence of published results. Of the remaining 13 reports, two were excluded because they found no incident events in the HCQ arm [17] or at all [18] (Supplementary Fig. 1 contains the PRISMA 2020 flow diagram). Eleven completed randomized trials were included in the meta-analysis: seven trials studied hydroxychloroquine as pre-exposure prophylaxis [19,20,21,22,23,24,25] and four as post-exposure prophylaxis [4, 5, 26, 27].

Table 1 summarizes the key design characteristics and effect estimates reported by each study. The primary outcome definition varied across trials, with some trials using the presence of symptoms with or without laboratory confirmation [4, 19, 23], others using laboratory-confirmed symptomatic COVID-19 [5, 17, 18, 24, 25, 27] and others laboratory-confirmed SARS-CoV-2 infection with or without symptoms [20, 21, 26]. Our risk of bias assessment identified the handling of incomplete ascertainment of the outcome and the exclusion of patients after randomization as possible sources of moderate bias (see Supplementary Table 1).

Table 1 Summary of randomized trials studying hydroxychloroquine as prophylaxis for COVID-19
Full size table

The seven pre-exposure prophylaxis trials were double-blinded, placebo-controlled trials that included healthcare workers with ongoing exposure to patients with COVID-19 [19,20,21,22,23,24,25]. The occurrence of COVID-19 was ascertained during a period between 4 and 12 weeks. Figure 1 shows the risk ratio estimates from the seven pre-exposure prophylaxis trials. The pooled risk ratio of COVID-19 for hydroxychloroquine vs. no hydroxychloroquine was 0.72 (95% CI: 0.58, 0.90) when using either a fixed effect or a standard random effects approach, and 0.72 (95% CI: 0.55, 0.95) when using the ad hoc modification of the Hartung-Knapp approach. The corresponding pooled risk difference was − 2.1 cases per 100 individuals (95% CI -3.7, -0.6) when using a fixed effect or a standard random effects approach and − 2.1 cases per 100 individuals (95% CI -4.0, -0.2) when using the ad hoc modification of the Hartung-Knapp approach (Supplementary Fig. 1). The pooled risk ratio estimate ranged from 0.71 to 0.74 under the alternative outcome definition and in a sensitivity analysis that used only studies published in peer-reviewed journals [19,20,21,22, 25], but the 95% CIs were wider (Supplementary Figs. 2–3). Supplementary Table 2 contains the prediction intervals.

Fig. 1
figure 1

Risk ratio estimates of COVID-19 for hydroxychloroquine vs. no hydroxychloroquine in randomized trials of pre-exposure prophylaxis, pooled and by study. These estimates are based on the definition of COVID-19 reported in the primary analysis of each study

Full size image

Four post-exposure prophylaxis trials included asymptomatic contacts of confirmed COVID-19 cases. The time from exposure to initiation of prophylaxis was relatively long: in one trial, about a third of participants were enrolled 4 days after exposure (none was enrolled later) [4]; in the other, about 10% of participants were enrolled 7 or more days after exposure [5]; in the third study, 8% of participants received a first dose 5 days or later after exposure [26]. For comparison, post-exposure prophylaxis for HIV is recommended in the first 6–72 h after the exposure [28, 29]. The occurrence of COVID-19 was ascertained during a period of 2 weeks. A fourth post-exposure prophylaxis trial was not labelled as such by the authors, but 60% of the participants reported contact with a COVID-19 positive patients before study entry (time from contact to study entry not reported) [27]. The pooled risk ratio of COVID-19 for assignment to hydroxychloroquine vs. no hydroxychloroquine was 0.91 (95% CI: 0.72, 1.16) when using either a fixed effect or a standard random effects approach, and 0.91 (95% CI: 0.62, 1.35) when using the ad hoc modification of the Hartung-Knapp approach (Fig. 2). Supplementary Table 2 contains the prediction intervals.

Fig. 2
figure 2

Risk ratio estimates of COVID-19 for hydroxychloroquine vs. no hydroxychloroquine in randomized trials of post-exposure prophylaxis, pooled and by study. These estimates are based on the definition of COVID-19 reported in the primary analysis of each study

Full size image

All trials found a similarly low rate of serious adverse effects in the HCQ and no HCQ groups. As expected, the proportion of mild gastrointestinal side effects was greater in the HCQ group. Adherence to treatment was heterogeneous across trials (Table 1). A funnel plot of the included studies is presented in Supplementary Fig. 4.

Discussion

When considered together, the available pre-exposure prophylaxis randomized trials yield a point estimate of an approximately 28% lower risk of COVID-19 for assignment to HCQ compared with no HCQ among PCR-negative individuals at randomization. Any effect between approximately 45% and a 5% reduction in risk is highly compatible with the data from these trials. The pooled effect estimate for the post-exposure randomized trial was closer to the null and both substantial reduction and moderate increase in risk were highly compatible with the data from the trials.

The choice of between common effect and random effects approaches for meta-analysis had little impact on the point estimates because the statistical heterogeneity across studies, as measured by the I2, was close to zero. In the pre-exposure prophylaxis trials, the period over which the outcome was ascertained varied by a maximum of only 4 weeks and, given the long half-life of HCQ [30], variations in the order of weeks are not expected to substantially alter the effect estimates.

Our pooled estimates are based on the design and analytic choices made by the investigators of each trial. Ideally, these investigators could coordinate analyses of individual-level data with standardized outcome definitions; corrections for differences in length of follow-up and treatment dose; adjustment for losses to follow-up and other deviations from protocol [31]; and adoption of a more causally-interpretable meta-analytic approach [32]. All but two pre-exposure prophylaxis trials used laboratory-confirmed infection as the primary outcome; the remaining trials [19, 23], and one of the four post-exposure prophylaxis trials [4], did not require laboratory confirmation and included some self-reported test results. Excluding them resulted in a similar point estimate (Supplement Fig. 2).

As the results of pre- and post-exposure prophylaxis trials appeared between June and September of 2020, the pooled point estimate hovered around a 20% reduction with increasingly narrower compatibility intervals (Fig. 3). Yet throughout this entire period the opinion of many medical researchers was that HCQ was ineffective for prevention and that additional trials were unnecessary. The point estimate moved to around 10% reduction in December 2020 after the publication of a post-exposure prophylaxis cluster-randomized study [26] with unusual findings (no outcomes were recorded among individuals with unavailable PCR at baseline, different proportion of individuals with negative PCR at baseline between randomized groups) that may explain the investigators’ decision to deviate from protocol to adjust for baseline variables [33]. The final pooled risk ratio estimate, including all pre- and post-exposure trials, was 0.80 (95% CI 0.68, 0.95) when using the standard random effects approach and 0.80 (95% CI 0.67, 0.97) when using the ad hoc modification of the Hartung-Knapp approach.

Fig. 3
figure 3

Cumulative estimate of the pooled risk ratio of COVID-19 for hydroxychloroquine vs. no hydroxychloroquine in randomized trials of pre- and post-exposure prophylaxis. Confidence intervals are computed using the standard random effects method

Full size image

The pooled estimates in Fig. 3 combine pre- and post-exposure prophylaxis trials, under the assumption of a similar mechanism of action for both preventive approaches [30]. However, the long time from exposure to treatment initiation in the post-exposure trials (several days as opposed to the 48 h recommended for other viruses like HIV) weakens the case for pooling studies of pre- and post-exposure prophylaxis because the latter can resemble studies of early treatment. Therefore, as the number of completed randomized trials increased in 2021 and 2022, meta-analyses like ours need to present separate estimates for pre-exposure and post-exposure prophylaxis. Such distinction is not always easy: for example, one of the pre-exposure prophylaxis trials identified and isolated 346 PCR positive cases at baseline in the premises where the study was run [21].

This systematic review offers an important lesson for future research on drug repurposing: Recruitment for most trials of HCQ prophylaxis was severely impeded by incorrect interpretations of the evidence from the early, mostly post-exposure prophylaxis, trials. The findings from the first reported trials were widely (and incorrectly) portrayed as definite evidence of the lack of effectiveness of HCQ, simply because they were not “statistically significant” when taken individually. Thus, the common confusion between the concepts “no effect” and “no statistical significance” led many to prematurely conclude that HCQ had no prophylactic effect when the correct conclusion was that the effect estimate was too imprecise [34, 35].

Concerns about potentially deleterious effects of treatment with HCQ [36] and backslash against unsubstantiated claims in the lay media [37] may have also affected the timely completion of the prophylaxis trials. In one study [24], conversations with investigators in participating hospitals confirmed that recruitment was difficult because of toxicity concerns. Of four eligible trials labelled as “completed” or “terminated” in clinicaltrials.gov but without posted results (NCT04414241, NCT04374942, NCT04372017 and NCT04344379), at least one had to be terminated because of poor recruitment, and another was closed after recruitment of one patient because researchers considered that the risk/benefit balance of HCQ was unfavorable (personal communications from the investigators). Yet these safety concerns were unwarranted as demonstrated by the findings of the randomized trials included in our meta-analysis. We encourage the authors of living systematic reviews [38] to update their findings frequently when, as in this case, public opinion interferes with the generation of the very evidence that will feed their systematic review.

In summary, throughout much of the pre-vaccine period of the pandemic, the available evidence was compatible with HCQ being viable as prophylaxis. Yet an incorrect interpretation of early inconclusive studies interfered with recruitment into additional trials. Though the availability of effective COVID-19 vaccines reduces the need for pharmacological prophylaxis, it is important to improve the process by which the medical community generates and interprets evidence before the next public health emergency arrives. To avoid he proliferation of small studies with different methodologies, national regulators and international health organizations can play a key role in the coordination and harmonization of the design of the randomized trials that they approve or endorse.