Introduction

Results from randomized controlled clinical trials (RCTs) have transformed cardiovascular medicine from a field that was anecdote-driven into one that is evidence-based. At present, the introduction of a medication, device or diagnostic method into clinical practice without thorough evaluation of clinical outcomes in RCTs is inconceivable. However, as investigators, regulators, policy-makers and patients become more reliant on evidence-based results to support clinical care, the failure of contemporary RCTs to address the entire spectrum of clinically relevant questions is becoming increasingly apparent. Most cardiovascular RCTs conducted to date have been primarily ‘explanatory’ in nature; that is, they were designed to study the efficacy of interventions in ‘optimized’ conditions, with highly selected patient populations and protocolized assessments of safety and efficacy1,2. Although such trial designs increase internal validity, they provide limited information about the effectiveness of the intervention in the great variety of scenarios that exist in routine clinical practice but were not represented in the RCT, thereby compromising the generalizability of the trial results1,3. Therefore, whereas the demonstration of efficacy in explanatory RCTs is commonly the stepping-stone for an intervention being approved by regulators and introduced into the market, the degree to which the intervention is beneficial in routine clinical practice often remains unclear. Moreover, explanatory trials are often burdensome, with high demands for eligibility of participants, site and data monitoring, and local research staff and investigators. Therefore, this method of generating evidence is both resource and time intensive, which results in substantial opportunity cost: a number of clinically relevant questions are unable to be assessed and there are delays to application.

The concept of ‘pragmatism’ in clinical trials addresses concerns about the limited generalizability of findings from explanatory RCTs and their burdensome nature2,4. The central tenets of pragmatism in RCTs are twofold. Firstly, to test interventions in routine clinical settings with patients who are broadly representative of the condition under study. Secondly, to streamline clinical trial conduct by minimizing trial-specific visits and testing to reduce the burden on investigators and participants so that the costs and complexity of the trials are reduced. Owing to the high number of RCTs conducted in the field, cardiovascular medicine is a leading discipline in evidence-based clinical care and is in a unique position to lead the shift towards pragmatism in clinical trials. The purpose of this Review is to shed light on the need for increased pragmatism in cardiovascular RCTs across all disciplines, assess current challenges and potential solutions to the implementation of pragmatism in clinical trials, and highlight selected cardiovascular RCTs with pragmatic trial designs.

The explanatory–pragmatic continuum

Characterizing trial designs

Pragmatism in clinical trials exists on a spectrum, and the level of pragmatism in trial design should align with the goals of the trial. Explanatory RCTs generally address the question ‘can the intervention work and, if so, how well?’. Therefore, the design of explanatory trials is centred around maximizing the estimated effect size and safety of the intervention. These trials often involve stringent eligibility criteria that identify participants who are most likely to benefit from treatment and least likely to experience adverse effects5, delivery of the intervention by experts, and stringent monitoring and follow-up. Another element of explanatory trial design is the use of intermediate outcomes (end points that have clinical benefit but are not ‘hard’ events) such as improvement in quality of life or relief of dyspnoea6,7. Intermediate outcomes allow the statistical analysis of treatment effects using smaller sample sizes but might be of limited relevance to patients and have uncertain associations with clinical outcomes. Explanatory RCTs maximize power by minimizing dropout and non-adherence to trial protocol as well as factors that increase variability in the estimates of treatment effect. The motivation for conducting explanatory RCTs is often to demonstrate the effectiveness of an intervention to meet regulatory requirements for product-labelled indications.

By contrast, pragmatic RCTs generally attempt to answer the question ‘does the intervention work in the general population?’. These trials use broader eligibility criteria than those for explanatory trials, which maximize the heterogeneity of participants to better reflect the relevant clinical population, providers and health-care settings. Pragmatic RCTs minimize protocolized clinical evaluations and testing beyond standard care. These aspects of trial design minimize the research burden on participants, investigators and providers but require complex coordination and multi-level stakeholder engagement. Pragmatic designs inevitably increase non-adherence to the trial protocol, dropout and crossover, thereby requiring larger sample sizes for adequate statistical power.

Measuring pragmatism in RCTs

The Pragmatic Explanatory Continuum Indicator Summary (PRECIS) tool8, introduced in 2009 by Thorpe and colleagues, allows clinical trialists to somewhat quantitatively assess the level of pragmatism in their trial design. In 2015, this tool was revised through a collaborative and iterative process involving 80 international trialists, leading to the PRECIS-2 tool9. Since its introduction, the PRECIS-2 tool has been used to design >500 RCTs. Figure 1 provides a visual depiction of the PRECIS-2 toolkit, in which the pragmatism of a trial design is rated on each of nine domains: eligibility, recruitment, setting, organization, flexibility in delivery, flexibility in adherence, follow-up, primary outcome and primary analysis. Through team-based discussions, each of the nine domains of the tool are scored on a five-point scale, with a score of one reflecting a totally explanatory characteristic and a score of five reflecting a totally pragmatic characteristic. Of note, the PRECIS-2 tool was developed to help investigators to match their design choices to their intended degree of pragmatism before conducting the trial. Whether the PRECIS-2 is a reliable tool to retrospectively rate the pragmatism of study designs is unknown, although such analysis has been performed10,11.

Fig. 1: The PRECIS-2 wheel.
figure 1

A visual representation of how pragmatic a trial is on the explanatory–pragmatic continuum. For each of the nine domains, the level of pragmatism can range from 1 (most explanatory, least pragmatic) to 5 (least explanatory, most pragmatic). Adapted with permission from ref.9, BMJ.

Full size image

The need for pragmatism

The need for pragmatic cardiovascular RCTs is emphasized by the disconnect between outcomes observed in explanatory clinical trials and those in routine clinical settings. Table 1 highlights questions that often remain unanswered in traditional cardiovascular RCTs and how a shift towards pragmatism can help to provide answers. The case of heart failure (HF) is pertinent in this context. The past three decades have brought considerable advances in the field of HF, particularly in HF with reduced ejection fraction (HFrEF)12. Sodium–glucose cotransporter 2 inhibitors, angiotensin receptor–neprilysin inhibitors, β-blockers and mineralocorticoid receptor antagonists have all demonstrated significant reductions in all-cause mortality in RCTs of patients with HFrEF12. Based on these findings, quadruple therapy is currently recommended, which is predicted to reduce the risk of death by up to 73%13. In addition to pharmacotherapy, device therapies such as implantable cardioverter–defibrillators and cardiac resynchronization therapy have been adopted14,15. Additionally, results from RCTs of alternative structures and processes of health-care delivery have documented the value of multidisciplinary teams and specialist nurses in HF16,17. Despite these advances, multiple longitudinal studies have demonstrated that the reduction in the risk of death is far below that expected in high-income countries — a moderate initial decline in mortality was followed by a plateau and a consistent incidence of death over the past 15 years18,19,20,21. As in HF, hospitalization rates and all-cause mortality have also stabilized in the past few decades for other chronic cardiovascular conditions, such as atrial fibrillation and valvular heart disease, despite therapeutic advances made on the basis of RCT results22,23,24.

Table 1 Pragmatic trial designs can answer important clinical questions
Full size table

Several explanations are possible for why the extent of benefit seen in RCTs of HF therapies is not reflected in routine practice. First, epidemiological studies have shown that, whereas cardiovascular mortality among patients with HF has declined over time, the proportion of non-cardiovascular death in these patients is increasing, resulting in negligible changes in all-cause mortality21. The increase in non-cardiovascular mortality is likely to be driven by the increasing prevalence of comorbidities as the population ages. In explanatory RCTs, patients are selected if their risk of death is primarily related to the condition being treated and patients with a high competing risk of death are often excluded so that reductions in mortality with the intervention are easily demonstrated. Pragmatic RCTs are needed to assess whether new cardiovascular therapies can reduce all-cause mortality and improve quality of life in routine clinical settings or if any improvement in cardiovascular outcomes would simply be substituted by an increase in non-cardiovascular morbidity. In the latter case, pragmatic RCTs provide an important platform to test comprehensive interventions that also target non-cardiovascular mortality. Second, the uptake of RCT-proven HF therapies in routine clinical practice lags substantially behind the data25,26. Clinicians might not apply findings from explanatory RCTs to patients because of uncertainty about the ‘real-world’ effectiveness and safety of these medications in their patients. This lack of outcome expectancy could be particularly prominent in patients with HF and multiple comorbidities, who are often under-represented in conventional RCTs27. Pragmatic RCTs can potentially elucidate the effectiveness of therapies in routine practice and highlight barriers to implementation that need to be addressed.

Observational studies are insufficient

Observational studies can generate ‘real-world evidence’ and are often designed to test the hypotheses of explanatory RCTs in routine clinical settings. Clinical trial findings are often assumed to be applicable in the real world if they are supported by results of observational studies with valid methodologies. However, observational studies can be prone to important selection biases (including healthy-user bias, socioeconomic bias, health-care access bias, immortal time bias, confounding by indication, reverse causality and outcome ascertainment bias) that can potentially lead to inaccurate inferences28,29. Therefore, when the findings between RCTs and observational studies are discordant, the results from RCTs are considered to be more reliable and practice rarely changes. For example, observational studies have demonstrated an association between vitamin supplementation and reduced cardiovascular mortality but, in large, well-conducted RCTs, supplements failed to improve cardiovascular outcomes30. The findings from observational studies were attributed to healthy-user bias, wherein participants taking vitamin supplements had other heathy lifestyle habits that were unaccounted for30. In the presence of sufficient data, certain analytical techniques, such as propensity-matched analysis and instrumental variable analysis, can reduce biases and confounding and improve the accuracy of treatment effect estimates in the absence of RCTs. However, the possibility of residual confounding can never be completely eliminated in an observational study. Furthermore, no analytical techniques are widely accepted as the standard, and different techniques can yield conflicting results. For example, in an analysis of the AFFIRM trial database, digoxin use (when analysed as a time-dependent variable) was associated with an increased risk of death in patients with atrial fibrillation31. By contrast, in another analysis of the same database, mortality was not increased among patients who started digoxin therapy at baseline compared with patients who did not32. Therefore, depending on the statistical methods used, observational analyses of the same data set aiming to answer the same question can yield different findings, making it difficult to interpret the clinical implications of an intervention. In this context, pragmatic RCTs offer the best of both worlds — randomization together with insight into the real-world effectiveness of an intervention. Figure 2 provides an overview of the various biases in observational studies, pragmatic RCTs and explanatory RCTs. In the following sections, we compare the characteristics of explanatory and pragmatic RCTs across the domains of the PRECIS-2 toolkit9.

Fig. 2: Relative likelihood of biases and limitations in each type of clinical study.
figure 2

Biases might not always be present, and each study should be individually assessed. Lack of generalizability: limited representation of patients who would receive the intervention and providers who would deliver the intervention in routine practice. Hawthorne bias: change in behaviour or perceived effect in patients as a result of awareness of being observed. Confounding bias: a distortion in the measure of the association between an exposure and a health outcome as a result of extraneous factors that are independently associated with both the exposure and the outcome. Prevalent user bias: occurs when users and non-users of a study intervention are compared without a fixed ‘time zero’; patients who start or continue using a particular intervention are likely to differ in characteristics from non-users or those who discontinue treatment. Immortal time bias: patients in the treatment group are more likely to have longer survival times or less serious disease than those in the control group because, owing to the study definition, patients in the treatment group cannot experience the outcome in the period between the start of the study and the initiation of treatment. Observer bias: a researcher’s expectations, opinions or prejudices influence what they perceive or record in a study; can occur in the absence of blinding.

Full size image

Comparisons across PRECIS-2 domains

Eligibility

Explanatory RCTs

Conventionally, cardiovascular RCTs have included selected populations33. For example, trials often exclude older adults (aged >75 years), women of child-bearing age and those with common comorbidities33,34,35,36,37 due to concerns about safety, drug interactions and the presence of competing causes of death. The under-representation of these groups in RCTs limits our understanding of the safety and efficacy of interventions in these populations, rendering optimal drug combinations and doses in these patients uncertain. By contrast, certain trials limit inclusion to patients with severe disease in order to accumulate event rates with a small sample size and short follow-up period5,37. In this case, the efficacy of interventions in a healthier population remains unclear. Therefore, the narrow eligibility criteria used in explanatory RCTs often result in the enrolment of homogeneous, and potentially non-representative, patient populations.

The MITRA-FR38 and COAPT39 trials were both designed to study the role of transcatheter mitral valve repair in patients with HF and functional mitral regurgitation. However, these trials yielded strikingly different results. The COAPT trial39 demonstrated significant reductions in HF-related hospitalization and mortality and improved quality of life with transcatheter mitral valve repair, whereas MITRA-FR38 showed that transcatheter mitral valve repair had no effect on these outcomes. The discordant results could be due to subtle differences in the inclusion criteria for each study. In the COAPT trial39, patients with more severe functional mitral regurgitation and less dilated ventricles (that is, valve dysfunction was a major component of the disease) were included, whereas in MITRA-FR38, patients with severely dilated ventricles were enrolled. These trials also had different thresholds for defining the failure of guideline-directed medical therapy, which could also have altered the benefits of transcatheter mitral valve repair.

Another example of treatment efficacy differing with patient characteristics is seen with the PARADIGM-HF40 and LIFE41 trials, both of which evaluated the efficacy of sacubitril–valsartan (LCZ696) in patients with HFrEF. In the PARADIGM-HF trial40, patients with mild-to-moderate HF were primarily included, whereas patients with advanced HF were enrolled in the LIFE trial41. Again, the two trials showed markedly different results, with sacubitril–valsartan demonstrating large and significant reductions in the plasma levels of N-terminal pro-B-type natriuretic peptide (NT-proBNP; a marker of prognosis in patients with HF) in the PARADIGM-HF trial40 but no significant change in the LIFE trial41. Therefore, variations in patient populations can considerably modify the efficacy of an intervention, and the high costs of RCTs often restrict multiple investigations in various patient populations.

Pragmatic RCTs

Pragmatic RCT eligibility criteria allow the inclusion of patients in whom the results can be generalized to real-world practice. These studies involve as parsimonious a list of eligibility criteria as possible so that the target population is most representative of routine clinical practice. This approach can answer valuable questions about how cardiovascular interventions work across age groups, sexes and racial/ethnic groups and in patients with complex combinations of comorbidities and prescription medications.

Recruitment

Explanatory RCTs

Recruitment has historically been a challenge in cardiovascular RCTs. For instance, in phase II–IV trials for HF, the mean enrolment rate is ~0.5 patients per site per month42. Investigators historically disseminate recruitment information about trials through their networks, health-care settings and mass mailings43. This approach inevitably leads to selection bias: only those participants who hear about the trial can potentially be recruited. Despite these efforts, unsuccessful recruitment is the leading factor in clinical trial failure44. Furthermore, current recruitment practices often result in certain populations — particularly women and ethnic minority groups — being under-represented relative to the prevalence of disease in these groups37,45. A bibliometric review published in 2020 demonstrated that women were significantly under-represented compared with disease distribution in pivotal trials of acute coronary syndrome (participation-to-prevalence ratio 0.68), HF (participation-to-prevalence ratio 0.58) and coronary heart disease (participation-to-prevalence ratio 0.52)46; similar findings were seen for minority groups46. The under-representation of women and minority groups can extend beyond their quantitative under-recruitment into the study to the actual qualitative characteristics of the patients enrolled. For example, an analysis of the ASCEND-HF trial47 showed that the magnitude of differences between trial participants and real-world patients in the USA was magnified among women and Black patients, when compared with men and white patients. Therefore, challenges arising from the low proportions of women and minority groups enrolled in clinical trials are compounded by concerns that those women and individuals from a minority group who are actually enrolled might not fully represent these populations in routine practice.

Pragmatic RCTs

In pragmatic trials, entire practices or registries (rather than individual patients) are often targets for inclusion, increasing the likelihood that a large and diverse population is recruited. Recruitment is commonly community based and reaches people who might not typically access health-care settings48. Moreover, the likelihood of recruitment increases due to the minimal requirement for in-person follow-up and trial-specific labs and imaging. A reduced trial burden is particularly likely to increase the representation of historically marginalized groups with transportation, financial or time constraints due to work hours or caregiving roles. In pragmatic RCTs, randomization can be performed at either the site or the patient level.

Setting and flexibility in delivery

Explanatory RCTs

In explanatory trials, treatment is usually delivered by experts at specialist centres. However, in real-world practice, diagnostic evaluation and treatment are often conducted by non-experts and at non-specialist centres. The effect of an intervention might vary depending on the personnel who deliver it and the setting in which it is delivered. Moreover, explanatory RCTs often require the intervention to be delivered as per the protocol, whereas delivery of the intervention is more flexible in routine clinical practice.

The GUIDE-IT trial49 was designed to study whether NT-proBNP-guided HF treatment allows accurate titration of guideline-directed medications and improves clinical outcomes. The NT-proBNP-guided strategy resulted in no additional benefit over usual care. However, the results of this trial could have been neutral because the majority of study sites had expertise in delivering care for HF. Physical assessment by an expert has been shown to be as accurate as NT-proBNP level in assessing congestion50, rendering the NT-proBNP measurements redundant in this setting. If more non-specialist sites that routinely provide HF care (primary-care and secondary-care institutions) had been included in the study, the likelihood of detecting a demonstrable benefit with an NT-proBNP-guided strategy might have been increased.

Pragmatic RCTs

Pragmatic trials can be embedded within usual clinical practice and, because they typically require less infrastructure and fewer dedicated research personnel, can include a range of practices or centres. Furthermore, in pragmatic RCTs, treatment is delivered by practitioners who would generally offer the intervention in real-world settings51. As such, pragmatic RCTs can also elucidate differences in how the intervention works in various health-care settings, such as hospitals, clinics or physician practices, and develop real-world strategies for the identification and treatment of eligible patients.

Flexibility in adherence

Explanatory RCTs

In explanatory trials, ensuring adherence to the assigned treatment is an important objective for investigators for two reasons. Firstly, to ensure that efficacy is estimated accurately and, secondly, to meet regulatory requirements, which often prioritize high adherence rates52. Adherence is generally monitored by direct enquiry, retrospective questionnaires, diary evidence and direct measures of study medication consumed52. In some cases, trials have run-in phases to select patients who are adherent and most likely to tolerate the intervention40,53. Frequent monitoring of this type results in the Hawthorne effect (wherein participants modify their behaviour in response to their awareness of being observed), which results in higher adherence than that seen in routine practice. Although measures to promote adherence can improve the efficiency of a trial, they also lead to overestimation of treatment efficacy relative to non-experimental conditions. Moreover, the use of run-in periods means that the patients who are ultimately enrolled in an explanatory RCT are those who tolerate a period of medication use, but patient tolerance is never known when starting a treatment in routine clinical care.

The HOMERUS trial54 was designed to assess whether treatment decisions based on home blood-pressure measurements led to a reduction in the use of antihypertensive drugs and associated costs. An analysis of adherence rates showed that enrolment in the trial led to a significant increase in adherence to both trial and non-trial medications, and adherence rates decreased again when the trial ended55. Similarly, low adherence to medications in non-trial settings has been noted in several other cases such as the use of statins for hyperlipidaemia56 and the use of spironolactone for HFrEF57.

Pragmatic RCTs

To evaluate the true effectiveness of an intervention in the real world, flexibility in adherence could be allowed, and run-in phases and excessive monitoring should be avoided. However, tracking medication use (for example, through pharmacy dispensation data) is important and can elucidate reasons for non-adherence and help to develop strategies to improve adherence to treatment. Nonetheless, excessive flexibility, with many patients crossing over between treatment groups, can result in the effect of the intervention becoming unclear and should be avoided.

Organization and follow-up

Explanatory RCTs

Large-scale, explanatory RCTs often require additional infrastructure, which is costly and generally has little value for reuse58,59. Moreover, extensive amounts of data are collected in these trials, much of which is used to understand physiology and mechanisms of potential benefit. Although collecting such data on this scale can provide important insights, costs are substantially increased and the ability to assess questions directly applicable to care is limited.

The time and monetary investments required to organize a conventional trial, sponsored by a pharmaceutical company, are usually not disclosed but estimates are available. Researchers in conjunction with pharmaceutical industry experts estimated the resources required to execute an open-label, phase III trial evaluating the efficacy of a drug in acute coronary syndrome60. The total cost of the hypothetical trial was predicted to be US$ 83 million, with almost 80% of this cost stemming from site-related (site monitoring and payments) and data management expenses. The projected time for the entire study (from planning to manuscript submission) was 32 months. The authors estimated that reducing the amount of data collected and the site-related expenses could reduce the cost of the trial by >40%60.

Pragmatic RCTs

Pragmatism in clinical trials involves collecting data during routine rather than trial-specific care. Any additional follow-up needed can be conducted virtually or over the telephone. Instead of having specialist teams to collect data, electronic health records (EHRs), patient registries and administrative databases are often used. These methods of data collection can be less cumbersome, cheaper and potentially more applicable to care than those used in explanatory RCTs. Reduced trial burden can encourage the participation of both treatment centres and patients, and reduced costs allow the enrolment of larger patient populations and follow-ups of longer duration. In this way, well-powered assessment of important outcomes, such as cardiovascular and all-cause mortality, is facilitated. This approach is exemplified by the TASTE trial61, which was designed to test the effectiveness of routine intracoronary thrombus aspiration before percutaneous coronary intervention in 7,244 patients with ST-segment elevation myocardial infarction. In this trial, a clinical population-based registry — the national comprehensive Swedish Coronary Angiography and Angioplasty Registry — was used to simplify the process of patient enrolment and data collection. Leveraging the existing clinical infrastructure resulted in low costs (US$ 350,000 or approximately US$ 50 per patient).

The limited organization and follow-up required to conduct pragmatic trials also offers the potential to conduct robust research in regions with limited research infrastructure and trained personnel, which are grossly under-represented in trials relative to disease prevalence1. For example, the SSaSS trial62, which was conducted in 600 rural villages in China (~21,000 participants), demonstrated that the use of a salt substitute (75% sodium chloride and 25% potassium chloride) reduced the risk of stroke compared with regular salt (100% sodium chloride). A similar study, in which 2,376 participants from 6 villages in Peru were enrolled, showed that use of a salt substitute significantly reduces the risk of developing hypertension48.

Outcomes and analysis

Explanatory RCTs

Explanatory outcomes include surrogate or physiological primary end points, which can indicate the efficacy of an intervention when established concordance exists between the surrogate end point and clinically important outcomes. As this situation does not always exist, clinical primary end points are preferred in phase III trials because they are most relevant to policy and clinical care decisions63. Often, conventional cardiovascular RCTs measure physiological parameters as secondary end points in addition to primary clinical outcomes to provide insight into the mechanism of action of the intervention. However, this strategy adds substantially to the burden of data collection and the trial in general and is of limited relevance to patients and stakeholders.

Pragmatic RCTs

Pragmatism involves the study of outcomes that are relevant to patients such as survival, quality of life and functional status. Measurement of quality of life and functional status can be considered as allowing the ‘patient’s voice’ to be heard. Assessing these outcomes is important to provide a holistic perspective on the efficacy of an intervention, thereby informing public expenditure on health care. Pragmatic outcome assessment is particularly important for chronic, or end-stage, cardiovascular diseases associated with deteriorating functionality and quality of life. Although progress in this area has been slow, contemporary RCTs frequently evaluate patient-relevant outcomes, especially in HF. Pragmatic RCTs are well suited to test quality of life, which can be evaluated remotely through online questionnaires. Moreover, wearable devices can provide detailed insight into the functional status of patients. However, because these outcomes are not routinely evaluated, processes and infrastructure to record them have to be set up separately in a pragmatic setting. Moreover, accurate and complete assessment of patient-relevant outcomes in a remote setting can be challenging given language, education and cognitive barriers.

Leveraging technology for pragmatism

As we have discussed, explanatory clinical trials have several inefficiencies in terms of identification, recruitment and follow-up of participants as well as in data acquisition and analysis. Leveraging EHRs and mobile, wearable technologies has the potential to revolutionize the operational aspects of clinical trials and allow a shift towards pragmatism64,65,66.

EHRs can be queried to identify individuals who meet trial eligibility criteria. These patients can then be invited for a screening visit by their primary-care provider via e-mail, a telephone call or EHR portal. If an in-person visit is not essential, e-consent could be sought directly. Systems can be implemented to alert health-care providers of eligibility for trials when a patient visits the office. These measures would minimize recruitment efforts and maximize the number of eligible patients approached for enrolment into the trial64,65. Moreover, when patients have been recruited, EHR and linked administrative data sets allow the automatic capture of baseline and follow-up data, thereby substantially reducing the burden for front-line study personnel. Furthermore, smartphone applications, biosensors and wearable technologies can continuously track a participant’s heart rate and rhythm and daily activities, such as exercise and sleep quality, providing insight into a participant’s cardiovascular health and lifestyle64,65.

Dissemination and implementation

Cardiovascular interventions that are based on evidence from RCTs are often poorly implemented in routine clinical practice, which explains why the benefits demonstrated in RCTs might not be reflected in epidemiological studies25,26. The field of dissemination and implementation (D&I) research is focused on identifying barriers to the implementation of evidence-based interventions and potential solutions to overcome these barriers. Pragmatic RCTs can have an important role in D&I research. By allowing flexibility in intervention delivery and adherence and encompassing a broad patient population potentially eligible for evidence-based interventions, pragmatic RCTs can assess the frequency of uptake in routine clinical settings. Furthermore, in the absence of stringent trial protocols, investigators and stakeholders must pay attention to the concerns of D&I research, which are not usually considered early in the development of an intervention.

Simultaneously assessing effectiveness and implementation can potentially allow rapid translational gains in the uptake of clinical interventions67,68. Two study designs have been proposed to allow implementation to be assessed in pragmatic RCTs68. First, a pragmatic RCT that tests the effect of an intervention on clinical outcomes can simultaneously collect information on implementation. For example, pragmatic evaluation of new medication ‘X’ in patients with HF with preserved ejection fraction (HFpEF) might include assessment of providers and participants to identify reasons for non-prescription or non-adherence, respectively. These surveys could be conducted remotely via mobile applications or e-mails. Second, a trial might simultaneously study the effectiveness of an intervention as well as an implementation strategy. Such a trial would have co-primary aims: testing effectiveness (the effect of medication ‘X’ on clinical outcomes among patients with HFpEF in routine care) and testing an implementation strategy (EHR-based alerts to remind clinicians to prescribe medication ‘X’ to eligible patients). To achieve these aims in a multicentre RCT, patients with HFpEF could be randomly assigned to either medication ‘X’ or placebo to study effectiveness, with sites also being randomly assigned to the implementation strategy or usual care (cluster randomization). Further details of the design and conduct of effectiveness–implementation studies have been reviewed previously67,68,69.

Statistical analysis in pragmatic RCTs

The intention-to-treat (ITT) analysis is the preferred statistical technique to estimate treatment effects in RCTs. In the ITT method, patients who are non-adherent or crossover to other treatment groups are still analysed as part of the group to which they were initially randomly assigned. This approach reduces selection bias by preserving the benefits of randomization. The ITT technique works well in explanatory RCTs because there is usually little deviation from the assigned treatment due to close monitoring, the inclusion of selected cohorts who are likely to be compliant with therapy, and trial design features that are intended to improve adherence (such as run-in periods to confirm initial tolerability and adherence). However, in pragmatic RCTs, non-adherence to treatment and crossover between groups is substantially more likely than in explanatory trials. Indeed, high degrees of crossover, loss to follow-up or non-adherence to randomized therapy can limit the extent to which ITT analyses can be interpreted and drive results towards the null. An alternative approach in pragmatic RCTs is the per-protocol technique, which excludes patients who are lost to follow-up or who crossover between treatment groups from the analyses, censoring their data at the last point of protocol adherence. However, per-protocol analyses can introduce bias and disrupt the covariate balance between the randomized groups. Therefore, the choice of analysis in pragmatic RCTs can be challenging. Nonetheless, ITT and per-protocol analyses might best be viewed as complementary, with a primary analysis using the ITT method and sensitivity analysis using the per-protocol approach being a reasonable approach for many trials. Other, more advanced statistical techniques have also been proposed to address this problem such as the inverse probability of censoring weights method70, the iterative parameter estimation algorithm71, the two-stage method72 and the rank preserving structural failure time model73. However, these methods have rarely been used in pragmatic RCTs, probably due to residual confounding and suspicion of methods other than ITT72.

Analysing data from digital databases also brings unique challenges. ‘Lag times’ vary between health-care systems; for example, a death might be recorded in a national registry weeks after it has occurred and insurance claims can appear on the system several weeks after treatment, especially if the patient used out-of-network services. Another potential statistical dilemma can arise if a patient experiences a benefit from an intervention and stops interacting with the health-care system thereafter. However, in a pragmatic RCT, protocolized trial-specific visits might not occur and would lead to missing data for that patient. Therefore, appropriately defining outcomes and selecting the correct statistical approach in pragmatic RCTs is essential74.

Pragmatic cardiovascular RCTs

Completed studies

Selected, completed cardiovascular RCTs with a highly pragmatic design are summarized in Table 2. Analyses of 616 prominent cardiovascular RCTs that have been conducted over the past two decades revealed that pragmatism, as measured by the PRECIS-2 tool, increased modestly from 2000 (mean PRECIS-2 score 3.07) to 2015 (mean PRECIS-2 score 3.46)75. Although this trend towards pragmatism is to be commended, greater effort is needed to accelerate the adoption of pragmatic trial designs. A few selected, completed cardiovascular RCTs that had pragmatic designs warrant discussion in detail.

Table 2 Selected, completed cardiovascular clinical trials with pragmatic design elements
Full size table

The PACT-HF trial76,77 was designed to study the effectiveness of a patient-centred transitional care model in those hospitalized for HF. The key elements of the patient-centred approach were self-care education by nurses, a family physician follow-up scheduled <1 week after hospital discharge, a structured hospital discharge summary as well as heart function clinic care for patients at high risk. A stepped-wedge cluster randomization design was used to randomly assign hospitals to the intervention versus usual care. Several features of this trial were highly pragmatic, including the identification of eligible patients using EHRs, delivery of the intervention by existing hospital and home-care personnel, flexibility in adherence to the intervention, selection of outcomes by patients and decision-makers, and the methods of outcomes collection. Data were collected through administrative database linkages, with no trial-specific follow-up evaluations or monitoring additional to usual care. Patient-reported outcomes were collected remotely without face-to-face contact. No patients were lost to follow-up for clinical outcomes. The trial demonstrated no significant improvement in the composite of all-cause hospital readmission, emergency department visits or death at 3 months. Similarly, no significant difference was noted in the composite of all-cause readmission or emergency department visits at 30 days. These findings contrast with those from earlier explanatory RCTs, in which similar transitional care services reduced hospitalization and mortality78,79. Therefore, PACT-HF is a great example of how the treatment effects observed in explanatory RCTs can be attenuated in pragmatic trials and in real-world practice.

The randomized, double-blind CHIEF-HF trial80,81 sought to evaluate the efficacy of canagliflozin versus placebo on symptoms among patients with HF and is an example of a completely remote pragmatic trial. Patients were engaged through a trial website, following which e-consent was obtained. Patients were mailed the study investigational product, and primary outcome (quality of life) data were ascertained through a mobile application. Patients were also sent health trackers (Fitbit Versa 2) to allow monitoring of their physical activity, which was a secondary end point. Several aspects of the design of this trial are unique and pragmatic: the trial was entirely virtual and required no in-person visits, eligibility criteria were broad and outcomes were patient-centred. Such trials have the potential to accelerate the pace of recruitment, substantially reduce financial costs and increase participant satisfaction. This method also reduces the burden of time and effort associated with RCTs, which are primarily related to the requirement for in-person visits, evaluations, form-filling and generating documentation for site audits. The CHIEF-HF trial80,81 demonstrated that canagliflozin significantly improved quality of life at 12 weeks in patients with HF, regardless of HF type (HFrEF or HFpEF) or diabetes mellitus status.

Ongoing studies

Selected ongoing trials with pragmatic design elements are summarized in Table 3. The DAPA-MI study82 is among the first indication-seeking pragmatic trials and will evaluate the effect of dapagliflozin versus placebo on HF hospitalizations and cardiovascular death in patients with acute myocardial infarction in the preceding 7–10 days but without diagnosed diabetes. This trial is unusual among pragmatic RCTs in that it is industry-sponsored and is double-blinded and placebo-controlled to meet regulatory requirements. Pragmatic RCTs generally do not have industry sponsors, are open-label, and use usual care or alternative therapies as a control rather than placebo. The trial will incorporate pragmatism by enrolling patients from two high-quality national registries: SWEDEHEART in Sweden and MINAP in the UK. Automated capture of routine follow-up data from applicable registries will substantially reduce the burden on both patients and investigators. Other pragmatic elements include the use of mobile phone applications to query patients about clinical events and ‘CleverCap Lite’ bottle caps, which record the number of pills dispensed from the container by the patient and will allow real-time tracking of adherence to medication83.

Table 3 Selected, ongoing cardiovascular trials with pragmatic design elements
Full size table

Barriers and potential solutions

Pragmatic RCTs have various limitations that must be considered. First, these studies might not meet the requirements for regulatory approval of novel therapies because the data collected are not sufficiently detailed. Currently, the FDA requires efficacy data from blinded and placebo-controlled RCTs for the approval of new medications84. As a result, the focus of most completed and ongoing pragmatic RCTs is on comparing therapies that are clinically available rather than novel therapies that would require regulatory approval. Therefore, for the time being, RCTs for novel therapeutics should be designed to be as pragmatic as possible while still meeting regulatory requirements. The above-mentioned DAPA-MI trial82 is an excellent example of how pragmatism can be incorporated into an indication-seeking RCT.

Second, ethical and regulatory bodies have traditionally had some reservations about pragmatic RCTs. The original Good Clinical Practice (GCP) guidelines84,85, which were established in 1996, had discordance with certain aspects of pragmatic RCT design. For example, pragmatic trial designs aim to minimize trial burden by utilizing remote follow-ups and routinely collected data, whereas the 1996 GCP guidelines required that all patients enrolled in the trial should be followed up by physicians who were investigators or sub-investigators of the trial and emphasized careful site monitoring and data verification. However, in April 2021, the International Council of Harmonization released an early draft of the updated version of the GCP guidelines86, which was more accommodating of pragmatic design elements than the 1996 version had been (Table 4).

Table 4 Increasing pragmatism in the Good Clinical Practice Guidelines
Full size table

Third, pragmatic RCTs are likely to yield smaller estimates of treatment effect with less precision than explanatory RCTs. A heterogeneous patient population is often sought in pragmatic RCTs rather than one composed only of patients who are most likely to benefit from the intervention as in explanatory trials. Some would argue that the identification of a positive treatment effect in a selected cohort (explanatory RCT) might be preferable to missing the effect altogether if studied in a less-selective sample (pragmatic RCT). However, to counteract this reasoning, pragmatic RCTs operate with large sample sizes to provide sufficient power to detect significant clinical differences. Costs saved as a result of the decreased burden of data collection and organization allow most pragmatic RCTs to attain larger sample sizes than explanatory RCTs.

Fourth, although technology has a central role in increasing pragmatism in RCTs, several barriers exist to its digitization. For example, EHR systems need to be optimized with the development of interoperable and scalable platforms to support clinical research. The cost to build such systems can be high as seen with the EHR4CR project87, which involved 34 academic and pharmaceutical partners and cost >16 million Euros. To maximize research potential, nationally and internationally linked systems should be planned. EHR systems often use different coding formats for data; to successfully utilize data across EHRs for clinical trials, data should be harmonized under a common format. Moreover, in international trials, restrictions imposed by the General Data Protection Regulation as well as country-specific health-care privacy laws have to be considered before EHR data can be shared across borders. Although the initial costs of setting up interoperable EHR systems would be high, when established, embedding multiple RCTs into these systems and curating data for analyses would be extremely cost-effective.

A fifth limitation of pragmatic RCTs is that the quality of the data, particularly for outcome variables such as cause of death and hospitalization, has been called into question and poor-quality data are not convincing for regulatory bodies64. Pilot studies have demonstrated that routine ascertainment of outcomes might not be too different from adjudication by a clinical trial committee88,89; however, this approach has not been routinely used. Moreover, the accuracy of smartphone applications and wearables in tracking health data needs to be validated against gold standards before data from these devices are used to inform clinical decisions64. Regulatory bodies and guidelines have now started to recognize the potential of routinely collected digital data for conducting RCTs64. An extension of the Consolidated Standards of Reporting Trials statement was published in 2021 to highlight and standardize elements that should be reported when publishing RCTs conducted using cohorts and routinely collected data90.

Unanswered questions

As we move towards an era of increasing pragmatism in RCTs, several new questions arise about how designs will compare between pragmatic and explanatory trials (Box 1). The virtues and limitations of pragmatic RCTs will become clearer as they are conducted side-by-side with traditional explanatory RCTs. Two parallel, phase III RCTs are being conducted to study the effects of spironolactone in patients with HFpEF. The SPIRIT-HF study91 has a fairly explanatory design, components of which include double blinding, multiple trial-specific visits and manual data collection. The SPIRRIT trial92 will study the same question using a more pragmatic approach and is layered onto existing disease registries in Sweden and the USA to increase patient recruitment (the SPIRRIT trial is expected to enrol more than twice the number of patients than SPIRIT-HF). The SPIRRIT trial will be open label and will attempt to minimize patient and investigator burden by capturing follow-up and outcome data directly from registries whenever possible, with telephone calls and trial-specific visits available as alternatives. Comparison of the findings from SPIRIT-HF and SPIRRIT will yield valuable information about and explanations for the differences in outcomes between pragmatic and explanatory RCTs.

Conclusions

Explanatory RCTs provide important information about treatment effects in selected populations. These trials are designed to maximize estimates of treatment efficacy and safety, but the findings are often not easily generalized to the broad range of patients seen in clinical settings. Much is to be gained from conducting pragmatically designed RCTs, which can yield results that are more broadly generalizable to routine clinical practice and highlight barriers to the implementation of novel therapies, while simultaneously reducing trial burden and costs. A pragmatic design involves the recruitment of larger and more diverse patient populations, increased flexibility in trial organization and treatment delivery, reduced monitoring and data collection, and an increased emphasis on remote follow-up. The ability to enrol, treat and follow up patients remotely has become especially relevant since the start of the coronavirus disease 2019 (COVID-19) pandemic. Moreover, pragmatic trials are better suited than explanatory RCTs to testing the effects of implementation strategies and health policies. Leveraging technology and embedding pragmatic RCTs within routine care can reduce or eliminate the need for trial-specific follow-up and minimize the time burden on administrative personnel, thereby reducing costs. Although pragmatic trials attenuate estimates of treatment effects, these estimates are likely to be closer to the true effects and to provide clinicians and decision-makers with a realistic understanding of the population-level effectiveness and costs of an intervention.