Background

There is no doubt that the large-scale implementation of evidence-based psychological treatments (EBPTs) for mental illness could have a tremendous positive impact. However, barriers to implementation have been identified. In particular, the context for implementation often differs from the context in which the treatment was developed and initially tested. This can cause a lack of ‘fit’ between the context and the EBPT.1–5 Horner et al6 defined fit ‘as the match between the strategies, procedures or elements of an intervention, and the values, needs, skills and resources available in a setting’ (p 1). For EBPTs, poor fit is a common problem because the context for implementation typically differs from the original treatment development and initial testing contexts in a broad range of domains including client age, race, ethnicity and culture. Hence, adapting EBPTs to maximise the fit between the context and the intervention is a centrepiece of successful scaling and sustainability.7

Treatment adaptation is defined as the modification of programme content, format, process or logistics to accommodate the needs of a new group of consumers, a new group of providers and/or a new context.2 4 The treatment adaptation process can be ‘thoughtful and deliberate’ (p 1),8 which we will refer to as ‘planned adaptation’. Alternatively, ad hoc adaptations are the changes made by treatment providers to the delivery, structure or content of a treatment to address perceived patient-specific needs.2 9–11 On the one hand, there have been calls to embrace ad hoc adaptations as data to drive ongoing learning about optimal intervention delivery over time.10 12 Moreover, systematic study of ad hoc adaptations has been encouraged via an online ‘adaptome’ that would capture and classify adaptations.1 On the other hand, experts have highlighted ad hoc modifications as problematic because they are common, they reduce fidelity and they can contribute to poorer outcome.1 2 11 13–17 Most recently, this tension is resolving into a shift from a conversation about ‘fidelity versus adaptation (two opposing concepts) to balancing fidelity and adaptation (two competing concepts)’ (p 2, italics added).18

Fortunately, there have been great strides forward in characterising the nature, origin, timing and impact of treatment adaptations. Scholars have scientifically derived a ‘Framework for Reporting Adaptations and Modifications-Enhanced’ or ‘FRAME’ to characterise adaptations (eg, 8 19 20). Stirman, Gamarra, Bartlett, Calloway and Gutner21 used this coding system in a systematic review involving 108 primary studies and 3 meta-analyses of EBPTs. A meta-analysis could not be conducted due to the dearth of studies that compared an adapted version with the original EBPT, at least in part due to the large samples needed to appropriately power such research.21 Nonetheless, based on the evidence available, the authors were able to conclude that there is currently little evidence pointing to adaptations being detrimental. Meanwhile, a systematic review of 42 articles investigated the use of adaptation frameworks. The key finding was that adaptation frameworks were surprisingly infrequently employed.22 Also, in a scoping review by the same group of scholars, 13 adaptation frameworks were identified and used to distill 11 adaptation steps.23 Relatedly, Movsisyan et al24 conducted a systematic review of guidance on how to adapt complex population health interventions to new contexts (38 publications). These researchers distilled 11 common steps of guidance as to how to optimally conduct a treatment adaptation. Recently, the Model for Adaptation Design and Impact (MADI) has been published.18 MADI builds on prior frameworks and offers an approach to specify how types of adaptations impact outcomes and under which circumstances. MADI is geared toward providers and is devised to be used prospectively (when planning an adaptation) and retrospectively (when an adaptation has been completed). The main elements of each of these approaches are listed in table 1.

Following the treatment development traditions described by Clark,25 26 Salkovskis27 and Onken et al,28 the use of an empirical process involving scientific data should be the cornerstone of all treatment development and adaptation efforts. Also, the National Institutes of Mental Health emphasise that adaptations ‘should only be undertaken if there is an empirical rationale for the adaptation and for the corresponding mechanism by which the adapted intervention or augmentation is expected to substantially enhance outcomes’ (RFA-MH-16–410). Importantly, Stirman et al21 point out that it is not efficient nor feasible to expect appropriately powered randomised controlled trials (RCTs) to test adaptations to every intervention, context and adaptation type (including combinations of adaptation types).

Objective

The objective of the systematic review reported herein is to focus on the methods used to derive the treatment adaptation and to determine the extent to which the field is endeavouring to be data driven in as many steps as possible from the beginning to the end of the treatment adaptation process. Of note, while an empirical process is included within the adaptation processes captured in table 1 (eg, piloting, evaluation), there is room to more rigorously infuse scientific data as a basis for the treatment adaptation process so as to ensure adaptations optimise treatment outcomes.

Taken together, the fresh lens we seek to offer is to examine the methods used to conduct the treatment adaptation. In particular, we sought to establish the extent to which empirical methods and scientific data are used. The first aim was to evaluate the methods used to make planned adaptations. This will shed light on the extent to which the adaptation process is data-informed. We also examined the reasons for the treatment adaptation, who made the treatment adaptation and the nature of the treatment adaptation. The second aim was to examine ad hoc adaptations and document the following: the basis for the ad hoc adaptation, the frequency of ad hoc modifications, the measures used to identify the ad hoc adaptation and the impact on treatment outcome. A meta-analysis was not conducted given the dearth of studies designed to specifically illuminate the effect of the treatment adaptation; namely, a comparison of the adapted version with the standard version of an EBPT.21

Study selection and analysis

We followed the reporting guideline, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)29 and registered the protocol with PROSPERO (CRD42018106214) on 7th August 2018.

Screening

See figure 1 for the PRISMA flow diagram. All authors served as trained reviewers who independently screened within Covidence. The first round of screening focused on titles and abstracts only and yielded 829 publications. Reviewers were then instructed to re-review these using the full texts. This yielded 220 publications. These moved to the data extraction phase. The first author resolved disagreements between the reviewers. For the meta-analyses and systematic reviews, we reviewed the lists of included studies and compared them with our search results. For those not identified by the initial review, two reviewers screened the titles/abstracts. Four studies met eligibility criteria and were included in the data extraction phase.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Systematic review flow diagram (modified).

Findings

Our search terms provided 20 194 citations that yielded 152 distinct articles after the two levels of screening (see figure 1 for the PRISMA flow diagram). The main reasons for exclusion were a lack of description of the adaptation process, not being focused on a mental illness, and not focusing on a cognitive, behavioural and related EBPT.

Aim 1: planned adaptation

As depicted in figure 2, the top three methods of adaptation used were literature review (n=86, 57.7%), clinical intuition (n=70, 47.0%) and theory (n=58, 38.9%). The use of stakeholder interviews ranked fourth (n=32, 21.5%) and the use of data was ranked last (n=18, 12.1%). During data extraction, we noticed that the quality of reporting for the stakeholder interviews varied across studies. Hence, two coders reviewed the descriptions provided for the stakeholder interviews and coded them using the following system which was developed for this study. A score of ‘1’ was assigned when a full description of the methods and results was provided. A score of ‘2’ was assigned when there were some details provided but details were missing in one or more of the key domains (eg, number of participants, questions asked, results). A score of ‘3’ was assigned when there was mere mention of a stakeholder interview but no description of the methods or results. Disagreements were resolved by a third coder. Of the 31 papers which mentioned using a stakeholder interview as a method for conducting the treatment adaptation, 17 received a score of 1 (54.8%), 8 received a score of 2 (25.8%), and 6 received a score of 3 (19.4%).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Frequency of methods used to adapt evidence-based psychological treatments. Publications often reported more than one method.

Many publications used more than one method to conduct the treatment adaptation (n=88, 59.1%). The average number of methods used was 2.5 methods (range 1–5). Common combinations of methods used are depicted in figure 3. Theory and literature review or literature review or clinical intuition were the most common combinations.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Combinations of methods used to adapt evidence-based psychological treatments. Publications often reported more than one method.

As evident in table 2, the most common reasons for conducting the adaptation were to improve the fit with recipients (87.2%), improve effectiveness/outcomes (n=79.9%) and increase reach or engagement (53.7%). The most common people to make the adaptation were the researcher/s (92.6%) followed by the treatment/intervention teams (14.1%). The most common types of adaptations were tailoring/tweaking/refining (81.2%), adding elements (65.1%) and changing packaging or materials (57.0%).

Conclusions and implications

The first aim was to evaluate the methods used to guide planned adaptations. Consistent with the systematic review conducted by Stirman et al,21 the majority (98%) of the articles we reviewed reported on planned adaptations. It is laudable that many articles (n=88; 59.1%) used a combination of methods to conduct the adaptation. Multimethod approaches yield a more complete basis, and a fuller range of perspectives, for conducting adaptations.36 Future research could usefully invest in determining which of the combination approaches, depicted in figure 3, yield better outcomes relative to others.

The ‘data’ code was the least commonly used method for planned treatment adaptations (n=18; 12.1% of articles). This code was operationalised as pilot testing of the adapted treatment in order to inform the adaptation process or the use of any kind of research to empirically guide the adaptation. Hence, the use of an empirically grounded adaptation process appears to be rare. The approach articulated by Clark25 26 and Salkovskis27 emphasises the extensive use of data from experimental studies, surveys and other types of research to (a) develop, evaluate and refine a theory of the maintenance of the problem, and (b) clarify specific procedures within multicomponent EBPTs. It seems fair to say that returning to such a rigorous framework would be a wise path forward for the treatment adaptation enterprise. Of course, it is not realistic for every minor tweak to an EBPT to be empirically verified.21 27 Hence, it would be beneficial to develop consensus guidelines as to the types of adaptations that should be empirically derived and verified.

The ‘stakeholder interview’ code captured another important form of data. This was the second least commonly used method (n=32; 21.5% of articles). The much-needed pivot toward patient-centred outcomes has bought stakeholder interviews to the fore.37 However, a proportion of these studies (45.2%) did not provide sufficient detail on the methods and results of the stakeholder interview. Thus, the quality of the data produced is not known and there is a need to improve the reporting of stakeholder interviews.

Our consensus coding process indicates that the most common method used to adapt treatments was literature review (n=86, 57.7%). Done well, a literature review summarises the relevant data used to inform each step of the adaptation. The use of theory to guide the adaptation process was the third most common method (n=58, 38.9%). A valid theory has been derived and tested empirically and is a cornerstone for understanding the logic of the treatment and identifying the core components of the EBPT that should not be altered.23 Theory also serves as a ‘road map’ for deriving and adapting treatments.26 The conclusion we draw is that our field would greatly benefit from precisely articulating the constituents of a high-quality literature review and the skilful application of a validated theory. This would guide those who are conducting the adaptations as well as consumers who need to make judgements about the quality of the adaptation process.

Consistent with prior research showing that clinicians rely heavily on clinical intuition,38 we found that clinical intuition was the second most commonly used source for adapting treatments (n=70, 47%). On the one hand, clinical intuition can be a source of new hypotheses to be tested empirically25 27 and is effectively combined with an empirical approach.39 On the other hand, clinical intuition is prone to cognitive biases40 and feedback as to the veracity of intuition is unlikely to be available.39

We found only three articles to inform our second aim. These document that ad hoc modifications are common practice and that clinical intuition was the basis for the ad hoc adaptations made. Together, this indicates that these ad hoc adaptations are infrequently studied, rely on clinician intuition and should be prioritised in future research. Going forward, we need to work out when and how to go beyond clinical intuition and infuse an empirical process into the inevitable ad hoc adaptations that providers make to EBPTs, such as session-by-session progress monitoring.41 Also, we need to determine how to use ad hoc adaptations as a basis for a next generation of treatment adaptations, such as an online ‘adaptome’ that would capture and classify adaptations.1

There are several limitations. First, we focused on adults so we cannot generalise to younger people. Second, we only included published research studies because the ‘file drawer’ of unpublished studies in this field is likely to be very substantial. Including these would be impractical and unfeasible. Third, a meta-analysis could not be conducted. There is a need for studies comparing adapted and standard EBPTs. Fourth, CBT was the most common treatment in this review. Future research is needed to determine if the findings generalise to other treatment types. Finally, typical quality assessment tools were not a good match for the treatment adaptation phase of a research programme. There is a need to develop risk of bias assessment tools appropriate for research focused on the process of treatment adaptations.

In conclusion, this systematic review highlights several pressing needs. First, there is a need to educate providers and those seeking to adapt treatments to use empirical methods for the treatment adaptation process. A broad range of empirical approaches are relevant, including stakeholder interviews, case series, surveys and experiments. Second, there is a need to educate peer reviewers to ensure information about the methods used to derive treatment adaptations are included in reports. Third, there is a need to improve the reporting of stakeholder interviews that were conducted to derive a treatment adaptation. Fourth, there is a need to improve the quality of literature reviews conducted to guide a treatment adaptation. Fifth, the method used to conduct the adaptation is proposed as an additional domain for inclusion in FRAME and MADI. Finally, there is a need to establish regulatory requirements for EBPTs, and the associated adaptations, to ensure optimal implementation.42