Hostname: page-component-848d4c4894-m9kch Total loading time: 0 Render date: 2024-05-07T10:39:33.128Z Has data issue: false hasContentIssue false
  • English
  • Français

The quality of research exploring in-session measures of CBT competence: a systematic review

Published online by Cambridge University Press:  23 June 2021

Kathryn Rayson ,
Louise Waddington  and
Dougal Julian Hare
Kathryn Rayson
Affiliation:
South Wales Doctoral Programme in Clinical Psychology, 11th Floor, Tower Building, Park Place, CardiffCF10 3AT, UK
Louise Waddington*
Affiliation:
South Wales Doctoral Programme in Clinical Psychology, 11th Floor, Tower Building, Park Place, CardiffCF10 3AT, UK
Dougal Julian Hare
Affiliation:
South Wales Doctoral Programme in Clinical Psychology, 11th Floor, Tower Building, Park Place, CardiffCF10 3AT, UK
*
*Corresponding author. Email: cav_psychology.training@wales.nhs.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

Background:

Cognitive behavioural therapy (CBT) is in high demand due to its strong evidence base and cost effectiveness. To ensure CBT is delivered as intended in research, training and practice, fidelity assessment is needed. Fidelity is commonly measured by assessors rating treatment sessions, using CBT competence scales (CCSs).

Aims:

The current review assessed the quality of the literature examining the measurement properties of CCSs and makes recommendations for future research, training and practice.

Method:

Medline, PsychINFO, Scopus and Web of Science databases were systematically searched to identify relevant peer-reviewed, English language studies from 1980 onwards. Relevant studies were those that were primarily examining the measurement properties of CCSs used to assess adult 1:1 CBT treatment sessions. The quality of studies was assessed using a novel tool created for this study, following which a narrative synthesis is presented.

Results:

Ten studies met inclusion criteria, most of which were assessed as being ‘fair’ methodological quality, primarily due to small sample sizes. Construct validity and responsiveness definitions were applied inconsistently in the studies, leading to confusion over what was being measured.

Conclusions:

Although CCSs are widely used, we need to pay careful attention to the quality of research exploring their measurement properties. Consistent definitions of measurement properties, consensus about adequate sample sizes and improved reporting of individual properties are required to ensure the quality of future research.

Keywords

CBTclinical skillcognitive behavioural therapycompetencetraining
Type
Main
Information
Behavioural and Cognitive Psychotherapy , Volume 50 , Issue 1 , January 2022 , pp. 40 - 56
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of the British Association for Behavioural and Cognitive Psychotherapies

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

A copy of the Checklist for the Appraisal of Therapy Competence Scale Studies (CATCS) is in the supplementary material of the online version of this article. Further information regarding the development of the CATCS is available by contacting the author.

References

Alberts, G., & Edelstein, B. (1990). Therapist training: a critical review of skill training studies. Clinical Psychology Review, 10, 497–511. https://doi.org/10.1016/0272-7358(90)90094-Q CrossRefGoogle Scholar
Altman, D. G. (1991). Practical Statistics for Medical Research. London, UK: Chapman and Hall. http://doi.org/10.1002/sim.4780101015 Google Scholar
Barber, J. P., Liese, B. S., & Abrams, M. J. (2003). Development of the cognitive therapy adherence and competence scale. Psychotherapy Research, 13, 205–221. http://doi.org/10.1093/ptr/kpg019 CrossRefGoogle Scholar
Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive Therapy of Depression. New York, USA: Guilford Press.Google Scholar
Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446. http://doi.org/10.1017/S1352465801004040 CrossRefGoogle Scholar
Branson, A., Shafran, R., & Myles, P. (2015). Investigating the relationship between competence and patient outcome with CBT. Behaviour Research and Therapy, 68, 19–26. https://doi.org/10.1016/j.brat.2015.03.002 CrossRefGoogle ScholarPubMed
Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., … & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57, 225–238. https://doi.org/10.1016/S0376-8716(99)00049-6 CrossRefGoogle ScholarPubMed
Clark, D. M. (2011). Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. International Review of Psychiatry, 23, 318–327. https://doi.org/10.3109/09540261.2011.606803 CrossRefGoogle ScholarPubMed
Cohen, J. (1968). Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. http://doi.org/10.1037/h0026256 CrossRefGoogle ScholarPubMed
Crits-Christoph, P., Baranackie, K., Kurcias, J., Beck, A., Carroll, K., Perry, K., … & Gallagher, D. (1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, 81–91. https://doi.org/10.1080/10503309112331335511 CrossRefGoogle Scholar
Dittmann, C., Müller-Engelmann, M., Stangier, U., Priebe, K., Fydrich, T., Görg, N., … & Steil, R. (2017). Disorder- and treatment-specific therapeutic competence scales for posttraumatic stress disorder intervention: development and psychometric properties. Journal of Traumatic Stress, 30, 614–625. https://doi.org/10.1002/jts.22236 CrossRefGoogle ScholarPubMed
Dobson, K. S., & Kazantzis, N. (2003). The therapist in cognitive-behavioral therapy: Introduction to a special section. Psychotherapy Research, 13, 131–134. https://doi.org/10.1080/713869635 CrossRefGoogle Scholar
Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295–300. http://doi.org/10.1111/j.2044-8260.1985.tb00662 CrossRefGoogle ScholarPubMed
Fairburn, C. G., & Cooper, Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy, 49, 373–378. https://doi.org/10.1016/j.brat.2011.03.005 CrossRefGoogle ScholarPubMed
Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: durrent practice and recommendations. Social Psychological and Personality Science, 8, 370–378. https://doi.org/10.1177/1948550617693063 CrossRefGoogle Scholar
Gordon, P. K. (2006). A comparison of two versions of the Cognitive Therapy Scale. Behavioural and Cognitive Psychotherapy, 35, 343–353. https://doi.org/10.1017/S1352465806003390 CrossRefGoogle Scholar
Haddock, G., Devane, S., Bradshaw, T., McGovern, J., Tarrier, N., Kinderman, P., … & Harris, N. (2001). An investigation into the psychometric properties of the cognitive therapy scale for psychosis (CTS-Psy). Behavioural and Cognitive Psychotherapy, 29, 221–233. http://doi.org/10.1017/S1352465801002089 CrossRefGoogle Scholar
Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cognitive Therapy and Research, 36, 427–440. https://doi.org/10.1007/s10608-012-9476-1 CrossRefGoogle ScholarPubMed
Jacobson, N. S., & Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117. https://doi.org/10.1016/S0005-7967(99)00029-7 CrossRefGoogle ScholarPubMed
James, I. A., Blackburn, I. M., Milne, D. L., & Reichfelt, F. K. (2001). Moderators of trainee therapists’ competence in cognitive therapy. British Journal of Clinical Psychology, 40, 131–141. https://doi.org/10.1348/014466501163580 CrossRefGoogle ScholarPubMed
Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: review of the contemporary empirical evidence. Behaviour Change, 20, 1–12. https://doi.org/10.1375/bech.20.1.1.24845 CrossRefGoogle Scholar
Loades, M. E., & Armstrong, P. (2016). The challenge of training supervisors to use direct assessments of clinical competence in CBT consistently: a systematic review and exploratory training study. The Cognitive Behaviour Therapist, 9, e27. https://doi.org/10.1017/S1754470X15000288 CrossRefGoogle Scholar
McManus, F., Westbrook, D., Vazquez-Montes, M., Fennell, M., & Kennerley, H. (2010). An evaluation of the effectiveness of diploma-level training in cognitive behaviour therapy. Behaviour Research and Therapy, 48, 1123–1132. https://doi.org/10.1016/j.brat.2010.08.002 CrossRefGoogle ScholarPubMed
Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65, S63–67. Available at: http://winbev.pbworks.com/f/Assessment.pdf (accessed 15 December 2017).CrossRefGoogle ScholarPubMed
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G. and The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. PLOS Medicine, 6, e1000097. https://doi.org/10.1371/journal.pmed.1000097 CrossRefGoogle ScholarPubMed
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M. & de Vet, H. C. W. (2010a). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality of Life Research, 19, 539–549. https://doi.org/10.1007/s11136-010-9606-8 CrossRefGoogle Scholar
Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., Bouter, L. M. & de Vet, H. C. W. (2010b). The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Medical Research Methodology, 10, 22. https://doi.org/10.1186/1471-2288-10-22 CrossRefGoogle ScholarPubMed
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., … & de Vet, H. C. (2010c). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63, 737–745. https://doi.org/10.1016/j.jclinepi.2010.02.006 CrossRefGoogle ScholarPubMed
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., … & de Vet, H. C. (2012). COSMIN Checklist Manual. Amsterdam, The Netherlands: COSMIN. Available at: http://www.cosmin.nl/images/upload/files/COSMIN%20checklist%20manual%20v9.pdf (accessed 9 February 2018).Google Scholar
Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive-behavioural therapy. Clinical Psychology Review, 33, 484–499. https://doi.org/10.1016/j.cpr.2013.01.010 CrossRefGoogle ScholarPubMed
Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive-behavioural therapy: a qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology & Psychotherapy, 23, 246–259. https://doi.org/10.1002/cpp.1952 CrossRefGoogle ScholarPubMed
Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): an observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment, 29, 542–555. http://doi.org/10.1037/pas0000372 CrossRefGoogle ScholarPubMed
National Health Service Education for Scotland (2015). The Matrix: A Guide to Delivering Evidence-Based Psychological Therapies in Scotland. Edinburgh, UK: NHS Education for Scotland.Google Scholar
National Institute for Clinical Excellence (2004). Eating disorders. Clinical Guideline 9. London, UK: NICE.Google Scholar
National Institute for Health and Clinical Excellence (2007). Drug misuse: psychosocial intervention. Clinical Guideline 51. London, UK: NICE.Google Scholar
National Institute for Health and Clinical Excellence (2011). Common mental health problems: identification and pathways to care. Clinical Guideline 123. London, UK: NICE.Google Scholar
National Institute for Health and Clinical Excellence (2014a). Psychosis and schizophrenia in adults: treatment and management. Clinical Guideline 178. London, UK: NICE.Google Scholar
National Institute for Health and Clinical Excellence (2014b). Bipolar disorder: the management and assessment of bipolar disorder in adults, children and young people in primary and secondary care. Clinical Guideline 185. London, UK: NICE.Google Scholar
National Psychological Therapies Management Committee (2017). Matrics Cymru: Guidance for Delivering Evidence-Based Psychological Therapy in Wales. NPTMC, Public Health Wales.Google Scholar
Nunnally, J., & Bernstein, I. H. (1994). Psychometric Theory (3rd edn). New York, USA: McGraw Hill.Google Scholar
Polit, D. F. (2015). Assessing measurement in health: beyond reliability and validity. International Journal of Nursing Studies, 52, 1746–1753. https://doi.org/10.1016/j.ijnurstu.2015.07.002 CrossRefGoogle ScholarPubMed
Rosenkoetter, U., & Tate, R. L. (2018). Assessing features of psychometric assessment instruments: a comparison of the COSMIN checklist with other critical appraisal tools. Brain Impairment, 19, 103–118. https://doi.org/10.1017/BrImp.2017.29 CrossRefGoogle Scholar
Roth, A. D., & Pilling, S. (2007). The competences required to deliver effective cognitive and behavioural therapy for people with depression and with anxiety disorders. Available at: https://www.ucl.ac.uk/pals/sites/pals/files/migrated-files/CBT_competences_-_map_Sept_2015.pdf (accessed 23 January 2018).Google Scholar
Roth, A. D., & Pilling, S. (2008). Using an evidence-based methodology to identify the competences required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36, 129–147. https://doi.org/10.1017/S1352465808004141 CrossRefGoogle Scholar
Rozek, D. C., Serrano, J. L., Marriott, B. R., Scott, K. S., Hickman, L. B., Brothers, B. M., … & Simons, A. D. (2018). Cognitive behavioural therapy competency: pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy, 46, 244–250. https://doi.org/10.1017/S1352465817000662 CrossRefGoogle ScholarPubMed
Shafran, R., Clark, D. M., Fairburn, C. G., Arntz, A., Barlow, D. H., Ehlers, A., … & Salkovskis, P. M. (2009). Mind the gap: improving the dissemination of CBT. Behaviour Research and Therapy, 47, 902–909. https://doi.org/10.1016/j.brat.2009.07.003.CrossRefGoogle ScholarPubMed
Streiner, D. L., Norman, G. R. & Cairney, J. (2015). Health Measurement Scales: A Practical Guide to their Development and Use (5th edn). USA: Oxford University Press.CrossRefGoogle Scholar
Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., … & de Vet, H. C. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60, 34–42. https://doi.org/10.1016/j.jclinepi.2006.03.012 CrossRefGoogle ScholarPubMed
Terwee, C. B., Dekker, F. W., Wiersinga, W. M., Prummel, M. F., & Bossuyt, P. M. M. (2003). On assessing responsiveness of health-related quality of life instruments: guidelines for instrument evaluation. Quality of Life Research, 12, 349–362. https://doi.org/10.2307/4038855 CrossRefGoogle ScholarPubMed
Terwee, C. B., de Vet, H. C. W., Prinsen, C. A. C, & Mokkink, L. B. (2011). Protocol for systematic reviews of measurement properties. COSMIN: Knowledgecenter Measurement Instruments. Available at: http://www.cosmin.nl/images/upload/files/Protocol%20klinimetrische%20review%20version%20nov%202011(1).pdf (accessed 9 February 2018).Google Scholar
Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385. https://doi.org/10.1037/0022-006x.54.3.381 CrossRefGoogle ScholarPubMed
von Consbruch, K., Clark, D. M., & Stangier, U. (2012). Assessing therapeutic competence in cognitive therapy for social phobia: psychometric properties of the cognitive therapy competence scale for social phobia (CTCS-SP). Behavioural and Cognitive Psychotherapy, 40, 149–161. https://doi.org/10.1017/S1352465811000622 CrossRefGoogle Scholar
Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620. http://doi.org/10.1037/0022-006X.61.4.620 CrossRefGoogle ScholarPubMed
Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: influence of clinical experience. Journal of Nervous and Mental Disease, 199, 276–279. https://doi.org/10.1097/NMD.0b013e3182124617 CrossRefGoogle ScholarPubMed
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156. http://doi.org/10.1037/0022-006X.49.2.156 CrossRefGoogle ScholarPubMed
Young, J., & Beck, A. T. (1980). The Development of the Cognitive Therapy Scale . Unpublished manuscript. Philadelphia, USA: Center for Cognitive Therapy.Google Scholar
Supplementary material: PDF

Rayson et al. supplementary material

Rayson et al. supplementary material 1

Download Rayson et al. supplementary material(PDF)
PDF 292.6 KB
Supplementary material: File

Rayson et al. supplementary material

Rayson et al. supplementary material 2

Download Rayson et al. supplementary material(File)
File 22.7 KB
Submit a response

Comments

No Comments have been published for this article.