Skip to main content

Advertisement

SpringerLink
Log in

Developing Treatment Integrity Measures for Teacher-Delivered Interventions: Progress, Recommendations and Future Directions

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

While the measurement of treatment integrity is important to determine how much, and how well, interventions are delivered in schools, the science of treatment integrity is not well developed in education research. The purpose of this paper is to describe a program of research that has developed treatment integrity measures over the past 10 years to assess teacher delivery of an indicated program targeting reductions in problem behavior in early childhood and elementary school classrooms. Specifically, this paper will highlight the importance of active use of conceptual models to guide treatment integrity measure development, multidimensional assessment of treatment integrity and training procedures for observers, using several studies to illustrate the evolution and refinement of our measurement approach. Recommendations for researchers developing and evaluating interventions in schools are provided, as are recommendations to help the field move toward a more rigorous science of treatment integrity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  • Barber, J. P., Mercer, D., & Krakauer, I. (1996). Development of an adherence/competence rating scale for individual drug counseling. Drug and Alcohol Dependence, 43, 125–132.

  • Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12(1), 23–33. https://doi.org/10.1007/s11121-010-0186-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Botvin, G. J., & Griffin, K. W. (2004). Life skills training: Empirical findings and future directions. Journal of Primary Prevention, 25(4), 211–232.

    Article  Google Scholar 

  • Bronfenbrenner, U. (1979). The ecology of human development. Cambridge: Harvard University Press.

    Google Scholar 

  • Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. https://doi.org/10.1016/s0376-8716(99)00049-6.

  • Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566. https://doi.org/10.1037/a0014565.

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284. https://doi.org/10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  • Conroy, M. A., Sutherland, K. S., Algina, J., Ladwig, C., Werch, B., Martinez, J., et al. (2019). Outcomes of the BEST in CLASS intervention on teachers’ use of effective practices, self-efficacy, and classroom quality. School Psychology Review, 48, 31–45. https://doi.org/10.17105/SPR-2018-0003.V48-1.

    Article  Google Scholar 

  • Conroy, M. A., Sutherland, K. S., Algina, J., Werch, B., & Ladwig, C. (2018). Prevention and treatment of externalizing problem behaviors in young children: Clinical implications from a randomized controlled trial of BEST in CLASS. AERA Open, 4(1), 1–16. https://doi.org/10.1177/2332858417750376.

    Article  Google Scholar 

  • Conroy, M. A., Sutherland, K. S., Granger, K. L., Marcolouides, K. M., Feil, E., Huang, K., & Montession, A. (2020). Preliminary study of the effects of BEST in CLASS-Web on young children’s social-emotional and behavioral outcomes. Manuscript submitted for publication.

  • Conroy, M. A., Sutherland, K. S., Wilson, R. E., Martinez, J., Whalon, K. J., & Algina, J. (2015). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23, 144–155. https://doi.org/10.1177/1063426614532949.

    Article  Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework Advances in School Mental. Health Promotion, 1(3), 6–28. https://doi.org/10.1080/1754730X.2008.9715730.

    Article  Google Scholar 

  • Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.” Early Childhood Research Quarterly, 25(3), 348–357. https://doi.org/10.1016/j.ecresq.2010.03.003.

    Article  Google Scholar 

  • Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. https://doi.org/10.1007/s11121-015-0606-3.

    Article  PubMed  Google Scholar 

  • Evans, M., Piasecki, J., Kriss, M., & Hollon, S. (1984). Rater’s manual for the collaborative study psychotherapy rating scale- form 6 (CSPRS-6). University of Minnesota and St. Paul-Tamsey Medical Center. Unpublished manuscript.

  • Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665–679. https://doi.org/10.1007/s10802-005-7646-2.

    Article  PubMed  Google Scholar 

  • Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2014). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2.

    Article  Google Scholar 

  • Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy Theory, Research, Practice, Training, 33(2), 332. https://doi.org/10.1037/0033-3204.33.2.332.

    Article  Google Scholar 

  • Hollon, S., Evans, M., Auerbach, A., DeRubeus, R., Elkin, I., Lowerry, A., et al. (1988). Development of a system for rating therapies for depression: Differentiating cognitive therapy, interpersonal psychotherapy, and clinical management pharmacotherapy. Vanderbilt University, Nashville, TN. Unpublished manuscript.

  • Greenberg, L. S. (1986). Change process research. Journal of Consulting and Clinical Psychology, 54, 4–9.

    Article  Google Scholar 

  • Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., Reiner, R. H., & Liddle, H. A. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35(2), 137–147.

    Article  Google Scholar 

  • McLeod, B. D., Kunemund, R., Nemer, S. L., & Lyon, A. R. (2020). Leveraging Implementation Science and Practice to Support the Delivery of Evidence-Based Practices in Services for Youth with Emotional and Behavioral Disorders. In T. Farmer, M. Conroy, E. Farmer, & K. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders: Interdisciplinary Developmental Perspectives on Children and Youth (pp. 417–432). New York: Routledge/Taylor & Francis.

    Chapter  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020.

    Article  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., & Weisz, J. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38(4), 541.

    Google Scholar 

  • McLeod, B. D., Sutherland, K. S., Martinez, R. G., Conroy, M. A., Snyder, P. A., & Southam-Gerow, M. A. (2017). Identifying common practice elements to improve social, emotional, and behavioral outcomes of young children in early childhood classrooms. Prevention Science, 18(2), 204–213. https://doi.org/10.1007/s11121-016-0703-y.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436–443. https://doi.org/10.1080/15374411003691750.

    Article  Google Scholar 

  • Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12(4), 365–383. https://doi.org/10.1093/clipsy.bpi045.

    Article  Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  PubMed  Google Scholar 

  • Sanetti, L. M. H., Charbonneau, S., Knight, A., Cochrane, W. S., Kulcyk, M. C. M., & Kraus, K. E. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57, 901–922. https://doi.org/10.1002/pits.22364.

    Article  Google Scholar 

  • Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999–2009. Journal of Positive Behaviour Interventions, 14, 29–46. https://doi.org/10.1177/1098300711405853.

    Article  Google Scholar 

  • Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment fidelity of interventions with children in the school psychology literature from 1995–2008. School Psychology Review, 40, 72–84.

    Article  Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95–114. https://doi.org/10.1037/h0100835.

    Article  Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.

  • Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s… fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18(2), 142–147. https://doi.org/10.1111/j.1468-2850.2011.01245.x.

    Article  Google Scholar 

  • Schoenwald, S. K., Mchugh, R. K., & Barlow, D. H. (2012). The science of dissemination and implementation. In R. K. McHugh & D. H. Barlow (Eds.), Dissemination and implementation of evidence-based psychological interventions (pp. 16–42). Oxford: Oxford University Press.

    Google Scholar 

  • Sutherland, K. S., Conroy, M. A., Abrams, L., Vo, A., & Ogston, P. (2013a). An initial evaluation of the Teacher-Child Interaction Direct Observation System: Measuring teacher-child interaction behaviors in classroom settings. Assessment for Effective Intervention, 39, 12–23. https://doi.org/10.1177/1534508412463814.

    Article  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2018a). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Early Childhood Research Quarterly, 42, 31–43. https://doi.org/10.1016/j.ecresq.2017.08.001.

    Article  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., & Granger, K. L. (2020). BEST in CLASS A Tier-2 program for children with and at risk for emotional/behavioral disorders. In T. W. Farmer, M. A. Conroy, E. M. Z. Farmer, & K. S. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders (pp. 214–226). Abingdon: Routledge.

    Chapter  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Kunemund, R. L. (2018b). Factors associated with teacher delivery of a classroom-based Tier 2 prevention program. Prevention Science, 19(2), 186–196. https://doi.org/10.1007/s11121-017-0832-y.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Wu, E. (2018c). Teacher competence of delivery of BEST in CLASS as a mediator of treatment effects. School Mental Health, 10(3), 214–225. https://doi.org/10.1007/s12310-017-9224-5.

    Article  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., McLeod, B. D., Kunemund, R., & McKnight, K. (2019). Common practice elements for improving social, emotional, and behavioral outcomes of young elementary school students. Journal of Emotional and Behavioral Disorders, 27(2), 76–85. https://doi.org/10.1177/1063426618784009.

    Article  Google Scholar 

  • Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259. https://doi.org/10.1177/1063426613497258.

    Article  Google Scholar 

  • Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013b). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. https://doi.org/10.1177/1053815113515025.

    Article  Google Scholar 

  • Vroom, E. B., Massey, O. T., Yampolskaya, S., & Levin, B. L. (2020). The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program. School Mental Health, 12(1), 113–123. https://doi.org/10.1007/s12310-019-09333-1.

    Article  Google Scholar 

  • Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of consulting and clinical psychology, 61(4), 620. https://doi.org/10.1037/0022-006X.61.4.620.

    Article  PubMed  Google Scholar 

  • Williford, A. P., Wolcott, C. S., Whittaker, J. V., & Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of Banking Time with preschoolers who display disruptive behaviors. Prevention Science, 16(8), 1054–1063. https://doi.org/10.1007/s11121-015-0544-0.

    Article  PubMed  Google Scholar 

  • Yoder, P., & Symons, F. (2010). Observational measurement of behavior. Berlin: Springer Publishing Company.

    Google Scholar 

Download references

Funding

This research was supported by Grants from the U.S. Department of Education’s Institute for Education Sciences (R324A11017, R305A140487, R305A150246, R324A160158 and R305A180182). The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U.S. Department of Education.

Author information

Authors and Affiliations

  1. Department of Counseling and Special Education, Virginia Commonwealth University, 1015 W. Main St, PO Box 842020, Richmond, VA, 23284, USA

    Kevin S. Sutherland & Nicholas Mccormick

  2. Department of Psychology, Virginia Commonwealth University, 806 W. Franklin St, PO Box 842018, Richmond, VA, 23284, USA

    Bryce D. McLeod

  3. Department of Special Education, School Psychology and Early Childood Studies, University of Florida, 1345 Norman Hall, PO Box 117050, Gainesville, FL, 32611, USA

    Maureen A. Conroy

Authors
  1. Kevin S. Sutherland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bryce D. McLeod
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maureen A. Conroy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicholas Mccormick
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kevin S. Sutherland.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All study procedures involving human participants were in accordance with the ethical standards of the researchers’ institutional review boards and with the 1964 Helsinki Declaration and its later amendments of comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sutherland, K.S., McLeod, B.D., Conroy, M.A. et al. Developing Treatment Integrity Measures for Teacher-Delivered Interventions: Progress, Recommendations and Future Directions. School Mental Health 14, 7–19 (2022). https://doi.org/10.1007/s12310-021-09423-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-021-09423-z

Search

Navigation