Skip to main content
SpringerLink
Log in

Teacher Ratings of Acceptability of a Daily Report Card Intervention Prior to and During Implementation: Relations to Implementation Integrity and Student Outcomes

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

The goals of the study were to (a) examine teacher-reported acceptability of a daily report card (DRC) intervention for a student in their classroom prior to and during implementation; (b) examine factors that predict acceptability; and (c) explore the relations between teacher-reported acceptability, student and teacher characteristics prior to implementation, implementation integrity (treatment dose, adherence, and teacher competence), and student outcomes. Participants were 39 elementary school teachers and 39 students with or at risk of attention-deficit hyperactivity disorder (ADHD). Teachers were asked to implement the DRC for up to 16 weeks with consultation support provided by research team staff every other week. Teachers completed acceptability ratings about the DRC prior to and after two months of implementation. This multi-method assessment using correlation and regression analyses revealed that although acceptability ratings prior to implementation were related to teacher knowledge of ADHD, they were was not related to acceptability ratings during implementation, integrity, or student outcomes. Student’s initial positive response to the intervention (i.e., the magnitude of improvement in DRC target behaviors) was associated with higher acceptability ratings during the intervention. Greater increases in acceptability over time were associated with greater DRC dose (i.e., teacher compliance to procedures and longer DRC duration). Greater duration of implementation and responding appropriately to rule violations were associated with greater student achievement of DRC goals. Implications for interpreting acceptability ratings and for understanding factors related to implementation and outcomes are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Andersen, M., & Daly, E. J. (2013). An experimental examination of the impact of choice of treatment components on treatment integrity. Journal of Educational and Psychological Consultation, 23, 231–263.

    Article  Google Scholar 

  • Briesch, A. M., Briesch, J. M., & Chafouleas, S. M. (2015). Investigating the usability of classroom management strategies among elementary schoolteachers. Journal of Positive Behavior Interventions, 17(1), 5–14.

    Article  Google Scholar 

  • Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., & McCoach, D. B. (2009). Moving beyond assessment of treatment acceptability: An examination of the factor structure of the Usage Rating Profile-Intervention (URP-I). School Psychology Quarterly, 24(1), 36.

    Article  Google Scholar 

  • Classi, P., Milton, D., Ward, S., Sarsour, K., & Johnston, J. (2012). Social and emotional difficulties in children with ADHD and the impact on school attendance and healthcare utilization. Child and Adolescent Psychiatry and Mental Health, 6, 1–8.

    Article  Google Scholar 

  • Conroy, M. A., Sutherland, K. S., Algina, J. J., Wilson, R. E., Martinez, J. R., & Whalon, K. J. (2015). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23, 144–155.

    Article  Google Scholar 

  • Cox, J. R., Martinez, R. G., & Southam-Gerow, M. A. (2019). Treatment integrity in psychotherapy research and implications for the delivery of quality mental health services. Journal of Consulting and Clinical Psychology, 87, 221–233.

    Article  PubMed  Google Scholar 

  • Cunningham, C. E., Barwick, M., Short, K., Chen, Y., Rimas, H., Ratcliffe, J., et al. (2014). Modeling the mental health practice change preferences of educators: A discrete choice conjoint experiment. School Mental Health, 6, 1–14.

    Article  PubMed  Google Scholar 

  • Dart, E. H., Cook, C. R., Collins, T. A., Gresham, F. M., & Chenier, J. S. (2012). Test driving interventions to increase treatment integrity and student outcomes. School Psychology Review, 41(4), 467.

    Article  Google Scholar 

  • Domitrovich, C. E., Pas, E. T., Bradshaw, C. P., Becker, K. D., Keperling, J. P., Embry, D. D., et al. (2015). Individual and school organizational factors that influence implementation of the PAX good behavior game intervention. Prevention Science, 16(8), 1064–1074.

    Article  PubMed  Google Scholar 

  • Egan, T. E., Wymbs, F. A., Owens, J. S., Evans, S. W., Hustus, C., & Allan, D. M. (2019). Elementary school teachers’ preferences for school-based interventions for students with emotional and behavioral problems. Psychology in the Schools, 56, 1633–1653.

    Article  Google Scholar 

  • Elliott, S. N. (1988). Acceptability of behavioral treatments: Review of variables that influence treatment selection. Professional Psychology: Research and Practice, 19, 68–80.

    Article  Google Scholar 

  • Elliott, S. N., Witt, J. C., Galvin, R., & Peterson, M. (1984). Acceptability of positive and reductive interventions: Factors that influence teachers’ decisions. Journal of School Psychology, 22, 353–360.

    Article  Google Scholar 

  • Evans, S. W., Owens, J. S., Wymbs, B., & Ray, R. (2018). Evidence-based psychosocial treatments for children and adolescents with attention-deficit/hyperactivity disorder. Journal of Clinical Child and Adolescent Psychology, 47, 157–198.

    Article  PubMed  Google Scholar 

  • Evans, S. W., Pelham, W. E., Smith, B. H., Bukstein, O., Gnagy, E. M., Greiner, A. R., et al. (2001). Dose–response effects of methylphenidate on ecologically valid measures of academic performance and classroom behavior in adolescents with ADHD. Experimental and Clinical Psychopharmacology, 9, 163–175.

    Article  PubMed  Google Scholar 

  • Fabiano, G. A., Pelham, W. E., Majumdar, A., Evans, S. W., Manos, M. J., Caserta, D., et al. (2013a). Elementary and middle school teacher perceptions of attention-deficit/hyperactivity disorder incidence. Child & Youth Care Forum, 42, 87–99.

    Article  Google Scholar 

  • Fabiano, G. A., Pelham, W. E., Jr., Waschbusch, D. A., Gnagy, E. M., Lahey, B. B., Chronis, A. M., et al. (2006). A practical measure of impairment: Psychometric properties of the impairment rating scale in samples of children with attention deficit hyperactivity disorder and two school-based samples. Journal of Clinical Child and Adolescent Psychology, 35(3), 369–385.

    Article  PubMed  Google Scholar 

  • Fabiano, G. A., Vujnovic, R. K., Pelham, W. E., Waschbusch, D. A., Massetti, G. M., Pariseau, M. E., et al. (2010). Enhancing the effectiveness of special educational programming for children with attention deficit hyperactivity disorder using a daily report card. School Psychology Review, 39, 219–239.

    Article  Google Scholar 

  • Fabiano, G. A., Vujnovic, R. K., Waschbusch, D. A., Yu, J., Mashtare, T., Pariseau, M. E., et al. (2013b). A comparison of workshop training versus intensive, experiential training for improving behavior support skills in early educators. Early Childhood Research Quarterly, 28, 450–460.

    Article  Google Scholar 

  • Fischer, A. J., Dart, E. H., LeBlanc, H., Hartman, K., Steeves, R. O., & Gresham, F. M. (2016). An investigation of the acceptability of videoconferencing within a school-based behavioral consultation framework. Psychology in the Schools, 53, 240–252.

    Article  Google Scholar 

  • Frank, J. L., & Kratochwill, T. R. (2014). School-based problem-solving consultation. In W. P. Erchul & S. M. Sheridan (Eds.), Handbook of research in school consultation (2nd ed., pp. 18–39). New York, NY: Routledge.

    Google Scholar 

  • Fristad, M. A., Teare, M., Weller, E. B., Weller, R. A., & Salmon, P. (1998). Study III: Development and concurrent validity of the Children’s Interview for Psychiatric Syndromes-Parent version (P-ChIPS). Journal of Child and Adolescent Psychopharmacology, 8, 221–226.

    Article  PubMed  Google Scholar 

  • Gilbertson, D., Witt, J. C., Singletary, L. L., & VanDerHeyden, A. (2007). Supporting teacher use of interventions: Effects of response dependent performance feedback on teacher implementation of a math intervention. Journal of Behavioral Education, 16(4), 311–326.

    Article  Google Scholar 

  • Girio, E. L., & Owens, J. S. (2009). Teacher acceptability of evidence-based and promising treatments for children with attention-deficit/hyperactivity disorder. School Mental Health, 1, 16–25.

    Article  Google Scholar 

  • Greene, R., Beszterczey, S. K., Katzenstein, T., Park, L., & Goring, J. (2002). Are students with ADHD more stressful to teach? Patterns of teacher stress in an elementary school sample. Journal of Emotional and Behavioral Disorders, 10, 79–89.

    Article  Google Scholar 

  • Gresham, F. M., & Lopez, M. F. (1996). Social validation: A unifying concept for school-based consultation research and practice. School Psychology Quarterly, 11(3), 204.

    Article  Google Scholar 

  • Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.

    Google Scholar 

  • Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 655–679.

    Article  Google Scholar 

  • Harrison, J. R., State, T. M., Evans, S. W., & Schamberg, T. (2016). Construct and predictive validity of school acceptability: Scores from high school teacher ratings on the school intervention rating form. Journal of Positive Behavior Interventions, 18(2), 111–123.

    Article  Google Scholar 

  • Holdaway, A. S., Hustus, C. L., Owens, J. S., Evans, S. W., Coles, E. K., Himawan, L., Zoromski, A. K., Dawson, A. E., & Mixon, C. S. (2020). Incremental benefits of a daily report card over time for youth with disruptive behavior: Replication and Extension. School Mental Health.

  • Hoza, B. (2007). Peer functioning in children with ADHD. Academic Pediatrics, 7(1), 101–106.

    Google Scholar 

  • Johnson, L. D., Wehby, J. H., Symons, F. J., Moore, T. C., Maggin, D. M., & Sutherland, K. S. (2014). An analysis of preference relative to teacher implementation of intervention. Journal of Special Education, 48, 214–224.

    Article  Google Scholar 

  • Jones, H. A., & Chronis-Tuscano, A. (2008). Efficacy of teacher in-service training for attention-deficit/hyperactivity disorder. Psychology in the Schools, 45(10), 918–929.

    Article  Google Scholar 

  • Kazdin, A. E. (1981). Acceptability of child treatment techniques: The influence of treatment efficacy and adverse side effects. Behavior Therapy, 12, 493–506.

    Article  Google Scholar 

  • Loe, I. M., & Feldman, H. M. (2007). Academic and educational outcomes of children with ADHD. Journal of Pediatric Psychology, 32, 643–654.

    Article  PubMed  Google Scholar 

  • Martens, B. K., Witt, J. C., Elliott, S. N., & Darveaux, D. (1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research and Practice, 46, 191–198.

    Article  Google Scholar 

  • Merikangas, K. R., He, J., Brody, D., Fisher, P. W., Bourdon, K., & Koretz, D. S. (2010). Prevalence and treatment of mental health disorders among U.S. children in 2001–2004 NHANES. Pediatrics, 125, 75–81.

    Article  PubMed  Google Scholar 

  • Murray, D. W., Rabiner, D., Schulte, A., & Newitt, K. (2008). Feasibility and integrity of a parent–teacher consultation intervention for ADHD students. Child & Youth Care Forum, 37, 111–126.

    Article  Google Scholar 

  • Nastasi, B. K., & Truscott, S. D. (2000). Acceptability research in school psychology: Current trends and future directions. School Psychology Quarterly, 15(2), 117.

    Article  Google Scholar 

  • Noell, G. H., Gresham, F. M., & Gansle, K. A. (2002). Does treatment integrity matter? A preliminary investigation of instructional implementation and mathematics performance. Journal of Behavioral Education, 11, 51–67.

    Article  Google Scholar 

  • Noell, G., Witt, J., Gilbertson, D., Ranier, D., & Freeland, J. (1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77–88.

    Article  Google Scholar 

  • Owens, J. S., Coles, E. K., & Evans, S. W. (2014, September). The role of teacher knowledge, skills, and beliefs in the implementation of classroom management skills. Presentation at the Annual School Mental Health Research Summit. Pittsburgh, PA.

  • Owens, J. S., Coles, E. K., Evans, S. W., Himawan, L. K., Girio-Herrara, E. L., Holdaway, A. S., et al. (2017). Using multi-component consultation to increase the integrity with which teachers implement behavioral classroom interventions: A pilot study. School Mental Health, 9, 218–234.

    Article  Google Scholar 

  • Owens, J. S., Evans, S. W., Coles, E. K., Himawan, L. K., Holdaway, A. S., et al. (2020). Consultation for classroom management and targeted interventions: Examining benchmarks for teacher practices that produce desired change in student behavior. Journal of Emotional and Behavioral Disorders, 28(1), 52–64.

    Article  Google Scholar 

  • Owens, J. S., Holdaway, A. S., Zoromski, A. K., Evans, S. W., Himawan, L. K., Girio-Herrera, E., et al. (2012). Incremental benefits of a daily report card intervention over time for youth with disruptive behavior. Behavior Therapy, 43, 848–861.

    Article  PubMed  Google Scholar 

  • Owens, J. S., Murphy, C. E., Richerson, L., Girio, E. L., & Himawan, L. K. (2008). Science to practice in underserved communities: The effectiveness of school mental health programming. Journal of Clinical Child and Adolescent Psychology, 37, 434–447.

    Article  PubMed  Google Scholar 

  • Pelham, W. E., Gnagy, E. M., Greenslade, K. E., & Milich, R. (1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 31, 210–218.

    Article  PubMed  Google Scholar 

  • Pelham, W. E., Greiner, A. R., & Gnagy, E. M. (2008). Student behavior teacher response observation code manual (Unpublished manual).

  • Pintello, D. (2020). Commentary: Establishing scientific rigor and excellence in implementation science training to improve the deployment of evidence-based mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 47, 265–271.

    Article  PubMed  Google Scholar 

  • Pisecco, S., Huzinec, C., & Curtis, D. (2001). The effect of child characteristics on teachers’ acceptability of classroom-based behavioral strategies and psychostimulant medication for the treatment of ADHD. Journal of Clinical Child Psychology, 30(3), 413–421.

    Article  PubMed  Google Scholar 

  • Power, T. J., Hess, L. E., & Bennett, D. S. (1995). The acceptability of interventions for attention-deficit hyperactivity disorder among elementary and middle school teachers. Developmental and Behavioral Pediatrics, 16, 238–243.

    Article  Google Scholar 

  • Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34.

    Article  PubMed  Google Scholar 

  • Pyle, K., & Fabiano, G. A. (2017). Daily report card intervention and attention deficit hyperactivity disorder: A meta-analysis of single-case studies. Exceptional Children, 83(4), 378–395.

    Article  Google Scholar 

  • Reimers, T. M., Wacker, D. P., & Koeppl, G. (1987). Acceptability of behavioral interventions: A review of the literature. School Psychology Review, 16, 212–227.

    Article  Google Scholar 

  • Robb, J. A., Sibley, M. H., Pelham, W. E., Foster, E. M., Molina, B. S., Gnagy, E. M., et al. (2011). The estimated cost of ADHD to the US education system. School Mental Health, 3(3), 169–177.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vannest, K. J., Davis, J. L., Davis, C. R., Mason, B. A., & Burke, M. D. (2010). Effective intervention for behavior with a daily behavior report card: A meta-analysis. School Psychology Review, 39(4), 654–672.

    Article  Google Scholar 

  • Vereb, R. L., & DiPerna, J. C. (2004). Teachers’ knowledge of ADHD, treatments for ADHD, and treatment acceptability: An initial investigation. School Psychology Review, 33(3), 421.

    Article  Google Scholar 

  • Volpe, R. J., DuPaul, G. J., DiPerna, J. C., Jitendra, A. K., Lutz, G., Tresco, K., et al. (2006). Attention deficit hyperactivity disorder and scholastic achievement: A model of mediation via academic enablers. School Psychology Review, 35, 47–61.

    Article  Google Scholar 

  • Vujnovic, R. K., Holdaway, A. S., Owens, J. S., & Fabiano, G. A. (2014). Response to intervention for youth with attention-deficit/hyperactivity disorder: Incorporating an evidence-based intervention within a multi-tiered framework. In Weist, M. D., Lever, N. A., Bradshaw, C. P., & Sarno Owens, J. (Eds.), Issues in clinical child psychology. Handbook of school mental health: Research, training, practice, and policy (pp. 399–411).

  • Wechsler, D. (2011). Wechsler abbreviated scale of intelligence-second edition (WASI-II). San Antonio, TX: NCS Pearson.

    Google Scholar 

  • Williams, K., Noell, G., Jones, B. A., & Gansle, K. A. (2012). Modifying students’ classroom behaviors using an electronic daily behavior report card. Child and Family Therapy, 34, 269–289.

    Article  Google Scholar 

  • Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. R. Kratochwill (Ed.), Advances in school psychology (Vol. 4, pp. 251–288). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Witt, J. C., Moe, G., Gutkin, T. B., & Andrews, L. (1984). The effects of saying the same thing in different ways: The problem of language and jargon in school-based consultation. Journal of School Psychology, 22, 361–367.

    Article  Google Scholar 

  • Wolraich, M. L., McKeown, R. E., Visser, S. N., Bard, D., Cuffe, S., Neas, B., et al. (2014). The prevalence of ADHD: its diagnosis and treatment in four school districts across two states. Journal of Attention Disorders, 18, 563–575.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The investigators would like to thank the teachers, parents, and students who participated in this study.

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant (R324A120272). The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Author information

Authors and Affiliations

  1. Department of Psychology, Towson University, Towson, MD, 21252, USA

    Erin Girio-Herrera

  2. Department of Child and Adolescent Psychiatry and Behavioral Sciences, Children’s Hospital of Philadelphia, Philadelphia, PA, USA

    Theresa E. Egan & Alex S. Holdaway

  3. Department of Psychology, Ohio University, Athens, OH, USA

    Julie Sarno Owens, Steven W. Evans & Hannah D. Kassab

  4. Department of Psychology, Florida International University, Miami, FL, USA

    Erika K. Coles

  5. Department of Pediatrics, Ochsner Hospital for Children, Ochsner Health System, New Orleans, LA, USA

    Clifton S. Mixon

Authors
  1. Erin Girio-Herrera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theresa E. Egan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Julie Sarno Owens
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Steven W. Evans
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Erika K. Coles
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alex S. Holdaway
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Clifton S. Mixon
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hannah D. Kassab
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erin Girio-Herrera.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical Standards

Procedures were approved by the Institutional Review Boards at both universities and within all school districts. All procedures were performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Girio-Herrera, E., Egan, T.E., Owens, J.S. et al. Teacher Ratings of Acceptability of a Daily Report Card Intervention Prior to and During Implementation: Relations to Implementation Integrity and Student Outcomes. School Mental Health 13, 69–83 (2021). https://doi.org/10.1007/s12310-020-09400-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-020-09400-y

Keywords

Search

Navigation