Skip to main content

Advertisement

Log in

A Simple Matter of Time? School-Level Analysis of the Relationship between Time Allocation, Treatment Integrity, and Student Outcome

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

Treatment integrity is critical to realizing the effects of evidence-based programs and practices (EBPPs) that aim to promote student mental and behavioral health. Across disciplines, time is consistently referenced as one of the most significant barriers to successful implementation outcomes (e.g., treatment integrity) of EBPPs. However, there is limited to no research that has examined the relationship between time allocation, treatment integrity, and student outcomes, especially at the school organizational level. This study used data from 106 elementary schools implementing Tier 1 school-wide positive behavior interventions and supports to examine the associations among two types of time allocation (collaboration/planning and professional development), school-level treatment integrity, and suspension. Results supported significant associations between two types of time allocation and both implementation and student behavioral outcomes. Moreover, the mediation analysis suggested that treatment integrity mediated the relationship between both types of time allocation (collaboration/planning and professional development) and student behavioral outcomes at the school level. Limitations and implications for research and practice are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014). Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health, 35, 255–274.

    PubMed  PubMed Central  Google Scholar 

  • Adelman, H. S., & Taylor, L. (2006). Mental health in schools and public health. Public Health Reports, 121(3), 294–298.

    PubMed  PubMed Central  Google Scholar 

  • American Association for the Advancement of Science, & National Research Council. (2009). A survey of attitudes and actions on dual use research in the life sciences: A collaborative effort of the National Research Council and the American Association for the Advancement of Science. Washington: National Academies Press.

    Google Scholar 

  • Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120.

    Google Scholar 

  • Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of school psychology, 48(1), 5–37.

    PubMed  Google Scholar 

  • Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173.

    PubMed  Google Scholar 

  • Bonham, C. A., Sommerfeld, D., Willging, C., & Aarons, G. A. (2014). Organizational factors influencing implementation of evidence-based practices for integrated treatment in behavioral health agencies. Psychiatry Journal.

  • Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133.

    Google Scholar 

  • Cameron, E., & Green, M. (2019). Making sense of change management: A complete guide to the models, tools and techniques of organizational change. London: Kogan Page Publishers.

    Google Scholar 

  • Cimdi, C. (2012). Evidence-based practice (EBP) in rehabilitative physiotherapy. Internet Journal of Allied Health Sciences and Practice, 10(4), 10.

    Google Scholar 

  • Christie, C. A., Nelson, C. M., & Jolivette, K. (2004). School characteristics related to the use of suspension. Education and Treatment of Children, 509–526.

  • Collyer, H., Eisler, I., & Woolgar, M. (2019). Systematic literature review and meta-analysis of the relationship between adherence, competence and outcome in psychotherapy for children and adolescents. European Child & Adolescent Psychiatry, 2019, 1–15.

    Google Scholar 

  • Cook, C. R., Coco, S., Zhang, Y., Fiat, A. E., Mylien T., et al. (2018) Cultivating Positive Teacher-Student Relationships: Preliminary Evaluation of the Establish–Maintain–Restore (EMR) Method. School Psychology Review, 47 (3), 226–243.

  • Cook, C. R., Lyon, A. R., Kubergovic, D., Wright, DB., & Zhang, Y. (2015) A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi-tiered system of supports. School Mental Health, 7(1), 49–60.

    Google Scholar 

  • Cook, J. M., Schnurr, P. P., Biyanova, T., & Coyne, J. C. (2009). Apples don’t fall far from the tree: Influences on psychotherapists’ adoption and sustained use of new therapies. Psychiatric Services, 60, 671–676.

    PubMed  Google Scholar 

  • Daniels, N. (2016). Resource allocation and priority setting. In Public health ethics: Cases spanning the globe (pp. 61–94). Springer, Cham.

  • Dixon, M. A., & Cunningham, G. B. (2006). Data aggregation in multilevel analysis: A review of conceptual and statistical issues. Measurement in physical education and exercise science, 10(2), 85–107.

    Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327.

    PubMed  Google Scholar 

  • Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432.

    PubMed  Google Scholar 

  • Edmondson, A. C., Higgins, M., Singer, S., & Weiner, J. (2016). Understanding psychological safety in health care and education organizations: A comparative perspective. Research in Human Development, 13(1), 65–83.

    Google Scholar 

  • Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 157.

    PubMed  PubMed Central  Google Scholar 

  • Evans, S. W., & Weist, M. D. (2004). Commentary: Implementing empirically supported treatments in the schools: What are we asking? Clinical Child and Family Psychology Review, 7(4), 263–267.

    PubMed  Google Scholar 

  • Fan, C. H., Zhang, Y., Cook, C. R., & Yang, N. J. (2018). Exploring the Factor Structure of the RTI Readiness and Implementation Survey. Journal of Applied School Psychology, 34(4), 360–387.

    Google Scholar 

  • Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A.,… Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77.

    PubMed  Google Scholar 

  • Fritz, M. S., & MacKinnon, D. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18(3), 233–239.

    PubMed  Google Scholar 

  • Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39(1), 3.

    Google Scholar 

  • Goldsmith, K. A., MacKinnon, D. P., Chalder, T., White, P. D., Sharpe, M., & Pickles, A. (2018). Tutorial: The practical application of longitudinal structural equation mediation models in clinical trials. Psychological Methods, 23(2), 191.

    PubMed  Google Scholar 

  • Goense, P. B., Boendermaker, L., & van Yperen, T. (2016). Support systems for treatment integrity. Research on Social Work Practice, 26(1), 69–73.

    Google Scholar 

  • Greenberg, M. T., & Abenavoli, R. (2017). Universal interventions: Fully exploring their impacts and potential to produce population-level impacts. Journal of Research on Educational Effectiveness, 10(1), 40–67.

    Google Scholar 

  • Gutshall, C. A. (2013). Teachers’ mindsets for students with and without disabilities. Psychology in the Schools, 50(10), 1073–1083. https://doi.org/10.1002/pits.21725.

    Article  Google Scholar 

  • Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665–679.

    PubMed  Google Scholar 

  • Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79(2), 181–193.

    Google Scholar 

  • Hoagwood, K., & Johnson, J. (2003). School psychology: A public health framework: I. From evidence-based practices to evidence-based policies. Journal of School Psychology, 41(1), 3–21.

    Google Scholar 

  • Horner, R. H., & Sugai, G. (2015). School-wide PBIS: An example of applied behavior analysis implemented at a scale of social importance. Behavior Analysis in Practice, 8(1), 80–85.

    PubMed  PubMed Central  Google Scholar 

  • Horner, R. H., Kincaid, D., Sugai, G., Lewis, T., Eber, L., Barrett, S.,… Algozzine, B. (2014). Scaling up school-wide positive behavioral interventions and supports: Experiences of seven states with documented success. Journal of Positive Behavior Interventions, 16(4), 197–208.

    Google Scholar 

  • Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133–144.

    Google Scholar 

  • Imbens, G. W., & Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge: Cambridge University Press.

    Google Scholar 

  • Johnson, G., Melin, L., & Whittington, R. (2003). Micro strategy and strategizing: Towards an activity-based view. Journal of Management Studies, 40(1), 3–22.

    Google Scholar 

  • Kam, C.-M., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.

    PubMed  Google Scholar 

  • Kessler, R., & Glasgow, R. E. (2011). A proposal to speed translation of healthcare research into practice: Dramatic change is needed. American Journal of Preventive Medicine, 40(6), 637–644.

    PubMed  Google Scholar 

  • Kypri, K., Samaranayaka, A., Connor, J., Langley, J. D., & Maclennan, B. (2011). Non-response bias in a web-based health behaviour survey of New Zealand tertiary students. Preventive Medicine, 53, 274–277.

    PubMed  Google Scholar 

  • La Salle, T. P., Wang, C., Wu, C., & RochaNeves, J. (2020). Racial mismatch among minoritized students and white teachers: Implications and recommendations for moving forward. Journal of Educational and Psychological Consultation, 30(3), 314–343.

    Google Scholar 

  • Lane, K. L., Bocian, K. M., MacMillan, D. L., & Gresham, F. M. (2004). Treatment integrity: An essential—but often forgotten—component of school-based interventions. Preventing School Failure: Alternative Education for Children and Youth, 48(3), 36–43.

    Google Scholar 

  • Larson, M., Cook, C. R., Fiat, A., & Lyon, A. R. (2018). Stressed teachers don’t make good implementers: Examining the interplay between stress reduction and intervention fidelity. School Mental Health, 10(1), 61–76.

    Google Scholar 

  • Locke, J., Lawson, G. M., Beidas, R. S., Aarons, G. A., Xie, M., Lyon, A. R.,… Spaulding, C. (2019). Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools: A cross-sectional observational study. Implementation Science, 14(1), 1–9.

    Google Scholar 

  • Locke, J., Olsen, A., Wideman, R., Downey, M. M., Kretzmann, M., Kasari, C., & Mandell, D. S. (2015). A tangled web: The challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behavior Therapy, 46(1), 54.

    PubMed  Google Scholar 

  • Low, S., Smolkowski, K., & Cook, C. (2016). What constitutes high-quality implementation of SEL programs? A latent class analysis of Second Step® implementation. Prevention Science, 17(8), 981–991.

    PubMed  Google Scholar 

  • Lyon, A. R., & Bruns, E. J. (2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11(1), 106–114.

    PubMed  PubMed Central  Google Scholar 

  • Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., Aarons, G. A. (2018) Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13(1).

  • Lyon, A. R., Ludwig, K., Romano, E., Leonard, S., Vander Stoep, A., & McCauley, E. (2013). “If it’s worth my time, I will make the time”: School-based providers’ decision-making about participating in an evidence-based psychotherapy consultation program. Administration and Policy in Mental Health and Mental Health Services Research, 40(6), 467–481.

    PubMed  Google Scholar 

  • MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation, confounding and suppression effect. Prevention Science, 1(4), 173–181.

    PubMed  PubMed Central  Google Scholar 

  • Maxwell, S. E., & Cole, D. A. (2007). Bias in cross-sectional analyses of longitudinal mediation. Psychological Methods, 12, 23–44. https://doi.org/10.1037/1082-989X.12.1.23.

    Article  PubMed  Google Scholar 

  • Mendez, L. M. R., Knoff, H. M., & Ferron, J. M. (2002). School demographic variables and out-of-school suspension rates: A quantitative and qualitative analysis of a large, ethnically diverse school district. Psychology in the Schools, 39(3), 259–277.

    Google Scholar 

  • Manges, M. E., & Nickerson, A. B. (2020). Student Knowledge Gain Following the Second Step Child Protection Unit: The Influence of Treatment Integrity. Prevention Science, 21(8), 1037–1047.

    PubMed  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73.

    Google Scholar 

  • McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of schoolwide positive behavior support. Exceptional Children, 79(3), 293–311.

    Google Scholar 

  • McIntosh, K., Massar, M. M., Algozzine, R. F., George, H. P., Horner, R. H., Lewis, T. J., & Swain-Bradway, J. (2017). Technical adequacy of the SWPBIS tiered fidelity inventory. Journal of Positive Behavior Interventions, 19(1), 3–13.

    Google Scholar 

  • McIntosh, K., Gion, C., & Bastable, E. (2018). Do schools implementing SWPBIS have decreased racial and ethnic disproportionality in school discipline. OSEP NTAC on Positive Behavioral Interventions and Supports: PBIS Evaluation Brief.

    Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodrıguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32.

    Google Scholar 

  • Minugh, P. A., Janke, S. L., Lomuto, N. A., & Galloway, D. K. (2007). Adolescent substance abuse treatment resource allocation in rural and frontier conditions: The impact of including organizational readiness to change. The Journal of Rural Health, 23, 84–88.

    PubMed  Google Scholar 

  • Nilsen, P., & Bernhardsson, S. (2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19(1), 189.

    PubMed  PubMed Central  Google Scholar 

  • Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: A meta-analysis. School Psychology Review, 44(2), 224–240.

    Google Scholar 

  • Osborne, J. W., & Waters, E. (2002). Four assumptions of multiple regression that researchers should always test. Practical Assessment, Research, and Evaluation, 8(1), 2.

    Google Scholar 

  • Pas, E. T., Johnson, S. R., Debnam, K. J., Hulleman, C. S., & Bradshaw, C. P. (2019). Examining the relative utility of PBIS implementation fidelity scores in relation to student outcomes. Remedial and Special Education, 40(1), 6–15.

    Google Scholar 

  • Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004). Two replicable suppressor situations in personality research. Multivariate Behavioral Research, 39(2), 303–328.

    PubMed  Google Scholar 

  • Pas, E. T., & Bradshaw, C. P. (2012). Examining the association between implementation and outcomes. The Journal of Behavioral Health Services & Research, 39(4), 417–433.

    Google Scholar 

  • Pearl, J. (2013). Direct and indirect effects. arXiv preprint http://arXiv:1301.2300.

  • Peugh, J. L., Strotman, D., McGrady, M., Rausch, J., & Kashikar-Zuck, S. (2017). Beyond intent to treat (ITT): A complier average causal effect (CACE) estimation primer. Journal of School Psychology, 60, 7–24.

    PubMed  Google Scholar 

  • Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36(4), 717–731.

    Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A.,… Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.

    PubMed  Google Scholar 

  • Rimm-Kaufman, S. E., Baroody, A. E., Curby, T. W., Ko, M., Thomas, J. B., Merritt, E. G.,… DeCoster, J. (2014). Efficacy of the responsive classroom approach: Results from a 3-year, longitudinal randomized controlled trial. American Educational Research Journal, 51, 567–603.

    Google Scholar 

  • Saborit, J. A. P., Fernández-Río, J., Estrada, J. A. C., Méndez-Giménez, A., & Alonso, D. M. (2016). Teachers’ attitude and perception towards cooperative learning implementation: Influence of continuing training. Teaching and Teacher Education, 59, 438–445.

    Google Scholar 

  • Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment Integrity of Interventions With Children in the School Psychology Literature from 1995 to 2008. School Psychology Review, 40(1), 72.

    Google Scholar 

  • Sanetti, L. M. H., Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445

    Google Scholar 

  • Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education and Treatment of Children, 2008, 351–380.

    Google Scholar 

  • Spearman, C. (1961). The proof and measurement of association between two things.

  • Spear-Swerling, L., & Zibulsky, J. (2014). Making time for literacy: Teacher knowledge and time allocation in instructional planning. Reading and Writing, 27(8), 1353–1378.

    Google Scholar 

  • Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005). Establishing a causal chain: Why experiments are often more effective than mediational analyses in examining psychological processes. Journal of Personality and Social Psychology, 89(6), 845.

    PubMed  Google Scholar 

  • Stahmer, A. C., Rieth, S., Lee, E., Reisinger, E. M., Mandell, D. S., & Connell, J. E. (2015). Training teachers to use evidence-based practices for autism: Examining procedural implementation fidelity. Psychology in the Schools, 52(2), 181–195.

    PubMed  Google Scholar 

  • Suhrheinrich, J., Stahmer, A. C., Reed, S., Schreibman, L., Reisinger, E., & Mandell, D. (2013). Implementation challenges in translating pivotal response training into community settings. Journal of Autism and Developmental Disorders, 43(12), 2970–2976.

    PubMed  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., McLeod, B. D., Kunemund, R., & McKnight, K. (2019). Common practice elements for improving social, emotional, and behavioral outcomes of young elementary school students. Journal of Emotional and Behavioral Disorders, 27(2), 76–85.

    Google Scholar 

  • Swain-Bradway, J., Swoszowski, N. C., Boden, L. J., & Sprague, J. R. (2013). Voices from the field: Stakeholder perspectives on PBIS implementation in alternative educational settings. Education and Treatment of Children, 36(3), 31–46.

    Google Scholar 

  • Schwarzer, R., Lippke, S., & Luszczynska, A. (2011). Mechanisms of health behavior change in persons with chronic illness or disability: The Health Action Process Approach (HAPA). Rehabilitation Psychology, 56(3), 161.

    PubMed  Google Scholar 

  • Van Denise, T. B., Bunger, A., Burgin, S., Wilson, A. B., & Cuddeback, G. S. (2019). Using the consolidated framework for implementation research to examine implementation determinants of specialty mental health probation. Health & Justice, 7(1), 17.

    Google Scholar 

  • Tweed, S. (2013). Technology implementation: Teacher age, experience, self-efficacy, and professional development as related to classroom technology integration.

  • United States. (2004). Individuals with disabilities education improvement act of 2004 (IDEA). Bethesda, MD: ProQuest

  • Vincent, C. G., Sprague, J. R., Pavel, M., Tobin, T. J., & Gau, J. M. (2015). Effectiveness of schoolwide positive behavior interventions and supports in reducing racially inequitable disciplinary exclusion (p. 207). Closing the school discipline gap: Equitable remedies for excessive exclusion.

    Google Scholar 

  • Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200.

    PubMed  PubMed Central  Google Scholar 

  • Weiner, B. J., Mettert, K. D., Dorsey, C. N., Nolen, E. A., Stanick, C., Powell, B. J., & Lewis, C. C. (2020). Measuring readiness for implementation: A systematic review of measures’ psychometric and pragmatic properties. Implementation Research and Practice.

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yanchen Zhang.

Ethics declarations

Availability of data and material

Deidentified data is available upon request from the corresponding author.

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Cook, C.R. & Lyon, A.R. A Simple Matter of Time? School-Level Analysis of the Relationship between Time Allocation, Treatment Integrity, and Student Outcome. School Mental Health 14, 73–87 (2022). https://doi.org/10.1007/s12310-021-09412-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-021-09412-2

Keywords

Navigation