Project ENHANCE: Assessing Professional Learning Needs for Implementing Comprehensive, Integrated, Three-Tiered (Ci3T) Models of Prevention

Abstract

We report findings from a multistate survey of 720 faculty and staff from 25 elementary schools in five districts across three states and geographic regions participating in an IES Network grant examining integrated tiered systems. In this preregistered study, we replicated and extended previous inquiry examining educators’ views of (1) implementation of core components of their school’s Comprehensive, Integrated, Three-Tiered (Ci3T) model of prevention; and (2) preference for professional learning (content and avenue). Results indicated more than half of respondents indicated high levels of implementation of core features of Ci3T across Tiers 1, 2, and 3. Educators reported high levels of implementation for 10 out of 19 research-based educational practices used within tiered systems with a statistically significant relation between ratings of implemented practices and the desire for support with most practices. Respondents identified their top three areas for professional development needed in the coming year as behavior deescalation techniques, small-group social skills instruction, and strategies for supporting students with internalizing behavior patterns. For potential professional learning avenues, respondents’ top ratings were in-district, during-school workshops, course for college credit on-line, teacher collaboratives/networks, and one-to-one coaching or mentoring. There were many similarities among educators’ ratings across implementation year and state. Low levels of implementation across many core Ci3T and common educational practices were reported by educators working within the most experienced schools. We conclude with a discussion of implications, limitations, and future directions.

Introduction

Educational systems and educators are charged with the remarkable task of meeting academic, behavioral, and social emotional well-being needs of all students. Educational leaders have embraced this charge through policies and practices taking a system-level response so all students have the full set of skills necessary to be prepared to advance to college and careers (Every Student Succeeds Act, 2015). Although schools continue to focus on academic achievement, 29 states now also set standards for social emotional learning—recognizing implications for educational, career, and social fulfillment—and have adopted standards for interpersonal and self-determination skills needed to navigate one’s social environment across school, career, and social settings (Positive Action, 2020).

Educational leaders enact systemic policies and practices to stimulate school improvement efforts using tiered systems. Tiered systems utilize a framework for fulfilling the school’s mission and purpose through a data informed prevention and intervention approach where increases in student need are met with responses of matched intensity (McIntosh & Goodman, 2016). For example, primary (Tier 1) prevention encompasses the educational opportunities afforded to all students attending the school and is typically effective for approximately 80% of students. Secondary (Tier 2) prevention comprises strategies, practices, and programs to address targeted learning areas for approximately 10%–15% of students who need more than Tier 1 provides (e.g., oral reading fluency, attention to task, initiating social interactions). Tertiary (Tier 3) interventions are the most intensive interventions, and are intended for the small number of students (e.g., 5%) with the most intensive learning needs or who have multiple risk factors. Tiered systems widely used in schools today include response to intervention (RTI; Fuchs et al., 2012; focused on academic domains) and positive behavioral interventions and supports (PBIS; Sugai & Horner, 2009; focused primarily on behavioral domains). As the application of tiered systems in education continues to evolve, integrated tiered systems have emerged (Institute of Education Science [IES], 2018). For example, interconnected systems framework (ISF; Barrett et al., 2013; integrating PBIS with school-based mental health supports), multitiered system of supports (MTSS; integrating one or more academic domains with PBIS), and the Comprehensive, Integrated, Three-Tiered (Ci3T) model of prevention (Lane & Menzies, 2003; Lane et al., 2020c; addressing students’ academic, behavioral, and social needs in one coordinated model).

Comprehensive Integrated Three-Tiered Model of Prevention

In the Ci3T model, schools select and implement research- and evidence-based academic instruction, PBIS to address students’ behavioral learning, and a validated social skills curriculum such as Second Step® (Committee for Children, 1992) focused on developing students’ social skill sets. Ci3T offers a comprehensive integrated data-driven prevention model with structures for monitoring system- and student-level outcomes to determine effectiveness in meeting systems-level goals and to inform instruction for students. In the Ci3T model, educators examine multiple sources of data to inform decision making, with student performance measures analyzed alongside treatment integrity and social validity data. Ci3T is a prevention model and therefore relies on the early detection of students who need more than Tier 1. Systematic academic and behavioral screening data in tandem with other school data (e.g., attendance, office discipline referrals, course progress data) are used to monitor student progress. However, to accurately interpret these data, schools examine treatment integrity data to assess the extent to which students have been afforded the planned educational experiences (Buckman, 2021) and social validity data to monitor stakeholders’ acceptance of the procedures, goals, and outcomes. Data-informed decisions are made to inform students’ educational experiences (e.g., using data to connect students to relevant, research-based Tier 2 and Tier 3 supports) and adults’ experiences (e.g., using data to inform professional learning offerings to facilitate high-fidelity implementation; providing an opportunity to contribute to programmatic decisions).

Integrated tiered systems are complex, requiring effective collaboration among a variety of school-based professionals (e.g., administrators, special and general educators) and coordinated instructional delivery across multiple domains (e.g., academic, behavioral, social emotional well-being). Through our Ci3T partnership work, and in related work to develop online training to use behavioral assessments (e.g., Chafouleas et al., 2015), we have learned school teams and teachers benefit from on-demand professional learning resources to learn how to synthesize data for making systematic decisions regarding student intervention needs.

Professional Learning in Ci3T Models of Prevention

Systems-change efforts are sustained most effectively when organizational structures are in place to facilitate change (Fixsen et al., 2005; Horner et al., 2017). For example, policies, data systems, and professional learning facilitate successful systems change, recognizing the need for each to adapt for initial implementation, full operation, innovation, and sustained practice phases of implementation science (Fixsen et al., 2005; Taxman & Belenko, 2012). Professional learning plays a fundamental role in the implementation of system-level change efforts, such as tiered systems (McIntosh et al., 2013), with professional learning being an essential implementation driver for high-fidelity implementation (Horner et al., 2017). Effective professional learning practices include those focused on content and pedagogy, of sustained duration, and collaborative work (Wei et al., 2009). To engage educators and minimize burden, professional learning must be relevant for their context and delivered through socially acceptable avenues (Lane et al., 2015) across all change agents (Fixsen et al., 2009). Therefore, information is needed from educators (e.g., change agents) regarding their professional learning needs, interests, and desired avenues in order to support their implementation of their schoolwide plan.

Lane et al. (2015) developed the Ci3T Professional Learning Survey to inform professional development needs (e.g., Oakes et al., 2021) by assessing educator views regarding (1) implementation of their school’s Ci3T model of prevention components and (2) areas in which educators might benefit from professional learning, including desired venues (see detailed description in “Method,” below). The Ci3T Professional Learning Survey has been used to examine the professional learning needs of educators in two studies. First, Lane et al. (2015) conducted a statewide survey of 333 school administrators to examine (1) the degree to which Ci3T practices were in place, (2) content and skills desired for additional professional learning, and (3) avenues for them to engage in these learning activities. The authors found a positive relation between the practices in place and their interest in professional learning on these practices. Administrators indicated in-district workshops offered during the school day and practice guides to be the most favorable avenues for accessing professional learning. Second, Oakes et al. (2021) administered the Ci3T Professional Learning Survey with 253 educators in 21 schools within a Midwestern U.S. district at the completion of a 2-year IES-funded researcher–practitioner partnership. Educators reported high levels of use of Ci3T practices (higher scores at the elementary level compared to secondary level) with a positive relation between implemented practices and desires for professional learning in four areas: small-group self-determination instruction, peer-mediated support strategies, check-in/check-out, and strategies for internalizing behavior. Favored avenues were consistent with Lane et al.’s (2015) findings with the addition of courses for college credit (on-line).

Purpose

The current survey study is situated within Project ENHANCE, one of four research network grants (integrated multitiered systems of support [I-MTSS]) funded by the IES (2018) to examine how to design, implement, and evaluate complex integrated systems (I-MTSS, n.d.). As part of Project ENHANCE, we aimed to develop on-demand modules to assist Ci3T leadership teams with professional learning necessary to lead efforts in their schools and districts effectively and efficiently. We conducted this study during the 2019–2020 academic year to learn from stakeholders about their use of Ci3T core practices and needed professional learning related to data-informed Tier-1, -2, and -3 efforts. We gathered data using the Ci3T Professional Learning Survey to inform development of on-demand professional learning resources. We therefore extend the findings of Lane et al. (2015) and Oakes et al. (2021) by examining the professional learning needs of educators in three geographic U.S. regions and for schools in various stages of implementation. Research on systems change efforts rooted in implementation science suggests patterns in implementation are likely to change over time (Fixsen et al., 2005) and professional learning must be responsive to these changes. It is possible these changes will meaningfully affect the needs for professional learning to promote fidelity of Ci3T implementation.

Our research questions focused on current practices in schools (research questions 1 and 2), resources and professional development needs (research questions 3–6), and preferred avenues for professional development and learning (research questions 7–8). Specific questions were: (1) To what extent do respondents report their schools are currently implementing features common to Ci3T models? (2) Are there differences in the extent to which these features are implemented across districts and states? And stages of implementation? (3) To what degree do respondents report their educators in their school implement practices consistent with the framework of Ci3T models of prevention and what is their desire for additional professional development support for addressing these practices? (4) What is the relation between current implementation and desire for professional development? (5) Are there differences in the extent to which respondents implement and desire professional development to support these educational practices and supports (e.g., by district, state, stages of implementation)? (6) What areas do respondents prioritize for professional development? (7) What avenues do respondents prefer for professional development to learn more about Tier 1, 2, and 3 supports within Ci3T models? (8) Are there differences in the reported potential avenues for professional development and learning (e.g., by district, state, stages of implementation)?

Method

Participants

Participants were 720 faculty and staff from 25 elementary schools from five districts across three states participating in an IES Network grant examining integrated tiered systems. The three states were located in the West (WA), Midwest (KS), and Northeast (VT) regions of the United States. Most respondents were female (n = 651; 90.54%) and white (n = 641; 96.10%), with respondents having an average of 14.94 (SD = 10.04; range = 0–45) years of experience in education. See supplemental Tables S1 and S2 for participant and school characteristics. In our preregistered plan (Lane, Buckman, et al., 2020a), we indicated we would invite all faculty, staff, and administrators from elementary schools participating in Project ENHANCE to complete the survey. In spring 2020, we launched the Ci3T Professional Learning Survey in 25 schools, 5 fewer than proposed in the preregistration plans that committed to Washington (n = 6 proposed, 5 invited, 5 actual), Kansas (n = 18 proposed, 18 invited, 17 actual), and Vermont (n = 6 proposed, 5 invited, 3 actual). All elementary schools were implementing Ci3T and receiving implementation support as part of Project ENHANCE.

Procedures

Across districts Ci3T leadership teams participated each year in a university-supported year-long, five-session, Ci3T implementation professional learning Series (see https://www.ci3t.org/imp). As part of these sessions, team members analyzed their school-site data and engaged in data-informed professional learning efforts to empower teams to support their faculty and staff with implementation efforts. Data collection for the current study took place during a 14-week period in spring 2020 during the 1st year of a 5-year IES Network grant, Project ENHANCE. Educators at each school had a period of approximately 3–4 weeks to complete the survey (see below for further details related to survey distribution). After securing university and district approvals, we used the Qualtrics online survey platform to distribute an informational letter and survey (description below) via email to all faculty and staff employed at each school. The information letter indicated the intent of this study was to assess faculty and staff’s professional learning needs and preferences related to implementation fidelity and sustainability of Ci3T. All faculty and staff were aware their district was participating in Project ENHANCE to glean information from a range of implementers (initial to advanced) to inform the design, implementation, and evaluation of enhanced Ci3T professional learning materials to facilitate implementation. The informational letter emphasized the study was voluntary, the benefits and risks of participation, and the confidential nature of the results. Results were shared with schools in aggregate form (i.e., deidentified school-level results) to inform the development of future professional learning materials, as part of a data-informed process.

At the end of the information letter in Qualtrics there was an option to participate in the study. For surveys not completed initially, participants received two prompts (approximately 1 week later, and prior to the end of the data collection window; Dillman et al., 2008). We distributed surveys to 1,702 individuals across five districts, with response rates as follows: Washington District 4 (72/221 = 32.58%), Kansas District 1 (206/489 = 42.13%), Kansas District 2 (124/308 = 40.26%), Kansas District 3 (222/483 = 45.96%), and Vermont District 5 (96/201 = 47.76%).

Procedural Considerations in Regard to COVID-19

In response to the spread of COVID-19, the World Health Organization announced on January 30, 2020 a Public Health Emergency of International Concern and declared COVID-19 a pandemic on March 11, 2020. School responses to COVID-19 (e.g., shifts to remote learning) began March 16, 2020, during the data collection window. To promote and maintain respectful and responsible inquiry (Lane, et al. 2020b), we retained our original procedures, measures, and data analytic plan with the exception of working with district and school leaders to shift distribution and reminder schedules as schools made initial adjustments during the early stages of the pandemic.

Measures

The Ci3T Professional Learning Survey examines views of Ci3T practices in place and resources and professional learning needs to facilitate implementation (see Lane et al., 2015; Oakes et al., 2021). The survey includes 109 items organized in five includes sections, with most items using a 5-point Likert-type scale: (1) implementation of core Ci3T features, (2) resources and professional learning needs around educational practices commonly implemented within Ci3T models, (3) professional learning avenues, (4) skills and behaviors essential for success, and (5) respondents’ demographics. Operational definitions were not provided in order to limit completion time (Lane et al., 2015). Given the focus of the current study, the 20 items assessing skills and behaviors essential for success were excluded from analyses. We estimated participation would take less than 45 min.

Implementation of Core Ci3T Features

Respondents rated the degree to which 25 core features of Ci3T models were being implemented across academic, behavioral, and social domains (see Tables 1 and 2). Items were parallel to features in Lane et al. (2015), including instructional and curricular considerations (e.g., selection of a school-wide social skills curriculum); procedures for teaching, reinforcing, and monitoring (including treatment integrity and student performance); and data-informed decision-making processes. Items included Tier 1 features (e.g., teaching and reinforcement; 13 items), Tier 2 and 3 features (e.g., additional supports; 4 items), and features pertaining to monitoring and data-informed decision making (8 items). Respondents rated, “To what extent is your school currently implementing this feature” using a 5-point Likert-type scale (1 = not at all, 3 = somewhat, 5 = fully implemented). Cronbach’s alphas for the current sample were .91, .92, and .88, respectively.

Table 1 Ratings of features of three-tiered models currently being implemented

Table 2 Mean score comparisons between implementation stages: ratings of features of three-tiered models currently being implemented

Resources and Professional Development Needs

Respondents rated 19 educational practices typically implemented across the Ci3T model continuum, featuring research-based strategies and practices for each learning domain (e.g., academic, behavioral, social; see Table 3). Items included traditional Tier 2 supports (e.g., small-group interventions), low-intensity supports (e.g., behavioral contracts), teacher-level strategies (e.g., instructional choice), as well as Tier 3 supports (e.g., intensive reading instruction; functional behavioral assessments). Respondents rated (1) the degree to which their school was implementing the practice (Table 3) and (2) their desire for additional assistance to facilitate implementation (Table 4), with correlations between constructs (Table 5). Responses included implementation (1 = not at all, 3 = somewhat, 5 = fully) and desire for support (1 = no desire, 3 = some desire, 5 = strong desire). Respondents selected three priorities for professional development in the next school year. Cronbach’s alphas were .94 and .96.

Table 3 Educational practices and supports currently implemented by implementation stage

Table 4 Desire for professional development on how to implement educational practices by implementation stage

Table 5 Relation between educational practices and supports currently implemented and desire for professional development on how to implement educational practices

Professional Development Avenues

Participants rated how likely they were to engage in 24 potential avenues for professional development (e.g., state conferences, brief “good practice” guides; see Table 6), with options developed by Lane et al. (2015) based on the professional development literature (e.g., Garet et al., 2001; Penuel et al., 2007). We added additional items related to project-specific professional learning avenues (e.g., interactive eBook, web-based professional learning module). The 5-point Likert-type rating scale ranged from 1 = very unlikely, 3 = somewhat likely, to 5 = very likely, with the assumption each opportunity was available. Cronbach’s alpha was .93.

Table 6 Potential avenues for professional development and learning by implementation stage

Essential Demographics

At the opening of the survey, participants provided demographic information: gender (male, female, do not identify as male or female), age, ethnicity and race, highest degree obtained, role at their school, grade levels taught, as well as experience (e.g., years in education, Ci3T leadership team membership, professional learning hours earned in 2019–2020). See supplemental Table S1 for participant characteristics.

Design and Analysis

As defined in our preregistered data analytic plan (Lane, Buckman, et al., 2020a), we used descriptive and inferential statistics to answer our research questions. We employed descriptive statistics to summarize (1) current practices in schools, (2) educational practices currently in place as well as desire for professional development in these areas, and (3) preferences for potential avenues for professional development. In terms of current school practices, we conducted a series of one-way ANOVAs to compare mean levels, contrasting the views of different subgroups on (1) Tier 1, (2) Tiers 2 and 3, and (3) monitoring and decision making. In our preregistration, we planned to draw comparisons across stages of implementation as well as across districts and states. However, given the resulting sample size of 720 respondents with some districts having all schools in the same implementation phase, we focused on comparisons between stage of implementation and state, with the latter intended to examine regional differences. We used Tukey multiple comparisons (α = .05) to determine differences in mean scores for all comparisons proposed in the research questions. We included state comparison tables in supplemental files (S3–S6). Also, we computed Pearson correlation coefficients to explore relations between ratings of current implementation and desire for professional development for each educational practice (range: -1.0–1.0). We analyzed all available data from the 720 respondents, including partially completed surveys. Visual inspection of survey completion patterns showed no clear patterns of missing item-level data. For example, respondents did not have a pattern of stopping to respond half-way through the survey.

Results

Implementation of Core Features

To answer the first set of research questions, we present results for core features of (1) Tier 1, (2) Tiers 2 and 3, and (3) monitoring and data-informed decision making (Table 1). Then we explore differences in implementation of core Ci3T features according to building-level stages of implementation (see Table 2, for mean score comparisons by implementation stage and Supplemental Table S3 for comparisons by state).

Core Features of Tier 1 Efforts

More than 50% of respondents indicated a high level of implementation (ratings of 4 or 5) for each of the 13 Tier 1 features. All mean score ratings exceeded the 3.00 midpoint, with average scores ranging from 3.99 (SD = 1.06) for an established discipline plan for responding to rule infractions to 4.72 (SD = 0.59) for school-wide expectations for all key settings. More than 75% of respondents reported school-wide expectations for all key settings were fully implemented (rating a 5), yet monthly instruction of school-wide expectations were implemented at a lower level (M = 4.06). In terms of social skills, 58% of respondents indicated a school-wide social skills curriculum was fully implemented (i.e., rating a 5) and the same percentage of respondents indicated they fully implemented at least monthly instruction in the social skills curriculum.

Core Features of Tier 2 and 3 Efforts

More than 80% of respondents indicated high implementation (ratings of 4 or 5) for Tier 2 and Tier 3 support for academic issues; we observed similar—although slightly lower—implementation of Tier 2 and Tier 3 behavioral or social supports. Mean scores suggested high level of implementation, ranging from 4.00 (SD = 1.03, Tier 3 support for behavioral or social issues) to 4.27 (SD = 0.86, Tier 2 support for academic issues).

Core Features of Monitoring and Data-Informed Decision-Making Efforts

Sixty-five percent or more respondents reported a high level of implementation (ratings 4 or 5) for these features. Mean scores ranged from a low of 3.86 (SD = 1.04) for a method of gathering information from stakeholders on primary program to 4.67 (SD = 0.70) for academic screening of all students to benchmark progress (three times per year). For all eight features, mean scores far exceeded the scale midpoint.

Variation in Implementation: Implementation Stage and State

Results of a series of one-way ANOVAs contrasting ratings by school implementation stage (year 1, 2, 3, 4, and 6, respectively) on Tier 1, Tiers 2 and 3, and monitoring and data-informed decision-making features suggested relatively high and consistent implementation of most features during the first 4 years of implementation. For most Tier 1 features, implementation was lower for schools in the 6th year (range: 3.58–4.70) relative to schools in earlier stages of implementation (range: 3.94–4.73), but still well above the scale midpoint. In terms of Tier 2 and Tier 3 features, there were no statistically significant differences with respect to academic supports. However, there were significant distinctions in implementation of Tier 2 and Tier 3 interventions for behavioral issues, with respondents in the 2nd year of implementation reporting higher level of implementation relative to all other years for Tier 2 behavioral supports. For monitoring and data-informed decision-making efforts, behavior screening of all students to monitor progress three times a year was implemented with less fidelity in year 1 relative to years 2, 3, and 6. However, the mean score for year 1 implementers was still high at 4.20 (SD = 1.15), with no statistically significant differences in mean scores between years 2 through 6 with respect to behavior screening. For five monitoring and decision making items, year 6 implementers reported statistically significantly lower implementation than those in earlier implementation stages, potentially suggesting waning of core features.

Results of a series of one-way ANOVAs contrasting ratings by state across three geographic regions (West, Midwest, and Northeast) on Tier 1, Tiers 2 and 3, and monitoring and data-informed decision-making features also indicated relatively high and consistent implementation of most features across states. For most Tier 1 features there were no statistically significantly differences between states, with the exception of two items: differentiated instruction for academic tasks and monthly instruction in the social skills curriculum, with Midwest respondents indicating higher levels of implementation (see Supplemental Table S3). In terms of Tier 2 and Tier 3 features, there were statistically significant differences on all four features, again suggesting higher levels of implementation in the Midwest. It is noteworthy the mean scores were well-above the scale midpoint for all three states. For monitoring and data-informed decision-making efforts, this was the area with the most distinctions between states, in particular with respect to conducting behavior screenings which showed distinctions between all geographic states.

Implementation and Desire for Professional Development

Of the 19 common educational practices and supports examined (see Table 3), 50% or more respondents indicated a high level of implementation (ratings of 4 or 5) for 10 items, with mean scores as follows: small-group reading instruction (M = 4.59, SD = 0.64), behavior intervention plans (BIP; M = 3.81, SD = 1.01), increasing behavior-specific praise (BSP; M = 4.22, SD = 0.82), increasing opportunities-to-respond (OTR; M = 3.87, SD = 0.92), Check-in/ Check-out (CICO; M = 3.65, SD = 1.09), inclusive supports (M = 3.74, SD = 0.98), incorporating choice and preferred activities into instruction (M = 3.69, SD = 0.94), bullying prevention (M = 3.75, SD = 1.04), deescalation techniques (M = 3.67, SD = 1.00), and technology in the classroom (M = 4.28, SD = 0.81). Only two educational practices and supports had reported implementation averages below the scale midpoint: small-group self-determination instruction (M = 2.89, SD = 1.28) and peer-mediated support strategies (M = 2.76, SD = 1.22). Table 3 also includes mean levels of implementation for schools at different stages of implementation.

In terms of the desire for professional learning across educational practices (see Table 4), 50% or more indicated high desire (ratings of 4 or 5) for all but five items: small-group self-determination instruction, test-taking strategies instruction, functional behavioral assessment (FBA), BSP, and CICO. All mean scores were above the scale midpoint, suggesting interest in professional learning for all practices, with particular interest in deescalation techniques (M = 4.12, SD = 0.97) and strategies for internalizing behaviors (M = 3.94, SD = 0.98).

Relation between Implementation and Desire for Professional Development

Results indicated statistically significant correlations between educator ratings of currently implemented practices and desire for support for most (i.e., 13/19) educational practices and supports (see Table 5). Significant correlations were in the low-to-moderate range: 0.12 (incorporating choice and preferred activities into instruction) to 0.33 (small-group self-determination instruction and providing 1:1 reading or academic instruction).

Differences in Views across Implementation Stages

We conducted a series of one-way ANOVAs contrasting educator ratings at schools in various years of implementation (years 1, 2, 3, 4, and 6) examining (1) the extent to which they were implementing educational practices and supports that might be implemented as part of Ci3T models (Table 3) and (2) desire for additional support in addressing each of the 19 educational practices (e.g., training, coaching, print or web-based resources; Table 4). Results indicated statistically significant differences between participants from schools at different stages of implementation for the following: small-group social skills instruction F(4, 647) = 3.76, p = 0.005, R² = .023 (6 < 2, 4), self-monitoring strategy instruction, F(4, 640) = 3.94, p = 0.004, R² = .02 (6 < 2, 4); test-taking strategy instruction, F(4, 643) = 5.30, p = 0.0003, R² = .03 (6 < 1, 4); peer-mediated support strategies, F(4, 639) = 3.77, p = 0.005, R² = .02 (6 < 4); FBA, F(4, 630) = 7.85, p < 0.0001, R² = .05 (6 < 2, 4); behavior intervention plans (BIP), F(4, 644) = 2.75, p = 0.028, R² = .02 (6 < 2); providing 1:1 reading or academic instruction, F(4, 641) = 2.56, p = 0.04, R² = .02 (NS); increasing BSP, F(4, 643) = 2.89, p = 0.02, R² = .02 (6 < 2); CICO, F(4, 639) = 4.08, p = 0.003, R² = .02 (6 < 2); inclusive supports, F(4, 638) = 6.93, p < 0.0001, R² = .04 (6 < 2, 3; 2 > 1, 4); bullying prevention, F(4, 642) = 4.38 p = 0.002, R² = .03 (6 < 1, 2, 4); strategies for internalizing behaviors, F(4, 640) = 5.86, p = 0.0001, R² = .04 (6 < 2, 4); and deescalation techniques, F(4, 644) = 7.98, p < 0.0001, R² = .05 (6 < 2, 4; 2 > 1, 3). Multiple comparisons suggested implementation was reportedly lower for educators in schools in year six of implementation and highest for inclusive supports and deescalation techniques for those in their second year of implementation. Results also indicated few statistically significant differences in desire for professional learning between participants at schools in different stages of implementation: FBA, F(4, 615) = 2.92, p = 0.02, R² = .02 (6 < 4); strategies for internalizing behaviors, F(4, 625) = 2.46, p = 0.04, R² = .02 (NS); and technology in the classroom, F(4, 631) = 2.89, p = 0.02, R² = .02 (6 < 3), with all mean scores above the scale midpoint.

Next, we conducted a similar series of one-way ANOVAs contrasting educator ratings at schools between three states (Washington, Kansas, and Vermont) examining (1) the extent to which they were implementing educational practices and supports that might be implemented as part of Ci3T models and (2) desire for additional support with these practices (see Supplemental Table S4 and S5). Results indicated statistically significant differences between states as follows: small-group social skills instruction, F(2, 649) = 12.19, p < 0.0001, R² = .04 (KS > WA, VT); small-group reading instruction, F(2, 651) = 5.05, p = 0.007, R² = .02 (WA > VT); small-group self-determination instruction, F(2, 637) = 3.68, p = 0.03, R² = .01 (KS > VT); peer-mediated support strategies, F(2, 641) = 4.04, p = 0.02, R² = .01 (KS > VT); BIP, F(2, 646) = 16.84, p < 0.0001, R² = .05 (1, WA > VT); BSP, F(2, 645) = 4.34, p = 0.01, R² = .01 (KS > VT); increasing OTR, F(2, 639) = 4.14, p = 0.02, R² = .01 (NS); CICO, F(2, 641) = 15.02, p < 0.0001, R² = .04 (KS, WA > VT); inclusive supports, F(2, 640) = 18.64, p < 0.0001, R² = .06 (KS > WA, VT); incorporating choice and preferred activities into instruction, F(2, 644) = 8.27, p = 0.0003, R² = .03 (KS > WA, VT); strategies for internalizing behaviors, F(2, 642) = 3.68, p = 0.03, R² = .01 (NS); deescalation techniques, F(2, 646) = 7.16, p = 0.001, R² = .02 (KS > WA, VT); and technology in the classroom, F(2, 651) = 5.68, p = 0.004, R² = .02 (KS > WA). There was not a distinct pattern for differences; however, for 10 education practices and supports, implementation was reportedly higher in the Midwest. Results also indicated no statistically significant differences in desired professional learning between participants at schools in different geographic locales with the exception of FBA, F(2, 617) = 3.45, p = 0.03, R² = .01 (VT > KS), which was prioritized by participants in the Northeast. The desire for professional learning was above the scale median for all practices.

Priorities for Professional Development

Respondents rated their top three areas for professional learning in the coming year. Approximately 20% of respondents selected deescalation techniques and 11% selected small-group social skills instruction as their top priority. For the second priority, 16% of respondents selected deescalation techniques and 11% selected strategies for internalizing behaviors (e.g., cognitive restructuring). For the third priority, again deescalation techniques (12%) and strategies for internalizing behaviors (9%) were most valued.

Preferences for Professional Development Avenues

When providing input on their preferred avenues for professional development to learn about Tier 1, 2, and 3 supports within Ci3T models, the most popular endorsements were participating in in-district, during-school workshops (M = 4.04, SD = 1.08; see Table 6), courses for college credit on-line; (M = 3.61, SD = 1.16), and teacher collaboratives/networks (M = 3.44, SD = 1.05). A number of other avenues were rated well-above the scale midpoint. In-district weekend workshops was by far the least preferred avenue (M =1.82, SD = 1.04).

Differences in Preferred Avenues

Results of a series of one-way ANOVAs contrasting educator ratings at schools in various years of implementation (years 1, 2, 3, 4, and 6) indicated several similar preferences, based on the assumption that these options were actually available. There were statistically significant differences for out-of-district workshops, F(4, 641) = 3.07, p = 0.02, R² = .02 (6 < 2, 4); courses for college credit (on-line), F(4, 642) = 5.03, p = 0.001, R² = .03 (1 < 4); courses for college credit (on-campus), F(4, 640) = 4.83, p = 0.001, R² = .03 (1, 6 < 4); state conferences, F(4, 640) = 8.47, p < 0.0001, R² = .05 (1, 2, 3, 4 < 6); national conferences, F(4, 643) = 3.32, p < 0.01, R² = .02 (6 <1, 4); and webinars, F(4, 641) = 4.42, p = 0.002, R² = .03 (6 <4).

Results also indicated statistically significant differences in desired professional learning between participants in schools in different states: in-district after-school workshops, F(2, 645) = 11.39, p < 0.0001, R² = .03 (WA > KS); out-of-district workshops, F(2, 643) = 3.47, p = 0.03, R² = .01 (WA > KS); summer institutes (week long), F(2, 646) = 3.35, p < 0.04, R² = .01 (NS); courses for college credit (on-line), F(2, 644) = 11.65, p < 0.0001, R² = .03 (VT > KS, WA); courses for college credit (on-campus), F(2, 642) = 13.32, p < 0.0001, R² = .04 (VT > KS > WA); state conferences, F(2, 642) = 4.19, p = 0.02, R² = .01 (WA > KS); webinars, F(2, 643) = 4.44, p = 0.01, R² = .01 (WA > KS); and teacher study groups or “learning circles,” F(2, 645) = 4.68, p = 0.01, R² = .01 (VT > KS).

Discussion

Professional learning continues to be an important component of implementing and sustaining complex, integrated systems such as Ci3T. As part of this preregistered study funded by IES, to better understand how to provide enhanced professional learning for integrated tiered systems we invited educators from 27 elementary schools in various stages of Ci3T implementation representing five districts from three states. In Ci3T models, on-going data-informed professional learning is a hallmark characteristic of the implementation process. Given the complexities of the teaching enterprise and often-limited resources, it is particularly important that educators be provided a full scope of professional learning resources in terms of content and via multiple avenues to meet individualized, on-going, changing professional learning needs (Lane et al., 2015; Oakes et al., 2021). Part of accomplishing this lofty goal is understanding patterns of implementation and professional learning priorities across initial implementation, full operation, innovation, and sustained practice (Fixsen et al., 2009). As such, we sought current implementation of core Ci3T features and common educational practices and supports from educators, as well as their professional learning content and avenue preferences.

Implementation: Ci3T Core Features

We examined the degree to which core Ci3T components were implemented across three tiers of prevention. Similar to Oakes et al. (2021), most educators indicated high implementation levels across Tier 1 features, with the highest ratings identified for having school-wide expectations for all key settings. It is interesting that across this study and its two predecessors, educators reported higher levels of implementing monthly instruction using the school-wide social skills curriculum than monthly instruction on school-wide expectations (Lane et al., 2015; Oakes et al., 2021). Based on this finding, it may be a priority to create professional learning content focused on use of integrated lesson planning, which involves inclusion of academic (tied to core instruction), behavioral (tied to school-wide expectations), and social (tied to validated social skills curriculum) objectives within a lesson or unit (Lane et al., 2020c). Likewise, professional learning may be needed to support school leaders in creating structures (e.g., procedural integrity checklist, instructional schedule) to facilitate instruction on school-wide expectations throughout the year (Oakes et al., 2021).

Across Tier 2 and Tier 3 core features, respondents in the present study rated implementation of social-emotional and behavioral Tier 2 and 3 supports lower than Tier 2 and Tier 3 academic supports. Yet mean implementation ratings of social-emotional and behavioral interventions were nevertheless consistently above the scale midpoint. These findings were consistent with previous studies. Across studies, patterns in educators’ implementation of Tier 2 and 3 supports highlight areas for both celebration and refinement. It is promising that there appears to be an increasing emphasis on academic, behavioral, and social-emotional Tier 2 and 3 supports in the more recent studies (i.e., Oakes et al., 2021, and the present study). Yet based on consistently lower ratings of behavioral and social-emotional supports relative to academic supports, there appears to be an opportunity to use professional learning to assist educators in implementing these supports for students who may need behavioral and social-emotional supports beyond what is provided at Tier 1, in particular in response to supporting students during COVID-19 (Chafouleas et al., 2020).

With respect to core Ci3T features related to monitoring and data-informed decision making, findings again highlight successes and areas to target in future professional learning offerings within integrated tiered systems such as Ci3T. For instance, implementation of monitoring and data-informed decision-making features were higher in this three-geographic region sample relative to findings reported by Oakes et al. (2021). Across regions and range of implementation levels, the lowest implemented feature was a method of gathering information from stakeholders on primary programs (M = 3.86), whereas the highest implemented feature was academic screening to benchmark progress (M = 4.67). These findings were identical to patterns found by Oakes et al. (2021), with the exception mean values for these items were slightly higher in the present sample.

The consistent, relatively lower scores for collecting information from stakeholders were initially surprising given the emphasis on data-informed decision making within Ci3T. Ci3T implementation involving collection of Tier 1 programmatic data (e.g., treatment integrity, social validity). Each of the schools involved in the present study participated in collection of treatment integrity and social validity data two times per year (i.e., fall, spring) to ensure availability of feedback on fidelity of the primary (Tier 1) plan. Findings suggested professional learning is needed to provide clarity around these systems-level procedures for monitoring, including how, when, and why these measures are administered, and how these data are used to inform decisions at the school, classroom, and student levels. Empowering educators to collect and use these data, as well as gathering feedback from other stakeholders (e.g., families) to support work within integrated, tiered systems, is an important next step.

Regarding differences between ratings at various stages of implementation, we observed statistically significantly lower ratings for educators in schools in the 6th year of implementation relative to schools in earlier stages. We also noted that, despite being lower in schools in the 6th year of implementation, ratings across phases were consistently well above the scale midpoint. Differences in the most experienced schools may reflect innovations occurring as practices become adapted to fit within the unique context of a school or district’s organizational structure (Taxman & Belenko, 2012). Another possibility is certain core features have waned over time, potentially due to turnover in school personnel, initiative fatigue, a gradual release of formalized university-led supports, or the need for professional learning materials aimed specifically at schools in the sustainability phase (e.g., refresher trainings, formalized onboarding for new faculty and staff). An alternative explanation is that expectations may have shifted, and these ratings may indicate there is a desire or need for more advanced systems and training to facilitate various features of Ci3T such using data to connect students to Tier 2 and 3 supports and using schoolwide data to monitor treatment integrity and social validity, as well as to inform professional learning offerings.

Implementation: Common Educational Practices and Supports

In terms of educational practices and supports implemented, results were highly comparable to previous studies with small-group reading instruction, BIP, BSP, increased OTR, incorporating choice and preferred activities into instruction, bullying prevention, and technology in the classroom all reportedly implemented at a high level. In addition, the current sample of respondents also reported high levels of implementation of Check-in/Check-out, inclusive supports, and deescalation techniques.

Although this sample reported higher levels of implementation of most educational practices and supports relative to Oakes et al. (2021), small-group self-determination instruction and peer-mediated supports remained implemented at relatively low levels. Given the importance of self-determined behaviors and social competencies with peers, these will be important areas for future inquiry to support implementation as part of regular school practices. Given the integrated nature of Ci3T, we encourage research teams to explore efficient methods of developing intervention techniques that will carefully attend to generalizing self-determination and social competencies throughout and beyond the school day. For example, as with social skills instruction, it would be wise to communicate with educators the specific skills taught in the small-group setting so teachers and other adults could be watching for use or application of specific skills taught. When recognized, adults can provide BSP (which was implemented at a very high level in this sample, M = 4.22) to acknowledge students for generalized use of targeted skills beyond small group settings (Common et al., 2019).

In terms of educator desire for additional professional learning around common educational practices and supports, respondents indicated a high desire for most practices. Deescalation techniques, strategies for internalizing behaviors, small-group reading instruction, bullying prevention, and technology in the classroom were rated highest, whereas small-group self-determination instruction, test-taking strategy instruction, FBA, BSP, and CICO were rated lowest in terms of desirability. Similar to Oakes et al. (2021), most educators indicated they were open to professional learning, as evidenced by all mean scores falling above the scale midpoint. There was also a positive relation between educator ratings of currently implemented practices and desire for support for most practices and support (13/19), similar to administrators’ views in Lane et al. (2015) and divergent from Oakes et al. (2021), which showed only significant positive relations for four practices. Although significant, correlations were small across studies, though findings suggested educators are interested in increasing knowledge of interventions of which they already have some level of familiarity.

Regarding differences between educators’ ratings of common educational practices and supports across implementation year, patterns suggest implementation was lower for educators working within the most experienced schools (i.e., year 6). In terms of differences between educators across states, implementation tended to be higher in districts in Kansas. This may be due to the close proximity to a large university prioritizing research on evidence-based practices and inclusive supports. In contrast, most practices were not rated differently with regard to a desire for professional learning by school’s implementation year or locale. Notable differences in implementation of educational practices and desire for additional support highlighted the need to assess professional learning through data-informed processes to individualize content and avenues of professional learning.

Professional Learning Preferences

Clear priorities were related to students’ behavioral and social well-being: deescalation techniques (Colvin & Scott, 2015), social skills instruction (Common et al., 2019), and strategies for internalizing behaviors (Vannest et al., 2015). Deescalation techniques were also the top-rated professional learning priority reported by Oakes et al. (2021). These results are reaffirming in that educators across all three studies conducted to date are implementing educational practices and supports for the whole child (Table 3) and have a desire for professional learning related to supporting students’ behavioral and social well-being, in particular at Tiers 2 and 3 (Table 4).

Educators most commonly preferred in-district during-school workshops, course for college credit (on-line), and teacher collaboratives/ networks as avenues for professional learning, and were least likely to prefer in-district weekend workshops being least preferred. The most and least preferred avenues were similar with Lane et al. (2015) and Oakes et al. (2021). Other highly preferred avenues in the current study included webinars (i.e., web-based presentations), brief “good practice” guides, multimedia presentations, one-to-one coaching or mentoring, and web-based professional learning modules. Further, educators across years of implementation reported similar preferences for professional development avenues, with the exception of out-of-district workshops, course for college credit, conferences, and webinars with educators from schools in their 4th-year rating high preferences in comparison to those from school buildings in their 1st or 6th year. On the other hand, schools in their 6th-year preferred national conferences more so than educators from schools in all other implementation years. This may suggest professional learning needs shift as schools change from implementation phases associated with innovating and sustaining practices over time (Fixsen et al., 2005; Taxman & Belenko, 2012). In addition, findings highlight the desirability of technology-based professional learning experiences (e.g., webinars, web-based professional learning modules) as well as opportunities to interact with others (e.g., collaboratives, coaching).

Educators across states were also similar in their reported preferences for professional development avenues, with the exception of in-district, after-school workshop, out-of-district workshops, course for college credit (on-line and on-campus), state conferences, webinars, and teacher study groups or “learning circles.” Washington showed greater interest in in-district, after-school workshop and webinars and less interest in course for college credit (on-campus) in comparison to Kansas. Vermont showed greater interest in teacher study groups or “learning circles” in comparison to Kansas and greater interest in course for college credit (on-line) in comparison to Kansas and Washington. Districts in Kansas were in the same state as the lead University of Project ENHANCE; proximity to the university suggests there may be differences in accessibility to professional learning resources and research opportunities for districts near and far from partnering universities. This may be an important consideration in interpreting these comparisons.

Limitations and Future Directions

We encourage readers to interpret results relative to the following considerations. First, as noted in earlier published studies (Lane et al., 2015; Oakes et al., 2021), this particular measure does not include operational definitions of each term. We refrained from this level of precision due to potential concerns with extending the time that would be necessary for respondents to complete an already lengthy survey. Therefore, educators' familiarity with these concepts may affect ratings of features, practices, and professional learning avenues. For example, educators indicated the feature least implemented for monitoring and decision making was a method for gathering information from stakeholders on the primary program. It should be noted that each school included in this study had procedures in place to collect treatment integrity and social validity data from faculty and staff, which directly related to this item. Yet individual educators may not have considered or been aware of these data when responding, emphasizing the need for supporting school leaders in collecting, using, and sharing programmatic data (e.g., treatment integrity and social validity) as part of efforts to implement Ci3T. This example also exemplifies the importance of school leaders providing professional learning, context (e.g., the why), and transparency (e.g., sharing data) as part of implementing Ci3T and other tiered systems. Without purposeful information sharing and instruction, some stakeholders may have perceived features (e.g., data collection) as disconnected from the system.

Second, low levels of awareness of some research-based practices, in particular those commonly associated with special education rather than general education, may have affected ratings of implementation and desirability of professional learning for some strategies and programs (e.g., self-determination instruction, FBA). This emphasizes the need for strong researcher–practitioner partnerships to share information about research- and evidence-based practices that can be used as intensive interventions regardless of students’ eligibility, as well as collaboration between general and special educators to increase knowledge and access to these practices across the tiers. Future inquiry may involve assessing whether differences in implementation and professional learning preferences occur among school staff with various roles, and how those patterns shift over time.

Next, in this replication and extension study we conducted numerous statistical analyses with significant testing to answer the eight preregistered questions. Type I error is a possibility when conducting many significance tests. Therefore, results should be interpreted with caution. In addition, inferential statistics used in the present study indicated whether differences occurred between groups (i.e., implementation stage, state). We did not, however, conduct additional testing to examine the magnitude of differences. Although educator responses from schools in the 6th year of implementation indicated lower implementation across several features, responses were consistently above the scale midpoint, suggesting these features were still largely in place even if not rated as high as in schools in earlier stages. Future inquiry may seek to quantify the magnitude of statistically significant differences to assess the extent to which they are practically significant.

In addition, we encourage caution in generalizing findings. We did not use random sampling to collect responses, because the purpose of this study was to assess professional learning needs to create resources to support implementation. Moreover, one district previously participated in a similar professional learning survey as part of an IES researcher–practitioner partnership grant (Oakes et al., 2021), collecting data from K–12 educators 3 years prior. It is possible some respondents in the current sample provided input from the previous study. However, this study focused exclusively on the elementary experience. Given the focus on needs assessment to inform professional learning, with attention to professional learning related to implementation stage or region may be idiosyncratic to the districts and schools from which data were collected. Nevertheless, overall findings provide a cross sectional view of professional learning needs from a diverse sample in terms of region and implementation stage. Further studies may replicate these data collection procedures and analyses using a broader and randomly selected sample, as was the case in Lane et al. (2015), to investigate more generalizable findings.

Lastly, the present study assessed participant responses to each professional learning method individually rather than asking for preferences relative to one another. Future inquiry may advance findings by asking participants to rank order their preferences (e.g., select your top three preferred professional learning preferences). This added specificity may assist researchers as well as school and district leaders in using data-informed decision making to not only determine content, but also to prioritize how the content is provided. In addition, gathering data on the extent to which educators actually participated in each type of professional learning avenue (e.g., “Have you participated in out-of-district workshops in the past 3 years?”) may be beneficial. For example, gathering these additional data on educator preferences and patterns of use may reveal untapped areas in which professional learning innovation can occur. Avenues with high interest but low usage may benefit from further development. Or, combinations of preferred avenues may be explored, such as developing approaches to integrate social and technological elements of professional learning (e.g., teacher collaborative networks linked through social media; earning of micro-credentials that may be used to earn college credits or as skills to list on a résumé). Future inquiry may assess the extent to which data-informed, synergistic approaches can be leveraged to promote engagement and sustained behavioral change (e.g., adopting evidence-based practices).

Summary

Lessons learned over the last 2 decades of Ci3T inquiry (Lane et al., 2020c; Lane & Menzies, 2003) have led to the development and refinement of the Ci3T model. As part of the model’s emphasis on data-informed decision making, Project ENHANCE affords us the opportunity to develop professional learning resources to support Ci3T implementation at scale. Overall, more than half of educators in this study indicated high levels of implementation across core features of Ci3T as well as research-based strategies, practices, and programs. For many of these common practices there were significant relations between implementation and desire for support. One strength of this replication and extension study is the exploration of differences across stages of implementation as well as geographic regions across three states, as well as incorporation of open science practices in our study’s procedures to build confidence in findings presented (Cook et al., 2016). This information will be used to inform professional learning materials to facilitate a wide range of professional learning avenues, including resources to support workshops, web-based learning, learning groups, and coaching—including materials that could be embedded in university micro-credentials or badges. Our commitment to professional learning to promote sustained implementation of socially valid practices is guided by a commitment to empower school systems with the skills and resources to move from initial implementation to sustainability (Fixsen et al., 2005). Research in implementation of tiered systems suggests it can take from 3 to 5 years for schools to reach a high degree of fidelity (McIntosh et al., 2013). Results from this study, in addition to the professional learning needs around systematic screening (Briesch et al., 2021) and leadership (Royer et al., 2021) within integrated tiered systems will be used to inform future professional learning offerings that will be developed to enhance and sustain Ci3T implementation.