As students transition from elementary to middle school, reading comprehension becomes central to academic achievement as well as college and career readiness (ACT, 2017; Cain & Oakhill, 2006). Unfortunately, merely 36% of students in the USA reach proficient status in reading by the end of middle school (National Center for Education Statistics, 2018). Moreover, even with evidence-based reading instruction, not all students will respond adequately (McMaster et al., 2005; Fuchs et al., 2014). In this paper, we investigated factors we hypothesized would be associated with differential effectiveness of a reading intervention for middle school students with reading difficulties, including student characteristics and the type of instructional delivery system.

Inferential reading comprehension

A complex behavior such as reading comprehension draws on the reader’s cognitive abilities (e.g., attention and working memory), motivations, and their skills and knowledge, including word reading, vocabulary, world knowledge, inference-making, and comprehension monitoring (Cain et al., 2004; RAND Reading Study Group, 2002). Struggling readers often display deficits in one or more of these areas (Cain & Oakhill, 2006). Inference-making, the ability to infer information that is not explicitly stated in the text, is a crucial component of reading comprehension as reflected in several models of reading such as the Direct and Inferential Mediation (DIME) model (Cromley & Azevedo, 2007; Cromley et al., 2010), Structure Building Framework (Gernsbacher, 1991), Reading Systems Framework (Perfetti & Stafura, 2014), and the Landscape Model (van den Broek et al., 1999). Difficulties in making inferences to connect parts of the text and to integrate background knowledge are associated with poorer comprehension (Cain et al., 2001; Cain & Oakhill, 1999), and, as reviewed below, inference-making is considered to be a pressure point (Perfetti & Stafura, 2014) for reading comprehension in the secondary school years.

Among students in grades 7 to 12, after controlling for shared method variance, inference-making is the strongest direct predictor of reading comprehension compared to word reading, vocabulary, background knowledge, and reading strategies (Ahmed et al., 2016). Specifically, adolescents who struggle with reading comprehension have difficulty making inferences that require readers to connect parts of the text as well as those that require them to integrate necessary background knowledge to understand the text (Barnes et al., 2015; Barth et al., 2015). Additionally, inference-making mediates the relation between reading comprehension and other comprehension-related factors such as vocabulary and background knowledge (Ahmed et al., 2016; Silva & Cain, 2015). Therefore, a reading comprehension intervention that targets inference-making may be particularly beneficial for struggling readers in middle school.

A meta-analysis by Elleman (2017) observed that reading interventions that focus on inference-making increase inferential reading comprehension of students across grade levels and also improve literal comprehension for struggling readers. However, not all students will respond adequately to instruction (McMaster et al., 2005; Fuchs et al., 2014), including inferential reading comprehension interventions. In attempts to understand why students respond differently to reading interventions, researchers have increasingly utilized moderation analysis to statistically identify characteristics associated with differential response to reading interventions (e.g., Clemens et al., 2019; Miciak et al., 2014, Vaughn et al., 2019). There are many potential characteristics that could moderate students’ response to intervention. The current study examined the extent to which student characteristics that affect attention as well as student language status moderated the effects of two versions of an intervention designed to improve inference-making when reading.

Student characteristics that affect attention during reading

Attention is a core component of cognition (Chun et al., 2011) and is crucial to learning and academic achievement (Duncan et al., 2007; Rabiner et al., 2016a). For instance, a student cannot develop their reading skills and acquire knowledge if they are not attending to instructional content (Rabiner et al., 2016b). Further, poor attention can negatively influence students’ long-term academic outcomes in reading and math and can increase risk for not graduating from high school (Rabiner et al., 2016b). Attention is particularly important for reading comprehension (Smallwood et al., 2008), because the processes of maintaining a coherent representation of text, making connections within and across text, connecting to prior knowledge, and inferring meaning requires sustained attention while reading.

The current study tested the effects of three factors thought to influence the deployment of attention during reading on response to a reading comprehension intervention. The attention-related factors studied were the following: mind-wandering, which refers to a lack of internal control of attention (based on the Chun et al. (2011) taxonomy of attention); anxiety, where pervasive thoughts of worry can negatively affect attention; and mindset, which is the belief of the malleability of one’s own learning and can affect attention by altering students’ academic behavior. The hypothesized relations of each of these to reading are reviewed below.

Mind-wandering

Good readers not only attend to what they are reading, but they also maintain attention by regulating their thoughts (Mooneyham & Schooler, 2013). Mind-wandering during reading refers to the shift of attention away from the reading content to internal or unrelated thoughts (Smallwood et al., 2007). It is associated with poorer reading comprehension scores (Mrazek et al., 2013; Schooler et al., 2005) and with decreasing the reader’s ability to detect when an inference is necessary (Mooneyham & Schooler, 2013). Mind-wandering is also related to working memory, such that greater levels of mind-wandering are associated with lower working memory capacity (Unsworth & McMillan, 2013), which itself is related to both inference-making (Currie & Cain, 2015) and to reading comprehension (Peng et al., 2018). Mind-wandering can be measured using a reading probe procedure which involves intermittently interrupting the reader during reading to ask them what they were thinking about at that point in the text (Mooneyham & Schooler, 2013) or with a mind-wandering questionnaire (Mrazek et al., 2013). Distinctive from the reading probe technique that evaluates mind-wandering exclusive to reading, the mind-wandering questionnaire captures the broader, trait-like nature of mind-wandering based on the assumption that a tendency to mind wander interferes with a variety of tasks, including reading. Accordingly, mind-wandering questionnaires have been used to evaluate the relation between broad, trait-based mind-wandering and reading comprehension (e.g., Mrazek et al., 2013; Soemer et al., 2019) and were used in the current study.

Anxiety

Key features of anxiety include pervasive thoughts of fear and worry. A central symptom of anxiety is being preoccupied with personal thoughts or emotions both of which can lead to mind wandering (Smallwood et al., 2007; Unsworth & McMillan, 2013). Similar to mind-wandering, trait-based anxiety can interfere with inference-making and reading comprehension (Blicher et al., 2017; Gygax et al., 2007). In addition, higher levels of anxiety are associated with lower working memory capacity, which can interfere with attention to, processing of, and retention of the text (Albano et al., 2003). The relation between anxiety and low achievement is also stronger among adolescents compared to younger students (Riglin et al., 2014), which makes it of particular relevance in terms of studying it as a potential moderator of outcomes in reading interventions for middle school students. Although mind-wandering can be a chief indicator of anxiety, mind-wandering does not necessarily imply underlying anxiety. Hence, the current study measured anxiety and mind-wandering as separate constructs.

Mindset

Students’ mindset can also influence attention and comprehension by altering their academic behavior. According to the Mindset theory (Dweck et al., 1995), students with fixed mindsets believe that their abilities, skill difficulties, and other attributes related to learning are stable and unchangeable and so they may feel that expending effort on academic tasks is futile. In contrast, those with growth mindsets believe that achievement and academic success are malleable and depend on effort and determination, which require task-related sustained attention (Sisk et al., 2018). Meta-analyses have shown that overall growth mindset is related to higher reading outcomes for both children (r = 0.19) and adolescents (r = 0.15; Sisk et al., 2018). General mindset is particularly related to reading comprehension in children with weaker reading comprehension skills (Petscher et al., 2017). Specifically, reading comprehension was more strongly associated with general mindset for students with weaker reading comprehension skills, whereas reading comprehension was more strongly associated with reading-specific mindset for students with stronger reading comprehension skills (Petscher et al., 2017). Because the current study examined struggling readers, general mindset was measured.

Language status and reading

The national prevalence of English Learners or ELs is 9.5% and is projected to grow as the USA becomes a more diverse country (National Center for Education Statistics, 2018). Not only do ELs tend to have lower reading proficiency levels compared to monolingual students (National Center for Education Statistics, 2018), they also do not respond as well to reading instruction compared to monolingual students (Hall et al., 2016). However, notably, most of the intervention studies conducted with middle school ELs have focused on vocabulary instruction (e.g., Mancilla-Martinez et al., 2011; Lesaux et al., 2010; Lesaux et al., 2014, also see Hall et al., 2016 for review). No studies have assessed whether an inference-making intervention has similar effects for non-ELs vs. ELs.

The obvious source of the reading achievement disparity between ELs and monolingual students is the need to learn to decode the English writing system and develop knowledge of vocabulary, syntax, and other linguistic skills that mediate the comprehension of written language. However, poor reading achievement of ELs may also be partly associated with low academic engagement, which may be associated with increased anxiety and fixed mindset around academic tasks. For instance, the cultural discontinuity model suggests that if a student does not feel like their culture is valued in the school setting, they may experience anxiety associated with academic tasks (Bingham & Okagaki, 2012). Furthermore, the cultural ecological model indicates that minority students who experience oppression and discrimination may develop a fixed mindset that can impact achievement (Bingham & Okagaki, 2012). Thus, language status may have effects on intervention response through both language and socio-cultural factors, suggesting that the potential effect of language status on response to intervention should be considered in the context of the student characteristics discussed above.

Instructional delivery systems

In addition to student characteristics that may affect the deployment of attention and language status, instructional delivery systems (i.e., teacher-led vs. computer-delivered instruction) can also impact student learning and may interact with student characteristics and language status. For instance, a teacher may utilize behavioral management strategies or outline expectations to decrease anxiety and mind-wandering, while increasing motivation (Fredricks et al., 2004; Guthrie et al., 2012). In contrast, some computer programs encourage student engagement by offering rich visuals, multimedia, and game-like rewards (Guthrie et al., 2012). Computer programs can also provide one-on-one, self-paced instruction which may reduce students’ performance anxiety compared to instruction in whole-class settings (Hattie, 2017). Some research has examined attention in the context of computer-delivered instruction (for review, see Tobias et al., 2011); however, there is limited empirical work that compares the influence of attention and attention-related factors on outcomes across different intervention delivery systems for the same intervention. Similarly, there is little research on the effects of computer-delivered reading comprehension interventions for ELs (e.g., Barber et al., 2018; Macaruso & Rodman, 2011) and we are not aware of any studies that directly compare the effects of teacher-delivered vs. computer-delivered reading comprehension interventions for ELs.

Study purpose

The purpose of this study was to (a) determine if students’ mind-wandering, anxiety, mindset, and language status were associated with differential response to an inferential reading comprehension intervention in struggling middle school readers and (b) examine whether these effects vary across instructional delivery systems. An extant data set from a larger randomized control trial (Barnes et al., 2020) was utilized to investigate the following research question: Do language status and pre-intervention levels of anxiety, mind-wandering, and mindset influence the effects of a computer-delivered or teacher-delivered inferential reading comprehension intervention in struggling middle school readers?

Based on the review of the literature, prior to data collection and analyses (but after the intervention had been developed), we hypothesized differential effects across intervention modalities (i.e., teacher-led, computer-delivered). Specifically, the teacher-led intervention utilized behavioral management strategies and was therefore expected to be more beneficial for students with high levels of mind-wandering. In contrast, the computer-led intervention was anticipated to be better for students with higher levels of anxiety because the absence of a teacher or peers in the intervention would mitigate students’ fear of failure and embarrassment about their reading difficulties. Mindset and language status were expected to moderate treatment effects; however, we did not predict differential effects across intervention delivery systems.

The current study extends the literature on the effects of reading comprehension interventions for middle school readers with or at-risk for reading disabilities by (1) including mind-wandering, general anxiety, mindset, and language status as multiple potential moderators in a single study to determine if they moderate treatment effects and (2) examining whether these factors operate differently depending on how the instruction is delivered (i.e., teacher-led, computer-delivered).

Method

This study utilized secondary data analyses with a subsample of students from a randomized controlled trial (Barnes et al., 2020) that examined the efficacy of an inferential reading comprehension intervention for middle school students. The subsample completed additional pre-intervention measures which were used to examine the research questions.

Participants and setting

The parent study included three middle schools in the southwest USA, which each served students between grades 6 to 8. Students included in the present analyses were housed in two of the middle schools. These two schools were selected given their proximity to the first author, who collected additional data on potential moderator variables to support the present analyses. During the year of the study, 75.3% of school 1 and 81.8% of school 2 were categorized as economically disadvantaged. The state education agency reported that across grade levels, about 33% of the students attending school 1 and 58% of students attending school 2 attained grade-level proficiency in reading, based on the state accountability literacy test.

Participants were recruited and randomly assigned to groups as part of the parent study. Students were recruited in the fall semester from school 1 and in the spring semester from school 2. To identify struggling readers, we asked school administrators to nominate students that were receiving supplemental support for reading or who had performed poorly on the state accountability exam in reading the previous school year. Consent forms were distributed to these students, and those that returned consent forms (n = 67) were included in the study. The average reading comprehension ability of the sample was below-average (SS = 88.20; 21st percentile), as assessed by Reading Comprehension subtest of the Wechsler Individual Achievement Test 3rd ed. (WIAT-III) at pretest.

As part of the parent study, a stratified randomization procedure was utilized to randomly assign students to one of three conditions: (1) teacher-led intervention, (2) computer-delivered intervention, or (3) school-designed “business-as-usual” (BaU) control. The randomization process was conducted for each school separately by an analyst that was blind to study condition. First, students were grouped into strata based on grade level and reading comprehension ability. Then, within each stratum, students were randomly assigned to each condition. This process reduced the likelihood of statistical differences of reading comprehension or grade level across the groups. Attrition was low; one student in the BaU group moved to another school district during the study and post-intervention scores could not be collected. Therefore, final analyses were conducted with 66 students. Demographic information is in Table 1. Groups did not significantly differ with regard to gender, grade level, ethnicity, race, special education status, or language status.

Table 1 Participant demographic data
Full size table

Experimental conditions

The inferential reading comprehension intervention was developed and evaluated as part of the parent project and is described comprehensively in Barnes et al., (2020). The primary characteristics of the intervention and study conditions are provided here.

Intervention

The inferential reading comprehension intervention consisted of 26 lessons, implemented three to four times a week. Each lesson was approximately 25 min, equating to about 650 min of instruction across 6 to 8 weeks. The intervention began with an introduction lesson that utilized various forms of media (e.g., videos, passages, text message exchange) to illustrate the importance of making inferences in communication and how common they are in everyday life. The remaining 25 lessons were divided into five modules, with five lessons in each module. The first three modules focused on text-connecting inferences (Cain & Oakhill, 1999; Perfetti & Stafura, 2014), including identification of pronoun references, making lexical connections across texts, and inferring word meaning from context. The fourth module concentrated on retrieving and integrating background knowledge, and the fifth module provided interleaved practice of the various types of inferences taught in the previous modules. Instructional practices such as interleaved practice and feedback based on cognitive and educational research were utilized throughout the intervention (Archer & Hughes, 2010; Butler et al., 2013; Dunlosky et al., 2013) as were procedures provided by the two instructional platforms (e.g., drag and drop, graphic organizers, and so forth). Although the intervention was not designed to explicitly address mind-wandering, anxiety, or mindset, elements within the intervention did provide encouragement and reinforce effort. For example, lessons were often introduced with student avatars that were struggling with reading who overcame their shortcomings by utilizing an inference-making instructional strategy.

There were similarities and inherent differences across the invention delivery platforms, which are contrasted in Table 2. The teacher-led version of the intervention was implemented in small groups (i.e., no more than six students) by a trained interventionist. In the computer-delivered version, students completed the program independently but an interventionist was present to address technical issues and encourage students to remain on task. The teacher-led instruction was scripted to correspond to the instructional content, passage format, and question presentation of the computer-delivered version. The average procedural fidelity (i.e., percentage of intervention steps implemented) for the teacher-led instruction was 96%.

Table 2 Intervention delivery systems
Full size table

The experimental conditions also differed in feedback and engagement strategies. For students assigned to the computer-delivered condition, affirmative or corrective feedback was delivered based on the accuracy of student’s individual responses and, because it was programmed, was consistently delivered the same way each time. For those assigned to teacher-led instruction, affirmative and corrective feedback was based on the response of a student or the group, with more detailed feedback given if at least one student in the group answered incorrectly. Consequently, it was possible that students assigned to the teacher-led instruction received more in-depth feedback on more items compared to those in the computer-delivered condition. The computer-delivered condition was also limited to a single corrective explanation for an incorrect response. The corrective explanation utilized in the program was based on the most frequent mistake made by students during the pilot phase of the parent study. In contrast, the teacher-led instruction provided interventionists with the flexibility to address students’ specific errors and, thus, at times, more individualized feedback may have been given compared to computer-delivered condition.

The intervention emphasized completion, rather than accuracy; however, there were also differences in motivation and student engagement strategies across the experimental conditions. Students received game-like trophies and badges for completing lessons in the computer-delivered platform, while interventionists monitored the session and encouraged students to remain on task as needed (e.g., not getting up from the computer, not staying an extended amount of time a single screen). In contrast, the teacher-led instruction incorporated behavior management strategies (e.g., verbal praise and encouragement) to maintain students’ attention. Interventionists also encouraged participation of each student in every session.

BaU control

The control comparison group was typical for what struggling middle school readers, who had performed poorly on the state accountability exam in reading generally receive. Students assigned to the BaU group remained in their school-designed reading intervention or elective class (e.g., art, music). Resource limitations did not permit direct observation of BaU instruction. However, after the study was completed, the schools’ three reading intervention teachers completed a survey in which they reported the use of specific programs or curricula, which included the Fountas & Pinnel Leveled Literacy Intervention, Read 180, and Achieve 3000. Teachers also reported the components of reading comprehension they taught, and the techniques or strategies used, which included teaching summarization skills, how to monitor comprehension during reading, and asking students questions about the text.

Measures

The assessment battery included measures of word reading fluency, inferential reading comprehension, and general reading comprehension as well as measures of student self-reported mind-wandering, anxiety, and mindset.

Test of Word Reading Efficiency 2nd ed., Sight Word Efficiency

Word reading fluency was assessed using the Sight Words Efficiency (SWE) subtest of the Test of Word Reading Efficiency 2nd ed. (TOWRE-2) (Torgesen et al., 2012). Students are instructed to read aloud from a list of words that increase in difficulty. The score reflects the number of words read correctly in 45 s. The test authors report an average alternative-form reliability estimate of 0.92 for the SWE subtest (Torgesen et al., 2012). SWE was administered at pretest only.

Connect-IT Inferential Reading Comprehension Assessment

The Connect-IT Inferential Reading Comprehension Assessment (CIRCA) (Clemens & Barnes, 2018) is a 42-item test aligned with the inference-making skills targeted within the intervention (i.e., pronoun referents, text-connecting inferences, inferring word meaning, knowledge-based inferences), but did not include passages or questions that were included in the intervention. Students read silently and completed the test independently on an untimed basis. The reliability and validity of the CIRCA were assessed in the larger parent study (Barnes et al., 2020). Correlations between the CIRCA and concurrent administration of the Reading Comprehension subtest of the Wechsler Individual Achievement Test and the Bridging Inference Task were 0.68 and 0.52, respectively. The coefficient alpha for the CIRCA was 0.88. The CIRCA was administered at pretest and posttest.

Bridging Inference Task

The Bridging Inference Task (Bridge-IT) (Barth et al., 2015) is an untimed measure that assesses text-connecting inferences. Students read passages silently and then determine if a concluding sentence makes sense, without being able to refer back to the passage. The scores on the Bridge-IT were analyzed separately for items that required students to make text-connecting inferences in back-to-back sentences (i.e., Bridge-IT Near) and for items that required students to make these types of inferences across greater text distances (i.e., Bridge-IT Far). The average internal consistency for middle school students on the computerized version was 0.86 for near items and 0.85 for far items (Barth et al., 2015). The alternative form reliability for elementary students on pencil and paper version was 0.73 across near and far items (Pike et al., 2013). The pencil and paper version of the test was administered at pretest and posttest.

Wechsler Individual Achievement Test 3rd ed., Reading Comprehension

Reading comprehension was assessed with the Reading Comprehension subtest of the Wechsler Individual Achievement Test 3rd ed. (WIAT-III) (Wechsler, 2009). In this subtest, students read passages silently and verbally answer open-ended literal and inferential questions presented orally by the examiner. The test authors reported an average test-retest reliability estimate of 0.83 for students in grades 6 through 8 for the Reading Comprehension subtest (Wechsler, 2009). The Reading Comprehension subtest was administered at pretest and posttest.

Pretest moderator variables

Students’ language status (i.e., whether or not they were identified as an English learner) was collected from the school district. Students were classified as English learners if they had not achieved adequate English proficiency, as determined by the students’ scores on a school-administered test from the prior school year. Self-reported measures of mind-wandering, anxiety, and mindset were administered pre- and post-intervention and the pretest scores were examined as potential moderators. Students completed these measures independently, with privacy barriers, but a researcher was available to answer questions or provide clarification.

Mind-Wandering Questionnaire

The Mind-Wandering Questionnaire (MWQ) (Mrazek et al., 2013) is a five-question self-report measure of the frequency of spontaneous mind-wandering during reading and other tasks. It uses a 6-point Likert scale and has a Cronbach’s alpha for internal consistency reliability of 0.85 (Mrazek et al., 2013). In adolescence, the MWQ is a significant predictor of mind-wandering measured by a probe procedure during reading (Mrazek et al., 2013) and is less invasive than a mind-wandering probe procedure.

Multidimensional Anxiety Scale for Children 2nd Edition

Anxiety was measured with the Multidimensional Anxiety Scale for Children 2nd Edition (MASC-2) (March & Benton, 2007). The MASC-2 is a self-report questionnaire used to assess generalized anxiety in children and adolescents. It consists of 50 questions written at a second-grade reading level, with a Cronbach’s alpha for internal consistency reliability of 0.92 and a test-retest reliability of 0.89 (March & Benton, 2007). The MASC-2 has been found to be a valid measure of anxiety in adolescents identified with learning disabilities (Thaler et al., 2010) and has been employed to assess anxiety as a factor contributing to differential response to reading intervention (Grills-Taquechel et al., 2012; Grills et al., 2014).

Mindset Survey

Mindset was measured using the Mindset Survey (MSS) (Petscher et al., 2017). The MSS is a self-report, 10-item appraisal of students’ general academic motivation and beliefs. The MSS has been used to examine the relation between mindset and reading comprehension, with fixed mindsets being associated with lower reading comprehension ability compared to growth mindsets (Petscher et al., 2017). The Cronbach’s alpha for the MSS’s internal consistency reliability is 0.76 (Petscher et al., 2017).

Data analysis

Prior to analyses, scores for each measure were transformed into sample-based z scores. Although not the focus of the current study, preliminary analyses were performed to provide context for the moderation analyses. First, one-way ANOVAs were conducted to determine if there were group differences on the reading measures at pre-intervention (i.e., time 1) or on continuous moderator measures at pre-intervention (i.e., time 1). Second, separate ANCOVAs for each reading comprehension outcome were conducted to determine the overall effects of the intervention. The pre-intervention score for each reading outcome measure was included as a covariate in its respective analysis, in adherence with the What Works Clearinghouse guidelines and also to reduce within-group error variance.

To examine the research questions, moderation analyses were conducted to test whether factors were associated with differential responses to the interventions. Specifically, regression models estimated the effect of the intervention conditions compared to BaU and experimental conditions compared to each other on each outcome reading measures for each moderator separately. Two dummy codes were used to represent the differences among the three groups: teacher-led (TL) versus business-as-usual (BaU) and computer-delivered (CD) versus BaU. Pre-intervention scores for each reading outcome were controlled for in each model. Final models included all significant moderators for a particular reading outcome. The regression equations for the observed outcomes for each reading outcome are as follows:

Reading outcomeij = β0ijk + β1 (pretest reading) + β2 (TL) ij + β3 (CD) ij + β4 (moderator) ij + β5 (TL × moderator) ij + β6 (CD × moderator) ij + u0j + e0ij.

Follow-up analyses were conducted to further investigate significant moderated treatment effects to determine regions of significance using the Johnson-Neyman technique (Johnson & Fay, 1950).

Results

Pre- and post-intervention means and standard deviations of the reading and the continuous moderator measures (i.e., mind-wandering, anxiety, mindset) are reported in Table 3. Correlations among mind-wandering, anxiety, mindset, and reading measures at pretest (i.e., time 1) and posttest (i.e., time 2) are reported in Table 4. Standard conventions were used to determine the magnitudes of significant effects (Cohen, 1988, 1992). There was a statistically significant moderate, positive correlation between general anxiety and mind-wandering, such that higher levels of anxiety were associated with higher levels of reported mind-wandering at both pretest (time 1: r = 0.33, p = 0.007) and posttest (time 2: r = 0.42, p < 0.001). There were no other significant correlations of factors at pre- or posttest. At pretest, there was a moderate positive correlation between mind-wandering and the CIRCA (time 1: r = 0.41, p < 0.001), but not at posttest (time 2: r = 0.19, p = 0.128). At post-intervention, there was a moderate positive relationship between the Bridge-IT near and mind-wandering (time 2: r = 0.31, p = 0.012). These correlations indicate that higher levels of mind-wandering were associated with better performance on the CIRCA and Bridge-IT measures.

Table 3 Descriptive statistics for reading measures and moderators
Full size table
Table 4 Correlations of moderators and reading measures at pre-and post-intervention
Full size table

Preliminary analyses

Initial group differences

At pretest (i.e., time 1), the groups did not significantly differ on the TOWRE-2 [F (2,63) = 0.308, p = 0.736], CIRCA [F (2,63) = 1.088, p = 0.343], Bridge-IT near [F (2,63) = 1.632, p = 0.204], Bridge-IT far [F (2,63) = 0.136, p = 0.873], or WIAT [F (2,63) = 0.078, p = 0.925]. The groups did not significantly differ on anxiety [F (2,63) = 0.628, p = 0.537], mind-wandering [F (2,63) = 0.434, p = 0.650], or mindset [F (2,63) = 0.013, p = 0.987]. At pre-intervention, 19.70% (13 of 66 students) of the sample met the clinical criteria for anxiety and an additional 22.73% (15 of 66 students) of the sample were identified as at-risk for clinical levels of anxiety.

Overall treatment effects

There was a significant effect for group on the WIAT-III [F (2,62) = 3.18, p = 0.048], in which post hoc comparisons indicated that the teacher-led group significantly outperformed the BaU control group (p = 0.044, Hedge’s g = 0.59) and the computer-delivered group (p = 0.044, Hedge’s g = 0.56). There was no significant difference between the computer-delivered and BaU control groups on the WIAT-III (p = 0.986, Hedge’s g = − 0.06). There were no significant effects for the three groups on the CIRCA [F (2,62) = 3.10, p = 0.052], Bridge-IT near [F (2,62) = 0.34, p = 0.711], or Bridge-IT far [F (2,62) = 0.51, p = 0.600]. There were significant main effects of language status on the CIRCA [F (1,64) = 8.39, p = 0.005], Bridge-IT near [F (1,64) = 6.36, p = 0.014], and the WIAT-III [F (1,64) = 5.15, p = 0.027], indicating that non-ELs outperformed ELs.

Moderation effects

CIRCA

On the CIRCA, there was a statistically significant interaction involving anxiety and the computer-delivered group vs. BaU (β = 0.39, p = 0.006, Table 5). There was also a significant interaction between language status (i.e., EL, non-EL) and the computer-delivered group vs. BaU (β = − 0.69, p = 0.009). In the final model, the moderation effect of anxiety remained significant when controlling for language status (β = 0.40, p = 0.007), indicating that compared to BaU instruction, the computer-delivered intervention benefitted students with higher levels of anxiety. A follow-up analysis determined the region of significant difference between the computer-delivered group and BaU group was at 0.80 standard deviations above the sample mean in anxiety, illustrated in Fig. 1.

Table 5 Moderators of intervention effect on the CIRCA
Full size table
Fig. 1
figure 1

Interaction between computer-delivered vs. BaU and anxiety for the CIRCA

Full size image

The moderation effect of language status on the CIRCA also remained significant for the computer-delivered group vs. BaU group (β = − 0.66, p = 0.015) after controlling for anxiety in the final model. This indicated that, compared to EL students in the BaU condition, ELs in the computer-delivered group scored lower on the CIRCA (Fig. 2).

Fig. 2
figure 2

Interaction between computer-delivered vs. BaU and language status for the CIRCA

Full size image

Bridge-IT Near

On the Bridge-IT Near, there was a statistically significant interaction between the computer-delivered group vs. BaU group and mind-wandering (β = 0.50, p = 0.019) and also anxiety (β = 0.42, p = 0.036, Table 6). In the final model, the moderation effect of anxiety did not reach statistical significance (β = 0.38 p = 0.061) when controlling for mind-wandering, so no additional analyses were conducted for anxiety. After controlling for anxiety in the final model, the moderation effect of mind-wandering on the Bridge-IT Near remained significant (β = 0.50, p = 0.019), indicating that students with higher levels of mind-wandering in the computer-delivered group demonstrated higher performance on the Bridge-IT Near compared to BaU. Follow-up analysis revealed that the difference between the computer-delivered group and the BaU group transitioned to significance at 2.1 standard deviations above the sample mean on mind-wandering, as illustrated in Fig. 3. Notably, there were no observed cases in the current sample within this range.

Table 6 Moderators of intervention effect on Bridge-It Near
Full size table
Fig. 3
figure 3

Interaction between computer-delivered vs. BaU and mind-wandering for Bridge-It Near

Full size image

Bridge-IT Far and WIAT-III

There were no statistically significant interactions between the groups and any of the potential moderators on the Bridge-IT Far or on the WIAT-III Reading Comprehension subtest. In other words, intervention effects on the Bridge-IT Far and WIAT-III were similar across language status and pre-intervention levels of anxiety, mind-wandering, and mindset.

Discussion

The purpose of the current study was to examine whether student-level factors related to attention and language status moderated the treatment effects of two versions of an inferential reading comprehension intervention for struggling middle school readers. Significant moderating effects of mind-wandering, anxiety, and language status were found for some comprehension outcome measures. As predicted, the computer-delivered intervention was beneficial compared to BaU for students with higher levels of anxiety. In contrast to predictions, the computer-delivered intervention was also beneficial for students with higher self-reported mind-wandering (rather than the teacher-led condition, as hypothesized). With respect to EL status, a differential response was present for the computer-delivered format, but not for the teacher-led intervention, such that ELs in the BaU group performed better than their EL peers in the computer-led intervention group.

Moderation effects

Mind-wandering

In comparison to their peers in the BAU group with similarly high levels of self-reported mind-wandering, students in the computer-delivered intervention were better able to make text-connecting inferences on the Bridge-IT Near, a test of inferential comprehension. The computer-delivered instruction incorporated game-like rewards, which may have increased engagement and motivation and, in turn, decreased mind-wandering (Hess & Gunter, 2013; Ku & Cheng, 2016; Miller et al., 2011; Seli et al., 2019). Furthermore, the self-paced nature of the computerized instruction, along with immediate feedback after question answering, might minimize mind-wandering. For example, mind-wandering studies with college students indicate that frequent testing during lectures results in less mind-wandering than lectures with fewer testing demands (Szpunar, 2017).

In contrast to our predictions, mind-wandering did not moderate effects of the teacher-led instruction. Specifically, positive effects of the intervention on the WIAT-III for the teacher-led instruction were present regardless of students’ reported level of mind-wandering. Verbal praise and encouragement by the interventionists may have reduced the influence of mind-wandering in the teacher-led intervention. This interpretation is consistent with evidence in the literature that greater opportunities for students to actively respond to academic content (e.g., answering questions, reading, writing answers) is associated with stronger student engagement and academic achievement (e.g., MacSuga-Gage & Simonsen, 2015). The teacher-led intervention was implemented in small groups such that all students were included in the instruction and were expected to respond (e.g., read passages, respond to questions) frequently across the intervention session. In contrast, large classroom settings, such as those characterizing BaU, may include active participation of only a small number of students at any one time (i.e., fewer opportunities to actively respond for students overall), leaving other students with the opportunity to wander off task particularly during silent reading and “teacher talk,” as has been reported in the literature on mind-wandering in college lectures (e.g., Risko et al., 2012).

Anxiety

In comparison to similarly anxious peers in the BaU condition, students in the computer-led intervention performed better on a comprehension test that required them to make several different types of inferences. A similar pattern was found on another measure assessing text-connecting inferences, but the effect failed to reach statistical significance in the presence of mind-wandering. In the computer-delivered version of the intervention, in contrast to large group instruction, the student was the only individual who was aware of their reading pace, any reading errors they made, their responses to questions, and the feedback they received. Thus, the computer may have provided a private, safe space for struggling readers who would normally experience anxiety and self-consciousness when reading and answering questions in a typical classroom setting, such as the BaU classrooms in the current study. It is intriguing to think that computerized reading interventions might support learning for more anxious older students with or at-risk for reading disabilities. In this respect, it is worth noting that 43% of our struggling readers reported clinical or sub-clinical levels of self-reported anxiety.

As previously noted, a central symptom of anxiety is being preoccupied with personal thoughts, which is a form of mind-wandering (Smallwood et al., 2007; Unsworth & McMillan, 2013). In the current study, higher levels of anxiety were associated with higher levels of reported mind-wandering, validating this relation. However, notably, mind-wandering remained significant after controlling for anxiety, but anxiety did not remain significant after controlling for mind-wandering. This suggests that the MWQ may additionally capture inattention, unrelated to anxiety. Furthermore, this finding stresses the importance of including multiple factors within a model to aid in the interpretation of the results.

Mindset

A growth mindset is correlated with reading achievement (Sisk et al., 2018), particularly for students with lower levels of reading comprehension (Petscher et al., 2017). In the current study, treatment effects of the inferential reading comprehension intervention were similar regardless of the initial mindset. This finding is perhaps not too surprising given that mindset was not related to reading outcomes at either pretest or posttest in the current sample. It is unclear if this finding was due to a lack of power from the small sample given the small average effect sizes for mindset and academic achievement reported in the literature (Sisk et al., 2018). Another possibility is that general mindset measures may not be as sensitive as reading-specific mindset measures for older readers (Petscher 2010). Future research should further explore the influence of both general and reading-specific mindset on middle school students’ reading comprehension to determine whether mindset variables moderate older students’ response to reading interventions.

Language status

Consistent with the literature, ELs scored lower overall than non-ELs on all reading measures. More notably, there were differential effects for ELs across the intervention delivery systems. ELs in the computer-delivered intervention had lower performance than ELs in the control condition on the CIRCA, a measure in which they had to make several different types of inferences to understand text. In contrast, intervention effects for the teacher-led intervention were similar for students regardless of their language status.

Both teacher- and computer-led versions of the intervention incorporated instructional strategies that are beneficial for ELs, such as immediate affirmative and corrective feedback, frequent practice opportunities, and discussion of the text (Ehara, 2009; Jozwik, 2015; Taboada & Rutherford, 2011), suggesting that the difference in the influence of language status across instruction modalities may be due to inherent differences in teacher-led instruction (both for the inference intervention and BAU instruction) versus computer-delivered instruction. Computer-delivered instruction was limited to a standard delivery of affirmative feedback and corrective feedback that included the explanation of one incorrect example, which may not be sufficient for ELs. In contrast, interventionists in teacher-led intervention may have elaborated beyond the intervention script by providing richer and more elaborative affirmative feedback, additional examples and explanation in their corrective feedback, additional background information, vocabulary support, and individualized feedback when needed, all of which are known to increase ELs response to instruction (Burgoyne et al., 2013; Hall et al., 2016; Lesaux et al., 2010; Thordardottir, 2010). Support for these explanations would require observations of the control group and quantification of teacher elaborations of the script in the teacher-led version of the intervention, both of which were beyond the scope of the present study. However, we do think that finding a disadvantage for computerized instruction for older ELs is important and worthy of further investigation given that computerized literacy interventions were being used in some of the BaU classrooms (i.e., Read 180, Achieve 3000) and given the high prevalence of ELs in the US school system.

Limitations

There were several limitations of this study. First, this study had a relatively small sample. Small samples can be underpowered and more greatly influenced by outlying data points, compared to larger samples. Accordingly, caution must be used in interpreting and generalizing the findings. Similarly, despite a significant moderator effect in one analysis, there were no observed cases within the region of significance, suggesting that the findings reported in this study require replication with larger samples.

Second, it was not feasible to include or control for all of the factors that may influence students’ response to a reading intervention in the current study. Other language and cognitive factors may have contributed to students’ response to the interventions. For example, working memory and vocabulary have been shown to moderate the effects of reading-based interventions (e.g., Miciak et al., 2014; Swanson et al., 2018). Furthermore, the inclusion of working memory as a moderator in this study might have helped to clarify the findings for mind-wandering and anxiety given that mind-wandering has been found to mediate the relation of working memory to reading comprehension (McVay & Kane, 2012) and that anxiety can reduce WM capacity (Albano et al., 2003). Future research could explore additional potential moderators and their interactions with the factors studied here to better understand the response of older struggling readers to reading comprehension interventions.

Third, the students read and answered the MASC-2, MWQ, and MSS questionnaires silently. Although an assessor was present to answer questions, it is possible that the students misread or misunderstood the statements on the questionnaires. Fourth, not all students were currently receiving a reading intervention (i.e., subset of BaU), despite all students being identified as struggling readers. Although this resulted in a heterogenous BaU group, it also is an accurate representation of BaU for middle school struggling readers. Finally, it was not within the scope of the current study to record teacher-led sessions or have observations of the BaU instruction. However, a more comprehensive measurement of BaU instruction and intervention fidelity may have provided insight into some of the moderation effects in the current study.

Implications

Despite these limitations, there are theoretical and practical implications of the findings for older struggling readers. This study provides a first step in identifying student and instructional characteristics that may influence adolescents’ differential response to reading comprehension interventions. The findings support a multiple deficit model of learning disorders (McGrath et al., 2020) and suggest that mind-wandering, anxiety, and language status interact with struggling adolescent readers’ response to different types of interventions. Testing several student-level factors, controlling for effects of other moderators when a moderator was found to be significant, and comparing effects across teacher-led and computerized versions of the same intervention, are features of the current study that are useful for providing more precise information about for whom interventions are most effective and under what conditions. Understanding what student characteristics and instructional elements influence reading comprehension and impact response to reading-based interventions may lead to the development of more effective interventions to address the academic needs of middle school students with reading difficulties.

Our focus on factors associated with attention is consistent with research on the potential of such factors for affecting intervention effectiveness (e.g., Duncan et al., 2007; Rabiner et al., 2016a) and may be particularly important for adolescents with reading difficulties for whom anxiety and mind-wandering are a greater concern compared to younger students (Berthiaume et al., 2010; Kessler et al., 2007). For example, the level of anxiety in the sample was quite high (43% were in the clinical or at-risk range for anxiety) and the region of significance for the moderation effect for the computer intervention (compared to BaU) was less than one standard deviation above the sample mean in anxiety, suggesting that factors such as anxiety might affect a significant number of secondary school students in the context of comprehension interventions.

The differential effects across the two delivery systems are particularly important when selecting and developing reading interventions for older ELs. Prior research indicates that ELs generally do not respond as well to reading instruction compared to monolingual students (Hall et al., 2016). However, language status did not moderate the effects of the teacher-led intervention, indicating that ELs performed positively and similarly to non-ELs with a small-group teacher-led reading intervention focusing on inference-making. Moreover, the unanticipated finding that ELs in the BaU condition performed better than ELs in the computer-delivered intervention suggests that there should be a special consideration of language status when selecting modes of reading intervention delivery. Together, these findings suggest that customizing interventions for older readers might require consideration of both individual student characteristics and the type of instructional environment.

Despite these interesting findings, there is a need for replication in a larger sample with more detailed information on the counterfactual condition. Additionally, understanding how different features of the computerized intervention supported learning for adolescents with high levels of anxiety and mind-wandering, and how it may hinder learning for ELs, would be useful both for interpreting the current findings and for improving other reading interventions. For example, shorter instructional sessions, integration of self-monitoring techniques, and frequent interpolated testing reduce mind-wandering (DiCerbo et al., 2004; Hedin et al., 2011; Szpunar et al., 2013). To what extent does the computerized version of the intervention draw on these features and how might such features be further manipulated across intervention conditions?

Finally, as schools incorporate computer-based reading programs, it is important to reflect on how student characteristics might affect reading outcomes in the context of these computerized interventions. Computer-delivered reading interventions are often appealing for schools that have large numbers of students needing additional academic supports, and limited opportunities during the school day for specialized interventions in the secondary school grades. Furthermore, school closures due to the COVID-19 pandemic have elevated the need for remote learning options, a need that computer-based reading interventions could potentially fill. Our findings suggest a nuanced view is required regarding computer-delivered intervention. These interventions might be advantageous for some students (e.g., those with higher levels of anxiety and mind-wandering in the current study) but may have no benefit or may even be disadvantageous for other students (e.g., ELs in the current study).

Conclusion

The results indicated that individual student characteristics that affect their engagement with text and their interaction with intervention delivery systems affected how middle school students responded to an inferential reading comprehension intervention. These findings contribute to the growing literature on factors that produce differential response to intervention in individuals with learning disabilities (Fuchs & Fuchs, 2019). They highlight the need for future work to consider the potential moderating influences of student and intervention characteristics on response to intervention and the possibility that these moderating factors might change across development. The results suggest that consideration of student characteristics and instructional elements, such as group size and delivery by a computer or a teacher, may be particularly important for developing effective reading comprehension instruction for struggling middle school readers. These issues may be especially relevant for students with reading-related anxiety, elevated levels of mind-wandering, and English learners.