Skip to main content

Advertisement

SpringerLink
Log in

The influence of language knowledge and test components on reading comprehension scores

  • Published:
Annals of Dyslexia Aims and scope Submit manuscript

Abstract

This study investigated the dependability of reading comprehension scores across different text genres and response formats for readers with varied language knowledge. Participants included 78 fourth-graders in an urban elementary school. A randomized and counterbalanced 3 × 2 study design investigated three response formats (open-ended, multiple-choice, retell) and two text genres (narrative, expository) from the Qualitative Reading Inventory (QRI-5) reading comprehension test. Standardized language knowledge measures from the Woodcock Johnson III Tests of Achievement (Academic Knowledge, Oral Comprehension, Picture Vocabulary) defined three reader profiles: (a) < 90 as emerging, (b) 90–100 as basic, and (c) > 100 as proficient. Generalizability studies partitioned variance in scores for reader, text genre, and response format for all three groups. Response format accounted for 42.8 to 62.4% of variance in reading comprehension scores across groups, whereas text genre accounted for very little variance (1.2–4.1%). Single scores were well below a 0.80 dependability threshold (absolute phi coefficients = 0.06–0.14). Decision studies projecting dependability achieved with additional scores varied by response format for each language knowledge group, with very low projected dependability on open-ended and multiple-choice scores for readers with basic language knowledge. Multiple-choice scores had similarly low projected dependability levels for readers with emerging language knowledge. Findings evidence interactions between reader language knowledge and response format in reading comprehension assessment practices. Implications underscore the limitations of using a single score to classify readers with and without proficiency in foundational skills.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Best, R. M., Floyd, R. G., & McNamara, D. S. (2008). Differential competencies contributing to children’s comprehension of narrative and expository texts. Reading Psychology, 29, 137–164.

    Google Scholar 

  • Booher-Jennings, J. (2005). Below the bubble: “Educational triage” and the Texas accountability system. American Educational Research Journal, 42(2), 231–268.

    Google Scholar 

  • Brame, C. (2013). Writing good multiple-choice test questions. Retrieved from https://bit.ly/37BuBeO.

  • Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.

  • Cain, K., Oakhill, J. V., Barnes, M. A., & Bryant, P. E. (2001). Comprehension skill, inference-making ability, and their relation to knowledge. Memory & Cognition, 29(6), 850–859.

  • Catts, H. W., Compton, D., Tomblin, J. B., & Bridges, M. S. (2012). Prevalence and nature of late-emerging poor readers. Journal of Educational Psychology, 104(1), 166–181.

    Google Scholar 

  • Catts, H. W., Hogan, T. P., & Adlof, S. M. (2005). Developmental changes in reading and reading disabilities. In H. W. Catts & A. G. Kamhi (Eds.), The connections between language and reading disabilities (pp. 25–40). New Jersey: Lawrence Erlbaum.

    Google Scholar 

  • Center for Teaching Excellence (2013). Virginia Commonwealth University. Writing multiple-choice questions. Retrieved from https://bit.ly/38NnSid.

  • Clemens, N. H., Hsiao, Y.-Y., Lee, K., Martinez-Lincoln, A., Moore, C., Toste, J., & Simmons, L. (2020). The differential importance of component skills on reading comprehension test performance among struggling adolescent readers. Journal of Learning Disabilities Advanced online publication.

  • Colenbrander, D., Nickels, L., & Kohnen, S. (2017). Similar but different: Differences in comprehension diagnosis on the Neale Analysis of Reading Ability and the York Assessment of Reading for Comprehension. Journal of Research in Reading, 40, 403–419.

    Google Scholar 

  • Collins, A. A., Compton, D. L., Lindström, E. R., & Gilbert, J. K. (2020). Performance variations across reading comprehension assessments: Examining the unique contributions of text, activity, and reader. Reading and Writing: An Interdisciplinary Journal, 33(3), 605–634. https://doi.org/10.1007/s11145-019-09972-5.

  • Compton, D. L., Fuchs, L. S., Fuchs, D., Lambert, W., & Hamlett, C. (2012). The cognitive and academic profiles of reading and mathematics learning disabilities. Journal of Learning Disabilities, 45(1), 79–95.

    Google Scholar 

  • Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. Hoboken: Wiley.

    Google Scholar 

  • Cutting, L. E., & Scarborough, H. S. (2006). Prediction of reading comprehension: Relative contributions of word recognition, language proficiency, and other cognitive skills can depend on how comprehension is measured. Scientific Studies of Reading, 10, 277–299.

    Google Scholar 

  • Eason, S. H., Goldberg, L. F., Young, K. M., Geist, M. C., & Cutting, L. E. (2012). Reader-text interactions: How differential text and question types influence cognitive skills needed for reading comprehension. Journal of Educational Psychology, 104, 515–528.

    Google Scholar 

  • Francis, D. J., Fletcher, J. M., Catts, H. W., & Tomblin, J. B. (2005). Dimensions affecting the assessment of reading comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 369–394). New Jersey: Lawrence Erlbaum.

    Google Scholar 

  • Francis, D. J., Fletcher, J. M., Stuebing, K. K., Lyon, G. R., Shaywitz, B. A., & Shaywitz, S. E. (2005). Psychometric approaches to the identification of LD IQ and achievement scores are not sufficient. Journal of Learning Disabilities, 38(2), 98–108.

    Google Scholar 

  • Francis, D. J., Kulesz, P. A., & Benoit, J. S. (2018). Extending the simple view of reading to account for variation within readers and across texts: The complete view of reading (CVRi). Remedial and Special Education: RASE, 39(5), 274–288.

    Google Scholar 

  • García, J. R., & Cain, K. (2014). Decoding and reading comprehension: A meta-analysis to identify which reader and assessment characteristics influence the strength of the relationship in English. Review of Educational Research, 1, 1–38. https://doi.org/10.3102/0034654313499616.

  • Graesser, A. C., McNamara, D. S., & Kulikowich, J. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223–234 https://doi.org/10/cwtd84.

    Google Scholar 

  • Haladyna, T. M. (1999). Developing and validating multiple-choice test items (2nd ed.). New Jersey: Lawrence Erlbaum.

    Google Scholar 

  • Hogan, T. P., Adlof, S. M., & Alonzo, C. N. (2014). On the importance of listening comprehension. International Journal of Speech-Language Pathology, 16(3), 199–207.

    Google Scholar 

  • Hua, A. N., & Keenan, J. M. (2017). Interpreting reading comprehension test results: Quantile regression shows that explanatory factors can vary with performance level. Scientific Studies of Reading, 21(3), 225–238.

    Google Scholar 

  • Kamhi, A. G. (2009). Solving the reading crisis—Take 2: The case for differentiated assessment. Language, Speech, and Hearing Services in Schools, 40, 212–215.

    Google Scholar 

  • Keenan, J. M. (2014). Assessment of reading comprehension. In C. A. Stone, E. R. Silliman, B. J. Ehren, & G. P. Wallach (Eds.), Handbook of Language and Literacy: Development and Disorders (2nd ed., pp. 469–484). New York: Guilford.

    Google Scholar 

  • Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12, 281–300. https://doi.org/10.1080/10888430802132279.

    Article  Google Scholar 

  • Keenan, J. M., Hua, A. N., Meenan, C. E., Pennington, B. F., Willcutt, E., & Olson, R. K. (2014). Issues in identifying poor comprehenders. L’Annee Psychologique, 114(4), 753–777.

    Google Scholar 

  • Keenan, J. M., & Meenan, C. E. (2014). Test differences in diagnosing reading comprehension deficits. Journal of Learning Disabilities, 47, 125–135.

    Google Scholar 

  • Kintsch, W., & Rawson, K. A. (2007). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 209–226). Hoboken: Blackwell.

    Google Scholar 

  • Kulesz, P. A., Francis, D. J., Barnes, M. A., & Fletcher, J. M. (2016). The influence of properties of the test and their interactions with reader characteristics on reading comprehension: An explanatory item response study. Journal of Educational Psychology, 108(8), 1078–1097.

    Google Scholar 

  • Leslie, L., & Caldwell, J. (2011). Qualitative Reading inventory (5th ed.). London: Pearson.

    Google Scholar 

  • McGrew, K. S., Schrank, F. A., & Woodcock, R. W. (2007). Woodcock-Johnson III normative update: Technical manual. Rolling Meadows: Riverside.

    Google Scholar 

  • McNamara, D. S., Ozuru, Y., & Floyd, R. G. (2011). Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers’ prior knowledge. International Electronic Journal of Elementary Education, 4(1), 229–257.

    Google Scholar 

  • Miller, A. C., Davis, N., Gilbert, J. K., Cho, S.-J., Toste, J. R., Street, J., & Cutting, L. E. (2014). Novel approaches to examine passage, student, and question effects on reading comprehension. Learning Disabilities Research & Practice, 29(1), 25–35.

    Google Scholar 

  • Nation, K. (2019). Children’s reading difficulties, language, and reflections on the simple view of reading. Australian Journal of Learning Difficulties, 24(1), 47–73.

    Google Scholar 

  • Nation, K., & Snowling, M. (1997). Assessing reading difficulties: The validity and utility of current measures of reading skill. British Journal of Educational Psychology, 67, 359–370.

    Google Scholar 

  • Nunnally, J. C., Bernstein, I. H., & Berge, J. M. T. (1967). Psychometric theory (Vol. 226). New York: McGraw-Hill.

    Google Scholar 

  • Perfetti, C., & Stafura, J. (2014). Word knowledge in a theory of reading comprehension. Scientific Studies of Reading, 18(1), 22–37.

    Google Scholar 

  • Pfeiffer, S. I., Reddy, L. A., Kletzel, J. E., Schmelzer, E. R., & Boyer, L. M. (2000). The practitioner’s view of IQ testing and profile analysis. School Psychology Quarterly, 15(4), 376–385.

    Google Scholar 

  • Priebe, S. J., Keenan, J. M., & Miller, A. C. (2012). How prior knowledge affects word identification and comprehension. Reading and Writing, 25, 131–149.

    Google Scholar 

  • RAND Reading Study Group. (2002). Reading for understanding: Toward an R&D program in reading comprehension. RAND.

    Google Scholar 

  • Reardon, S. F., Kalogrides, D., Fahle, E. M., Podolsky, A., & Zárate, R. C. (2018). The relationship between test item format and gender achievement gaps on math and ELA tests in fourth and eighth grades. Educational Researcher, 47, 284–294.

    Google Scholar 

  • Reed, D. K., & Vaughn, S. (2012). Retell as an indicator of reading comprehension. Scientific Studies of Reading, 16, 187–217.

    Google Scholar 

  • Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming in written composition. In S. Rosenberg (Ed.), Advances in applied psycholinguistics (Vol. 2, pp. 142–175) Cambridge.

    Google Scholar 

  • Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922–932.

    Google Scholar 

  • Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer (Vol. 1). Sage.

  • Spencer, M., Gilmour, A. F., Miller, A. C., Emerson, A. M., Saha, N. M., & Cutting, L. E. (2019). Understanding the influence of text complexity and question type on reading outcomes. Reading and Writing: An Interdisciplinary Journal, 32, 603–637 Doi:10/cxrn.

    Google Scholar 

  • Spear-Swerling, L. (2004). Fourth Graders Performance on a State-Mandated Assessment Involving Two Different Measures of Reading Comprehension. Reading Psychology, 25, 121–148. https://doi.org/10.1080/02702710490435727.

  • Spencer, M., Quinn, J. M., & Wagner, R. K. (2014). Specific reading comprehension disability: Major problem, myth, or misnomer? Learning Disabilities Research & Practice, 29(1), 3–9.

    Google Scholar 

  • Stuebing, K. K., Fletcher, J. M., Branum-Martin, L., & Francis, D. J. (2012). Evaluation of the technical adequacy of three methods for identifying specific learning disabilities based on cognitive discrepancies. School Psychology Review, 41(1), 3–22.

    Google Scholar 

  • Swiss Society for Research in Education Working Group. (2012). EduG (Version 6.1) [Computer software]. Unpublished instrument. Retrieved from http://www.irdp.ch/edumetrie/englishprogram.htm.

  • Tannenbaum, K. R., Torgesen, J. K., & Wagner, R. K. (2006). Relationships between word knowledge and reading comprehension in third-grade children. Scientific Studies of Reading, 10(4), 381–398.

    Google Scholar 

  • Vellutino, F. R., Tunmer, W. E., Jaccard, J. J., & Chen, R. (2007). Components of reading ability: Multivariate evidence for a convergent skills model of reading development. Scientific Studies of Reading, 11(1), 3–32.

    Google Scholar 

  • Walker, R. (2017). The effect of response format and presentation conditions on comprehension assessments for students with and without a reading disability (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Full Text. (10601892)

  • Webb, N. M., Rowley, G. L., & Shavelson, R. J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81–90.

  • Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). 4 reliability coefficients and generalizability theory. Handbook of Statistics, 26, 81–124.

    Google Scholar 

  • Wixson, K. K. (2017). An interactive view of reading comprehension: Implications for assessment. Language, Speech, and Hearing Services in Schools, 48(2), 77–83.

    Google Scholar 

  • Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock Johnson III Tests of achievement. Rolling Meadows: Riverside.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Curriculum & Instruction, Texas State University, San Marcos, TX, 78666, USA

    Alyson A. Collins

  2. Department of Education and Human Services, Lehigh University, Bethlehem, PA, USA

    Esther R. Lindström

  3. Department of Special Education, The University of Texas at Austin, Austin, TX, USA

    Micheal Sandbank

Authors
  1. Alyson A. Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Esther R. Lindström
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Micheal Sandbank
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alyson A. Collins.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Collins, A.A., Lindström, E.R. & Sandbank, M. The influence of language knowledge and test components on reading comprehension scores. Ann. of Dyslexia 71, 238–259 (2021). https://doi.org/10.1007/s11881-020-00212-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11881-020-00212-y

Keywords

Search

Navigation