Abstract
Target language proficiency assessment has become an integral part of Second Language Acquisition (SLA) research design, with cloze tests frequently serving this purpose for reasons of practicality. Assumptions underlying the interpretation of such cloze test scores, however, are often not examined. With the goal of providing researchers with better means for drawing inferences from cloze test scores, we present an analysis of a combined dataset comprised of scores from 1,724 test takers on a frequently used English cloze test (Brown 1980). We examine variation in score distributions and reliability estimates among L2 groups, between L2 and native-speaker (NS) examinees, and for different scoring methods, and investigate the degree to which different sets of items were effective for classifying low- vs high-proficiency L2 examinees and L2 vs NS test takers. Standardized scores are provided for each scoring method so that future researchers can reference their scores to this larger set.
Acknowledgements
This project would not have been possible without the colleagues who generously shared their data from the Brown Cloze test with us. Mahalo to Michelle Adams, Anna Chrabaszcz, Takafumi Fukushima, Amanda Huensch, Kitaek Kim, Hye-Young Kwak, Sunyoung Lee, Akira Omaki, Shoko Sasayama, John Thurman, Annie Tremblay, and Fred Zenker.
References
Alderson, J. Charles. 1978. A study of the cloze procedure with native and non-native speakers of English. Edinburgh, Scotland: Doctoral dissertation.Search in Google Scholar
Brien, Christie. 2013. Neurophysiological evidence of a second language influencing lexical ambiguity resolution in the first language. Ottawa, Canada: University of Ottawa dissertation. https://ruor.uottawa.ca/bitstream/10393/26223/1/Brien_Christie_2013_thesis.pdf (accessed 5 February 2019).Search in Google Scholar
Brown, James Dean. 1978. Correlational study of four methods for scoring cloze tests. Los Angeles, CA: UCLA MA Thesis.Search in Google Scholar
Brown, James Dean. 1980. Relative merits of four methods for scoring cloze tests. The Modern Language Journal 64. 311–317. https://doi.org/10.1111/j.1540-4781.1980.tb05198.x.Search in Google Scholar
Brown, James Dean. 1982. Testing EFL reading comprehension in engineering English. Los Angeles, CA: UCLA dissertation.Search in Google Scholar
Brown, James Dean. 1983. A closer look at cloze: Validity and reliability. In John W. OllerJr. (ed.), Issues in language testing research, pp. 237-250. Rowley, MA: Newbury House.Search in Google Scholar
Brown, James Dean. 1984. A cloze is a cloze is a cloze? In Jean Handscombe, Richard Orem, & Barry Taylor (eds.) On TESOL ’83: The question of control: Selected papers from the 17th annual TESOL convention, 109–119, Toronto. Washington, DC: TESOL.Search in Google Scholar
Brown, James Dean. 1994. A closer look at cloze: Validity and reliability. In John W. OllerJr. & Jon Jonz (eds.), Cloze and coherence, pp. 189–196. Lewisburg, PA: Associated University Presses.Search in Google Scholar
Brown, James Dean. 2002. Do cloze tests work? Or, is it just an illusion? Second Language Studies 21(1). 79–125.Search in Google Scholar
Brown, James Dean, M. Azevedo Cunha & S. F. Nagem Frota. 2001a. The development and validation of a Portuguese version of the motivated strategies for learning questionnarie. In Zoltán Dörnyei & Richard Schmidt (eds.), Motivation and second language acquisition, pp. 257–280. Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawai‘i Press.Search in Google Scholar
Brown, James Dean, Geoff Robson & Patrick Rosenkjar. 2001b. Personality, motivation, anxiety, strategies, and language proficiency of Japanese students. In Zoltán Dörnyei & Richard Schmidt (eds.), Motivation and second language acquisition, pp. 361–398. Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawai‘i Press.Search in Google Scholar
Chrabaszcz, Anna & Nan Jiang. 2014. The role of the native language in the use of the English nongeneric definite article by L2 learners: A cross-linguistic comparison. Second Language Research 30. 351–379. https://doi.org/10.1177/0267658313493432.Search in Google Scholar
Chrabaszcz, Anna, Matthew Winn, Candise Y. Lin & William J. Idsardi. 2014. Acoustic cues to perception of word stress by English, Mandarin, and Russian speakers. Journal of Speech, Language, and Hearing Research 57. 1468–1479. https://doi.org/10.1044/2014_jslhr-l-13-0279.Search in Google Scholar
Dąbrowska, Ewa. 2012. Different speakers, different grammars: Individual differences in native language attainment. Linguistic Approaches to Bilingualism 2. 219–253.10.1075/lab.2.3.01dabSearch in Google Scholar
Dekydtspotter, Laurent, & A. Katherine Miller. 2009. Probing for intermediate traces in the processing of long-distance wh-dependencies in English as a second language. In Mellisa Bowles, Tania Ionin, Silvina Montrul & Annie Tremblay (eds.), Proceedings of the 10th generative approaches to second language acquisition conference (GASLA 2009), pp. 113–124. Somerville, MA: Cascadilla Proceedings Project.Search in Google Scholar
Ebel, Robert L. 1979. Essentials of educational measurement (3rd ed.). Englewood Cliffs, NJ: Prentice-Hall.Search in Google Scholar
Grosjean, François. 1998. Studying bilinguals: Methodological and conceptual issues. Bilingualism: Language and Cognition 1. 131-149. https://doi.org/10.1017/s136672899800025x.Search in Google Scholar
Grüter, Theres, Hannah Rohde & Amy J. Schafer. 2017. Coreference and discourse coherence in L2: The roles of grammatical aspect and referential form. Linguistic Approaches to Bilingualism 7. 199–229. https://doi.org/10.1075/lab.15011.gru.Search in Google Scholar
Huensch, Amanda. 2014. The perception and production of palatal codas by Korean L2 learners of English. Urbana, IL: University of Illinois at Urbana-Champaign dissertation. http://hdl.handle.net/2142/46739 (accessed 5 February 2019).Search in Google Scholar
Hulstjin, Jan H. 2010. Measuring second language proficiency. In E. Blom & S. Unsworth (eds.), Experimental methods in language acquisition research (EMLAR), pp. 185-199. Amsterdam: John Benjamins.10.1075/lllt.27.11hulSearch in Google Scholar
Hulstijn, Jan H. 2011. Language proficiency in native and non-native speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly 8. 229–249. https://doi.org/10.1080/15434303.2011.565844.Search in Google Scholar
Hulstjin, Jan H. 2012. The construct of language proficiency in the study of bilingualism from a cognitive perspective. Bilingualism: Language and Cognition 15. 422–433.10.1017/S1366728911000678Search in Google Scholar
Kim, Kitaek & Hyunwoo Kim. 2013. L1 Korean transfer in processing L2 English passive sentences. In Eric Voss, Shu D. Tai, & Zhi Li (eds.), Selected proceedings of the 2011 second language research forum, pp. 118–128. Somerville, MA: Cascadilla Proceedings Project.10.17154/kjal.2013.12.29.4.79Search in Google Scholar
Kurilecz, Margaret. 1969. Man and his world: A structured reader. New York: Crowell.Search in Google Scholar
Kwak, Hye Young. 2010. Scope interpretation in first and second language acquisition: Numeral quantifiers and negation. Honolulu, HI: University of Hawai‘i at Mānoa dissertation. http://www.ling.hawaii.edu/graduate/Dissertations/HyeYoungKwakFinal.pdf (accessed 5 February 2019).Search in Google Scholar
Leclercq, Pascale & Amanda Edmonds. 2014. How to assess L2 proficiency? An overview of proficiency assessment research. In Pascale Leclercq, Amanda Edmonds & Heather Hilton (eds.), Measuring L2 proficiency: Perspectives from SLA, pp. 3–23. Bristol, UK: Multilingual Matters.10.21832/9781783092291-004Search in Google Scholar
Lee, Sunyoung. 2010. Interpretation of scope by Korean L2 learners of English: A self-paced reading study. English Teaching 65. 59–78.10.15858/engtea.65.4.201012.59Search in Google Scholar
Lemhöfer, Kristin & Mirjam Broersma. 2012. Introducing LexTALE: A quick and valid lexical test for advanced learners of English. Behavior Research Methods 44. 325–343. https://doi.org/10.3758/s13428-011-0146-0.Search in Google Scholar
Marian, Viorica, Henrike K. Blumenfeld & Margarita. Kaushanskaya. 2007. The language experience and proficiency questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. Journal of Speech, Language, and Hearing Research 50. 940–967. https://doi.org/10.1044/1092-4388(2007/067).Search in Google Scholar
McCray, Gareth & Tineke Brunfaut. 2018. Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking. Language Testing 35. 51–73. https://doi.org/10.1177/0265532216677105.Search in Google Scholar
Norris, John M. (ed.). 2018. Developing C-tests for estimating proficiency in foreign language research. New York: Peter Lang.10.3726/b13235Search in Google Scholar
Norris, John M. & Lourdes Ortega. 2000. Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning 50. 417–528. https://doi.org/10.1111/0023-8333.00136.Search in Google Scholar
Oller, John W.Jr. (1979). Language tests at school: A pragmatic approach. London: Longman.Search in Google Scholar
Pakulak, Eric & Helen J. Neville. 2010. Proficiency differences in syntactic processing of native speakers indexed by event-related potentials. Journal of Cognitive Neuroscience 22. 2728–2744. https://doi.org/10.1162/jocn.2009.21393.Search in Google Scholar
Qin, Zhen, Yu Fu Chien & Annie Tremblay. 2017. Processing of word-level stress by Mandarin-speaking second language learners of English. Applied PsychoLinguistics 38, 541–570. https://doi.org/10.1017/s0142716416000321.Search in Google Scholar
Sasayama, Shoka. 2015. Validating the assumed relationship between task design, cognitive complexity, and second language task performance. Washington, DC: Georgetown University dissertation. https://repository.library.georgetown.edu/handle/10822/1029904 (accessed 5 February 2019).Search in Google Scholar
Schoonen, Rob. 2011. How language ability is assessed. In Eli Hinkel (ed.), Handbook of research in second language teaching and learning (Vol. II), pp. 701–716. London: Routledge.10.4324/9780203836507.ch42Search in Google Scholar
Thomas, Margaret. 1994. Assessment of L2 proficiency in second language acquisition research. Language Learning 44. 307–336. https://doi.org/10.1111/j.1467-1770.1994.tb01104.x.Search in Google Scholar
Thomas, Margaret. 2006. Research synthesis and historiography: The case of assessment of second language proficiency. In J. Norris & L. Ortega (eds.), Synthesizing research on language learning and teaching, pp. 279–298. Philadelphia, PA: John Benjamins.10.1075/lllt.13.13thoSearch in Google Scholar
Tremblay, Annie. 2008. Is second language lexical access prosodically constrained? Processing of word stress by French Canadian second language learners of English. Applied PsychoLinguistics 29, 553–584. https://doi.org/10.1017/s0142716408080247.Search in Google Scholar
Tremblay, Annie. 2011. Proficiency assessment standards in second language acquisition research: “Clozing” the gap. Studies in Second Language Acquisition 33, 339–372. https://doi.org/10.1017/s0272263111000015.Search in Google Scholar
Zenker, Fred & Bonnie D. Schwartz. 2017. Topicalization from adjuncts in English vs. Chinese vs. Chinese-English interlanguage. In Maria LaMendola & Jennifer Scott (eds.), Proceedings of the 41st Annual Boston University Conference on Language Development, pp. 806–819. Somerville, MA: Cascadilla Press.Search in Google Scholar
© 2020 Walter de Gruyter GmbH, Berlin/Boston
