-
Open Science in language assessment research contexts: A reply to Winke Language Testing (IF 2.2) Pub Date : 2024-08-08 Carol A. Chapelle, Gary J. Ockey
-
A Global South perspective on Open Science in language assessment: A response to Paula Winke Language Testing (IF 2.2) Pub Date : 2024-08-08 Atta Gebril, Maha Bali
-
Sharing, collaborating, and building trust: How Open Science advances language testing Language Testing (IF 2.2) Pub Date : 2024-08-08 Paula Winke
The Open Science movement is taking hold around the world, and language testers are taking part. In this viewpoint, I discuss how sharing, collaborating, and building trust, guided by Open Science principles, benefit the language testing field. To help more language testers join in, I present a standard definition of Open Science and describe four ways language testing researchers can immediately partake
-
Evaluating the impact of nonverbal behavior on language ability ratings Language Testing (IF 2.2) Pub Date : 2024-08-08 J. Dylan Burton
Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long
-
Can language test providers do more to support open science? A response to Winke Language Testing (IF 2.2) Pub Date : 2024-08-08 Spiros Papageorgiou
In this letter, I first present examples of the adoption of Open Science by the language assessment industry. I then discuss some of the inevitable challenges language assessment professionals face as they continue to adopt Open Science.
-
Considerations to promote and accelerate Open Science: A response to Winke Language Testing (IF 2.2) Pub Date : 2024-08-08 Rie Koizumi, Ryo Maie, Akifumi Yanagisawa, Yo In’nami
-
An industry perspective on open science: A response to Winke (2024) Language Testing (IF 2.2) Pub Date : 2024-08-05 Geoffrey T. LaFlair
Open science practices are now at the forefront of discussions in the applied linguistics research community. Proponents of open science argue for its potential to enhance research quality and accessibility while promoting a collaborative and equitable environment. Winke advocates for integrating open science into language assessment research to enhance research quality, accessibility, and collaboration
-
Do source use features impact raters’ judgment of argumentation? An experimental study Language Testing (IF 2.2) Pub Date : 2024-07-31 Ping-Lin Chuang
This experimental study explores how source use features impact raters’ judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were adapted from essays written by EPT test-takers. These responses
-
What is the best predictor of word difficulty? A case of data mining using random forest Language Testing (IF 2.2) Pub Date : 2024-07-30 Hung Tan Ha, Duyen Thi Bich Nguyen, Tim Stoeckel
Word frequency has a long history of being considered the most important predictor of word difficulty and has served as a guideline for several aspects of second language vocabulary teaching, learning, and assessment. However, recent empirical research has challenged the supremacy of frequency as a predictor of word difficulty. Accordingly, applied linguists have questioned the use of frequency as
-
Authenticity of academic lecture passages in high-stakes tests: A temporal fluency perspective Language Testing (IF 2.2) Pub Date : 2024-07-27 Hitoshi Nishizawa
Corpus-based studies have offered the domain definition inference for test developers. Yet, corpus-based studies on temporal fluency measures (e.g., speech rate) have been limited, especially in the context of academic lecture settings. This made it difficult for test developers to sample representative fluency features to create authentic listening passages. To address this issue, the Fluency Corpus
-
Open access in language testing and assessment: The case of two flagship journals Language Testing (IF 2.2) Pub Date : 2024-07-27 Meng Liu, Ali H. Al-Hoorie, Phil Hiver
This study is a systematic examination of the open access status of research in two flagship language testing and assessment journals: Language Testing and Language Assessment Quarterly. Coding and analysing 898 articles, we investigated (a) the prevalence of open access in four aspects—open manuscripts, open materials, open data, and open code, and (b) the relationship between open access and various
-
A Context-Aligned Two Thousand Test: Toward estimating high-frequency French vocabulary knowledge for beginner-to-low intermediate proficiency adolescent learners in England Language Testing (IF 2.2) Pub Date : 2024-07-26 Amber Dudley, Emma Marsden, Giulia Bovolenta
Vocabulary knowledge strongly predicts second language reading, listening, writing, and speaking. Yet, few tests have been developed to assess vocabulary knowledge in French. The primary aim of this pilot study was to design and initially validate the Context-Aligned Two Thousand Test (CA-TTT), following open research practices. The CA-TTT is a test of written form–meaning recognition of high-frequency
-
A scoping review of research on second language test preparation Language Testing (IF 2.2) Pub Date : 2024-05-31 Shanshan He, Anne-Marie Sénécal, Laura Stansfield, Ruslan Suvorov
Test preparation has garnered considerable attention in second language (L2) education due to the significant implications that successful performance on a language test may have for academic advancement, future career opportunities, and immigration prospects. Meanwhile, an overemphasis on test preparation has been criticized for encouraging the cultivation of construct-irrelevant test-taking strategies
-
Book review: From assessment to feedback by Inez De Florio Language Testing (IF 2.2) Pub Date : 2024-04-17 Salomé Villa Larenas
-
The effect of viewing visual cues in a listening comprehension test on second language learners’ test-taking process and performance: An eye-tracking study Language Testing (IF 2.2) Pub Date : 2024-04-17 Suh Keong Kwon, Guoxing Yu
In this study, we examined the effect of visual cues in a second language listening test on test takers’ viewing behaviours and their test performance. Fifty-seven learners of English in Korea took a video-based listening test, with their eye movements recorded, and 23 of them were interviewed individually after the test. The participants viewed the visual cues longer than the items in the multiple-choice
-
The moderating role of L2 proficiency in the predictive power of L1 fluency on L2 utterance fluency Language Testing (IF 2.2) Pub Date : 2024-04-17 Shungo Suzuki, Judit Kormos
The current study examined the extent to which first language (L1) utterance fluency measures can predict second language (L2) fluency and how L2 proficiency moderates the relationship between L1 and L2 fluency. A total of 104 Japanese-speaking learners of English completed different argumentative speech tasks in their L1 and L2. Their speaking performance was analysed using measures of speed, breakdown
-
Communal factors in rater severity and consistency over time in high-stakes oral assessment Language Testing (IF 2.2) Pub Date : 2024-04-10 Reeta Neittaanmäki, Iasonas Lamprianou
This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnishspeaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over
-
All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency Language Testing (IF 2.2) Pub Date : 2024-04-10 Reeta Neittaanmäki, Iasonas Lamprianou
This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected
-
Test score comparison tables: How well are they serving test users? Language Testing (IF 2.2) Pub Date : 2024-03-27 Ute Knoch, Jason Fan
While several test concordance tables have been published, the research underpinning such tables has rarely been examined in detail. This study aimed to survey the publically available studies or documentation underpinning the test concordance tables of the providers of four major international language tests, all accepted by the Australian Department of Home Affairs for Australian visa purposes. To
-
Book review: L2 Writing Assessment: An Evolutionary Perspective Language Testing (IF 2.2) Pub Date : 2024-03-22 Khaled Barkaoui
-
Evaluating methodological enhancements to the Yes/No Angoff standard-setting method in language proficiency assessment Language Testing (IF 2.2) Pub Date : 2024-02-12 Tia M. Fechter, Heeyeon Yoon
This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent mixed-methods study design. The study used the Yes/No
-
A shortened test is feasible: Evaluating a large-scale multistage adaptive English language assessment Language Testing (IF 2.2) Pub Date : 2024-02-08 Shangchao Min, Kyoungwon Bishop
This paper evaluates the multistage adaptive test (MST) design of a large-scale academic language assessment (ACCESS) for Grades 1–12, with an aim to simplify the current MST design, using both operational and simulated test data. Study 1 explored the operational population data (1,456,287 test-takers) of the listening and reading tests of MST ACCESS in the 2018–2019 school year to evaluate the MST
-
Setting standards for a diagnostic test of aviation English for student pilots Language Testing (IF 2.2) Pub Date : 2024-02-06 Maria Treadaway, John Read
Standard-setting is an essential component of test development, supporting the meaningfulness and appropriate interpretation of test scores. However, in the high-stakes testing environment of aviation, standard-setting studies are underexplored. To address this gap, we document two stages in the standard-setting procedures for the Overseas Flight Training Preparation Test (OFTPT), a diagnostic English
-
Korean Syntactic Complexity Analyzer (KOSCA): An NLP application for the analysis of syntactic complexity in second language production Language Testing (IF 2.2) Pub Date : 2024-02-06 Haerim Hwang, Hyunwoo Kim
Given the lack of computational tools available for assessing second language (L2) production in Korean, this study introduces a novel automated tool called the Korean Syntactic Complexity Analyzer (KOSCA) for measuring syntactic complexity in L2 Korean production. As an open-source graphic user interface (GUI) developed in Python, KOSCA provides seven indices of syntactic complexity, including traditional
-
The development of a Chinese vocabulary proficiency test (CVPT) for learners of Chinese as a second/foreign language Language Testing (IF 2.2) Pub Date : 2024-01-10 Haiwei Zhang, Peng Sun, Yaowaluk Bianglae, Winda Widiawati
In order to address the needs of the continually growing number of Chinese language learners, the present study developed and presented initial validation of a 100-item Chinese vocabulary proficiency test (CVPT) for learners of Chinese as a second/foreign language (CS/FL) using Item Response Theory among 170 CS/FL learners from Indonesia and 354 CS/FL learners from Thailand. Participants were required
-
Open Science should be welcomed by test providers but grounded in pragmatic caution: A response to Winke Language Testing (IF 2.2) Pub Date : 2024-01-04 Tony Clark, Emma Bruce
This article is temporarily under embargo.
-
Implementation of an accommodations policy for candidates with diverse needs in a large-scale testing system Language Testing (IF 2.2) Pub Date : 2023-05-16 Johanna Motteram, Richard Spiby, Gemma Bellhouse, Katarzyna Sroka
This article describes the implementation of a special accommodations policy for a suite of localised English language and numeracy tests, the Workplace Literacy and Numeracy (WPLN) Assessments. Th...
-
The relationship between written discourse features and integrated listening-to-write scores for adolescent English language learners Language Testing (IF 2.2) Pub Date : 2023-05-13 Ray J. T. Liao, Renka Ohta, Kwangmin Lee
As integrated writing tasks in large-scale and classroom-based writing assessments have risen in popularity, research studies have increasingly concentrated on providing validity evidence. Given th...
-
English foreign language reading and spelling diagnostic assessments informing teaching and learning of young learners Language Testing (IF 2.2) Pub Date : 2023-04-29 Janina Kahn-Horwitz, Zahava Goldstein
In order to inform English foreign language (EFL) diagnostic assessment of literacy, this study examined the extent to which 175 first-language Hebrew-speaking EFL young learners from fifth to tent...
-
Critical discursive approaches to evaluating policy-driven testing: Social impact as a target for validation Language Testing (IF 2.2) Pub Date : 2023-04-27 Dongil Shin
This paper addresses the intersection of testing and policy, situating test-driven impact and validation within the context of policy-led educational reform in Korea. I will briefly review the exis...
-
Speaking performances, stakeholder perceptions, and test scores: Extrapolating from the Duolingo English test to the university Language Testing (IF 2.2) Pub Date : 2023-04-24 Daniel R. Isbell, Dustin Crowther, Hitoshi Nishizawa
The extrapolation of test scores to a target domain—that is, association between test performances and relevant real-world outcomes—is critical to valid score interpretation and use. This study exa...
-
Establishing meaning recall and meaning recognition vocabulary knowledge as distinct psychometric constructs in relation to reading proficiency Language Testing (IF 2.2) Pub Date : 2023-04-24 Jeffrey Stewart, Henrik Gyllstad, Christopher Nicklin, Stuart McLean
The purpose of this paper is to (a) establish whether meaning recall and meaning recognition item formats test psychometrically distinct constructs of vocabulary knowledge which measure separate sk...
-
Modeling local item dependence in C-tests with the loglinear Rasch model Language Testing (IF 2.2) Pub Date : 2023-04-15 Purya Baghaei, Karl Bang Christensen
C-tests are gap-filling tests mainly used as rough and economical measures of second-language proficiency for placement and research purposes. A C-test usually consists of several short independent...
-
Examining the predictive validity of the Duolingo English Test: Evidence from a major UK university Language Testing (IF 2.2) Pub Date : 2023-04-03 Talia Isaacs, Ruolin Hu, Danijela Trenkic, Julia Varga
The COVID-19 pandemic has changed the university admissions and proficiency testing landscape. One change has been the meteoric rise in use of the fully automated Duolingo English Test (DET) for un...
-
The distribution of cognates and their impact on response accuracy in the EIKEN tests Language Testing (IF 2.2) Pub Date : 2023-03-26 David Allen, Keita Nakamura
Although there is abundant evidence for the use of first-language (L1) knowledge by bilinguals when using a second language (L2), investigation into the impact of L1 knowledge in large-scale L2 lan...
-
Measuring the development of general language skills in English as a foreign language—Longitudinal invariance of the C-test Language Testing (IF 2.2) Pub Date : 2023-03-25 Birger Schnoor, Johannes Hartig, Thorsten Klinger, Alexander Naumann, Irina Usanova
Research on assessing English as a foreign language (EFL) development has been growing recently. However, empirical evidence from longitudinal analyses based on substantial samples is still needed....
-
Operationalizing the reading-into-writing construct in analytic rating scales: Effects of different approaches on rating Language Testing (IF 2.2) Pub Date : 2023-03-20 Santi B. Lestari, Tineke Brunfaut
Assessing integrated reading-into-writing task performances is known to be challenging, and analytic rating scales have been found to better facilitate the scoring of these performances than other ...
-
Assessment of fluency in the Test of English for Educational Purposes Language Testing (IF 2.2) Pub Date : 2023-03-13 Parvaneh Tavakoli, Gill Kendon, Svetlana Mazhurnaya, Anna Ziomek
The main aim of this study was to investigate how oral fluency is assessed across different levels of proficiency in the Test of English for Educational Purposes (TEEP). Working with data from 56 t...
-
The relationship among accent familiarity, shared L1, and comprehensibility: A path analysis perspective Language Testing (IF 2.2) Pub Date : 2023-03-13 Yongzhi Miao
Scholars have argued for the inclusion of different spoken varieties of English in high-stakes listening tests to better represent the global use of English. However, doing so may introduce additio...
-
Strategy use in a spoken dialog system–delivered paired discussion task: A stimulated recall study Language Testing (IF 2.2) Pub Date : 2023-03-07 Nazlinur Gokturk, Evgeny Chukharev-Hudilainen
With recent technological advances, researchers have begun to explore the potential use of spoken dialog systems (SDSs) for L2 oral communication assessment. While several studies support the feasi...
-
Proficiency at the lexis–grammar interface: Comparing oral versus written French exam tasks Language Testing (IF 2.2) Pub Date : 2023-03-07 Nathan Vandeweerd, Alex Housen, Magali Paquot
This study investigates whether re-thinking the separation of lexis and grammar in language testing could lead to more valid inferences about proficiency across modes. As argued by Römer, typical s...
-
Investigating the impact of self-pacing on the L2 listening performance of young learner candidates with differing L1 literacy skills Language Testing (IF 2.2) Pub Date : 2023-03-02 Kathrin Eberharter, Judit Kormos, Elisa Guggenbichler, Viktoria S. Ebner, Shungo Suzuki, Doris Moser-Frötscher, Eva Konrad, Benjamin Kremmel
In online environments, listening involves being able to pause or replay the recording as needed. Previous research indicates that control over the listening input could improve the measurement acc...
-
Universal tools activation in English language proficiency assessments: A comparison of Grades 1–12 English learners with and without disabilities Language Testing (IF 2.2) Pub Date : 2023-02-02 Ahyoung Alicia Kim, Meltem Yumsek, Jason A. Kemp, Mark Chapman, H. Gary Cook
English learners (ELs) comprise approximately 10% of kindergarten to Grade 12 students in US public schools, with about 15% of ELs identified as having disabilities. English language proficiency (E...
-
L2 and L1 semantic context indices as automated measures of lexical sophistication Language Testing (IF 2.2) Pub Date : 2023-02-02 Kátia Monteiro, Scott Crossley, Robert-Mihai Botarleanu, Mihai Dascălu
Lexical frequency benchmarks have been extensively used to investigate second language (L2) lexical sophistication, especially in language assessment studies. However, indices based on semantic co-...
-
Linking scores from two written receptive English academic vocabulary tests—The VLT-Ac and the AVT Language Testing (IF 2.2) Pub Date : 2023-01-12 Marcus Warnby, Hans Malmström, Kajsa Yang Hansen
The academic section of the Vocabulary Levels Test (VLT-Ac) and the Academic Vocabulary Test (AVT) both assess meaning-recognition knowledge of written receptive academic vocabulary, deemed central...
-
Measuring bilingual language dominance: An examination of the reliability of the Bilingual Language Profile Language Testing (IF 2.2) Pub Date : 2023-01-12 Daniel J. Olson
Measuring language dominance, broadly defined as the relative strength of each of a bilingual’s two languages, remains a crucial methodological issue in bilingualism research. While various methods...
-
The vexing problem of validity and the future of second language assessment Language Testing (IF 2.2) Pub Date : 2023-01-11 Vahid Aryadoust
Construct validity and building validity arguments are some of the main challenges facing the language assessment community. The notion of construct validity and validity arguments arose from resea...
-
Epilogue—Note from an outgoing editor Language Testing (IF 2.2) Pub Date : 2023-01-11 Luke Harding
In this brief epilogue, outgoing editor Luke Harding reflects on his time as editor and considers the future Language Testing.
-
Reframing the discourse and rhetoric of language testing and assessment for the public square Language Testing (IF 2.2) Pub Date : 2023-01-11 Lynda Taylor
As applied linguists and language testers, we are in the business of “doing language”. For many of us, language learning is a lifelong passion, and we invest similar enthusiasm in our language asse...
-
Administration, labor, and love Language Testing (IF 2.2) Pub Date : 2023-01-11 April Ginther
Great opportunities for language testing practitioners are enabled through language program administration. Local language tests lend themselves to multiple purposes—for placement and diagnosis, as...
-
Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront Language Testing (IF 2.2) Pub Date : 2023-01-11 Tineke Brunfaut
In this invited Viewpoint on the occasion of the 40th anniversary of the journal Language Testing, I argue that at the core of future challenges and opportunities for the field—both in scholarly an...
-
Towards a new sophistication in vocabulary assessment Language Testing (IF 2.2) Pub Date : 2023-01-11 John Read
Published work on vocabulary assessment has grown substantially in the last 10 years, but it is still somewhat outside the mainstream of the field. There has been a recent call for those developing...
-
Reflections on the past and future of language testing and assessment: An emerging scholar’s perspective Language Testing (IF 2.2) Pub Date : 2023-01-11 J. Dylan Burton
In its 40th year, Language Testing journal has served as the flagship journal for scholars, researchers, and practitioners in the field of language testing and assessment. This viewpoint piece, wri...
-
Test design and validity evidence of interactive speaking assessment in the era of emerging technologies Language Testing (IF 2.2) Pub Date : 2023-01-11 Soo Jung Youn
As access to smartphones and emerging technologies has become ubiquitous in our daily lives and in language learning, technology-mediated social interaction has become common in teaching and assess...
-
Construct validity and fairness of an operational listening test with World Englishes Language Testing (IF 2.2) Pub Date : 2023-01-04 Hitoshi Nishizawa
In this study, I investigate the construct validity and fairness pertaining to the use of a variety of Englishes in listening test input. I obtained data from a post-entry English language placemen...
-
But who trains the language teacher educator who trains the language teacher? An empirical investigation of Chilean EFL teacher educators’ language assessment literacy Language Testing (IF 2.2) Pub Date : 2022-12-27 Salomé Villa Larenas, Tineke Brunfaut
Research has shown that language teachers typically feel underprepared for assessment aspects of their job. One reason may relate to how teacher education programmes prepare future teachers in this...
-
Towards more valid scoring criteria for integrated reading-writing and listening-writing summary tasks Language Testing (IF 2.2) Pub Date : 2022-12-12 Sathena Chan, Lyn May
Despite the increased use of integrated tasks in high-stakes academic writing assessment, research on rating criteria which reflect the unique construct of integrated summary writing skills is comp...
-
The typology of second language listening constructs: A systematic review Language Testing (IF 2.2) Pub Date : 2022-12-07 Vahid Aryadoust, Lan Luo
This study reviewed conceptualizations and operationalizations of second language (L2) listening constructs. A total of 157 peer-reviewed papers published in 19 journals in applied linguistics were...
-
Temporal fluency and floor/ceiling scoring of intermediate and advanced speech on the ACTFL Spanish Oral Proficiency Interview–computer Language Testing (IF 2.2) Pub Date : 2022-11-09 Troy L. Cox, Alan V. Brown, Gregory L. Thompson
The rating of proficiency tests that use the Inter-agency Roundtable (ILR) and American Council on the Teaching of Foreign Languages (ACTFL) guidelines claims that each major level is based on hier...
-
Challenges in rating signed production: A mixed-methods study of a Swiss German Sign Language form-recall vocabulary test Language Testing (IF 2.2) Pub Date : 2022-09-21 Aaron Olaf Batty, Tobias Haug, Sarah Ebling, Katja Tissi, Sandra Sidler-Miserez
Sign languages present particular challenges to language assessors in relation to variation in signs, weakly defined citation forms, and a general lack of standard-setting work even in long-establi...