Presentation-mode effects in large-scale writing assessments

Highlights

• Raters scored computer-typed transcripts higher than handwritten original essays.

• Presentation-mode effects were found for content quality and style quality.

• Text quality and grammatical erroneousness moderated the presentation-mode effects.

• Presentation-mode effects differed between text genres.

Abstract

To ensure valid measurement in large-scale assessments, avoiding incorporating construct-irrelevant aspects is crucial. We investigated a potential source of construct-irrelevant variance, i.e. the presentation mode of essays (handwritten vs. computer-typed) and its influence on scoring. Further, we investigated whether the presentation-mode effect is moderated by text quality and legibility, as well as by orthographic, grammatical, and punctuation erroneousness, and whether the presentation mode differs over text genres. Rater-teams scored two versions (originally handwritten vs. computer-typed transcripts) of 430 essays written by students at the end of lower secondary education in Germany on content quality and style quality, as well as the considered moderators. Statistical analyses showed a main effect for presentation mode: Computer-typed transcripts were scored higher than handwritten essays. This effect was moderated by text quality and grammatical erroneousness. Furthermore, the strength of the presentation-mode effect differed by text genre. We argued that the strength as well as the direction of the presentation mode are guided by these and further factors.

Introduction

We live in an era of ongoing digitalization. This issue affects education and educational assessment. In the Programme for International Student Assessment (PISA) 2015, computer-based assessment was the primary assessment mode for the first time (Jerrim, Micklewright, Heine, Salzer, & McKeown, 2018; OECD, 2017). More and more writing assessments have shifted (or plan to shift) to computer-based assessments as well (Applebee, 2011; Sandene et al., 2005). In the last few years, many studies have investigated whether, and in which way(s), computer-based writing is different from handwriting, and how these factors may affect the results of writing products (e.g. Chan, Bax, & Weir, 2018; Cheung, 2016; Horkay, Bennett, Allen, Kaplan, & Yan, 2006). Besides these writer-related differences between handwriting and computer-based writing, there may also be rater-based differences. Typewritten texts may be received, estimated, and—within the scope of writing assessment—scored differently than handwritten texts. This is a highly relevant point for assessment practice, especially if a mixture of writing modes is applied (computer-based and handwritten, either by choice of the writer or due to available resources in different test locations). For this case, the product-relevant writing processes as well as the rating and scoring should be sufficiently comparable. In addition, the transcription of paper-pencil-written essays is often considered in assessment practice to avoid additional costs because raters require more time to read essays that are not very legible due to poor handwriting and to ensure that ratings will not be influenced by the quality of the handwriting.

In this study, we addressed the question of whether handwritten essays are scored differently to computer-typed transcripts of the same essays in the context of a large-scale writing assessment which was part of a national educational study in Germany of students at the end of lower secondary education. Although the question of differences between ratings of handwritten and typed texts is not new—the initial studies go back to the 1960s—there are hardly any studies that have investigated these potential differences in standardized large-scale-assessments with intensely trained raters. In addition, we investigated different potential moderators of rating differences between handwritten and computer-typed text-versions that have already been detected by previous studies, and their interplay, which has not yet been investigated systematically.