The present study leveraged advances in automated essay scoring (AES) technology to explore a proof of concept for a writing screener using the Project Essay Grade (PEG) program. First, the study investigated the extent to which an AES-scored multi-prompt writing screener accurately classified students as at risk of failing a Common Core-aligned English language arts state test. Second, the study explored whether a similar level of classification accuracy could be achieved with a more efficient form of the AES-screener with fewer writing prompts. Third, the classification accuracy of the AES-scored screeners was compared to that of screeners scored for word count. Students in Grades 3–5 (n = 185, 167, and 187, respectively) composed six essays in response to multiple writing-prompt screeners on six different randomly assigned topics, consisting of two essays in each of three different genres (narrative, informative, and persuasive). Receiver operating characteristic (ROC) curve analysis was used to assess classification accuracy and to identify multiple cut scores with associated sensitivity and specificity values, and positive and negative posttest probabilities. Results indicated that the AES-scored multi-prompt screener and screeners with fewer prompts yield acceptable classification accuracy, are efficient, and are more accurate than screeners scored for word count. Overall, results illustrate the viability of writing screening using AES.

本研究利用自动论文评分 (AES) 技术的进步来探索使用项目论文评分(PEG) 程序的写作筛选器的概念证明。首先，该研究调查了 AES 评分的多提示写作筛选器在多大程度上将学生准确地归类为有可能无法通过与 Common Core 一致的英语语言艺术状态测试的风险。其次，该研究探讨了是否可以使用更有效的 AES 筛选器形式和更少的书写提示来实现类似水平的分类准确度。第三，将 AES 评分筛选器的分类准确性与字数评分筛选器的分类准确性进行比较。3-5 年级的学生 ( n = 185、167 和 187 分别针对六个不同的随机分配主题撰写了六篇文章，以回应多个写作提示筛选器，包括三种不同类型（叙述性、信息性和说服性）中的每一种的两篇文章。接受者操作特征 (ROC) 曲线分析用于评估分类准确性并识别具有相关敏感性和特异性值以及阳性和阴性后测概率的多个切分分数。结果表明，AES 评分的多提示筛选器和提示较少的筛选器产生可接受的分类准确度，是有效的，并且比对字数进行评分的筛选器更准确。总的来说，结果说明了使用 AES 进行写作筛选的可行性。