当前位置: X-MOL 学术J. School Psychol. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Cost analysis and cost-effectiveness of hand-scored and automated approaches to writing screening
Journal of School Psychology ( IF 6.033 ) Pub Date : 2022-03-28 , DOI: 10.1016/j.jsp.2022.03.003
Michael Matta 1 , Milena A Keller-Margulis 1 , Sterett H Mercer 2
Affiliation  

Although researchers have investigated technical adequacy and usability of written-expression curriculum-based measures (WE-CBM), the economic implications of different scoring approaches have largely been ignored. The absence of such knowledge can undermine the effective allocation of resources and lead to the adoption of suboptimal measures for the identification of students at risk for poor writing outcomes. Therefore, we used the Ingredients Method to compare implementation costs and cost-effectiveness of hand-calculated and automated scoring approaches. Data analyses were conducted on secondary data from a study that evaluated predictive validity and diagnostic accuracy of quantitative approaches for scoring WE-CBM samples. Findings showed that automated approaches offered more economic solutions than hand-calculated methods; for automated scores, the effects were stronger when the free writeAlizer R package was employed, whereas for hand-calculated scores, simpler WE-CBM metrics were less costly than more complex metrics. Sensitivity analyses confirmed the relative advantage of automated scores when the number of classrooms, students, and assessment occasions per school year increased; again, writeAlizer was less sensitive to the changes in the ingredients than the other approaches. Finally, the visualization of the cost-effectiveness ratio illustrated that writeAlizer offered the optimal balance between implementation costs and diagnostic accuracy, followed by complex hand-calculated metrics and a proprietary automated program. Implications for the use of hand-calculated and automated scores for the universal screening of written expression with elementary students are discussed.



中文翻译:

手工评分和自动化写作筛选方法的成本分析和成本效益

尽管研究人员已经调查了基于书面表达课程的措施 (WE-CBM) 的技术充分性和可用性,但不同评分方法的经济影响在很大程度上被忽略了。缺乏此类知识可能会破坏资源的有效分配,并导致采用次优措施来识别有写作成绩不佳风险的学生。因此,我们使用成分方法来比较手动计算和自动评分方法的实施成本和成本效益。对来自一项研究的二次数据进行了数据分析,该研究评估了对 WE-CBM 样本进行评分的定量方法的预测有效性和诊断准确性。调查结果表明,自动化方法比手动计算方法提供了更经济的解决方案;对于自动评分,使用免费的 writeAlizer R 软件包时效果更强,而对于手动计算的评分,更简单的 WE-CBM 指标比更复杂的指标成本更低。当每学年的教室、学生和评估次数增加时,敏感性分析证实了自动评分的相对优势;同样,与其他方法相比,writeAlizer 对成分变化的敏感度较低。最后,成本效益比的可视化说明 writeAlizer 在实施成本和诊断准确性之间提供了最佳平衡,其次是复杂的手动计算指标和专有的自动化程序。

更新日期:2022-03-28
down
wechat
bug