As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. In the present study we seek first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, we examine the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, we examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

中文翻译：

---

培训和评分者差异对口语能力评估的影响

由于非母语语言的判断与社会偏见密切相关，因此口语能力评级容易因评级者背景和社会态度而出错。在本研究中，我们首先寻求估计归因于评估者背景和态度变量对新手评估者 L2 口语评估的方差。其次，我们研究了最低限度的培训在降低那些与特质无关的评估因素的效力方面的影响。因此，我们研究了评分者差异对 TOEFL iBT® 口语分数的相对影响。82 位未经培训的评分员对 TOEFL® 考生生成的 112 个语音样本进行了评判。调查结果显示，大约 20% 的未经培训的评分者的分数差异部分是由他们的背景和态度因素造成的。最强的预测因素是评分者自己的母语状态。然而，最少的在线培训显着降低了评估者背景和态度变量对高和低严重程度评估者子样本的影响。暗示表明，简短且用户友好的评估员培训课程有望减轻评估员的偏见，至少在短期内是这样。该程序可用于评估和应用语言学的其他相关领域。