ABSTRACT

The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a speaker’s videoed talk in the IELTS Speaking Test. We devised graphs of the ‘noticing trajectories’ of raters to compare the relative degrees of convergence and divergence between all raters at any point. The possible reasons for convergence/divergence were elicited using stimulated recall interviews. We found that raters’ noticing and rating tended to converge in relation to easily identifiable features such as idioms. However, for some stretches of speaker talk, there was clear divergence in terms of features noticed and rated by raters; in real IELTS tests, a background score-monitoring system deals with divergence. The research design enables delicate depiction of how the process of rater convergence/divergence unfolds over time. Applications are suggested in terms of examiner training, as well as for classroom discourse research and CA data workshops.

摘要

同样的二语口语表现可能会被不同的教师或评分者以截然不同的方式分析和评估。我们提出了一种新的、技术辅助的研究设计，它打开了调查评估者之间收敛和发散轨迹的大门。我们跟踪并记录了不同评分者何时注意到的内容，同时对雅思口语测试中演讲者的视频谈话进行评分。我们设计了评估者“注意轨迹”的图表，以比较所有评估者在任何时候的相对收敛和发散程度。使用刺激回忆访谈引出了收敛/发散的可能原因。我们发现评分者的注意和评分往往会集中于容易识别的特征，例如成语。然而，对于一些演讲者的谈话，在评分者注意到和评分的特征方面存在明显差异；在真实的雅思考试中，背景分数监控系统会处理分歧。研究设计能够精确描述评估者收敛/发散的过程如何随时间展开。建议在考官培训以及课堂话语研究和 CA 数据研讨会方面进行应用。