This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments.

本研究采用混合方法来检查在美国一所大型大学的英语作为第二语言 (ESL) 写作分班考试为期一个学期的评估者认证计划中评估者的表现如何发展。从 2016 年到 2018 年，我们跟踪了三组新手评估者（n = 30) 在认证计划的四轮中。使用多方面的 Rasch 模型，根据评估者的一致性、评估者的一致性和评估者的严重程度来检查评估者的表现。对评级质量的这些测量估计进行了多变量分析，以检查评估者的表现是否以及如何在各轮中发生变化。对评估者对论文的评论进行了定性分析，以更深入地了解评估者如何随着时间的推移学习使用量表。定量结果显示，所有三组评分者的评分者表现都呈现非线性、三阶段的发展模式。这项研究的结果表明，评估者的发展类似于一个学习曲线，类似于一个人如何获得一门语言和其他技能。我们认为，了解评估者行为的发展模式不仅对理解评估者培训的有效性至关重要，而且对评估者认知和发展的调查也至关重要。我们还将讨论本研究对评估员培训写作评估所需的努力和期望的实际意义。