Comparing the effectiveness of self-paced and collaborative frame-of-reference training on rater accuracy in a large-scale writing assessment

There exists a large body of research on the effectiveness of rater training methods in the Industrial and Organizational Psychology literature. Far less research has been done on the effectiveness of rater training methods in large-scale writing assessments. The purpose of this dissertation is to compare the effectiveness of two widely-used rater training methodsself-paced and collaborative frame-of-reference trainingin the context of a large-scale, statewide writing assessment. Sixty-six raters were randomly assigned to the training methods. After training, all raters scored a common set of fifty representative essays. To determine raters accuracy on these essays, raters scores were compared to resolved expert scores and coded accurate (1) when the scores matched and inaccurate (0) otherwise. This approach was taken because over ninety-nine percent of these comparisons aligned either exactly or within one point. A series of logistic mixed models were then fitted to these binary data. Results suggested that the self-paced method was equivalent in effectiveness to the more time-intensive and costly collaborative method. Implications for large-scale writing assessments and suggestions for further research are discussed.