The accuracy of performance task scores after resolution of rater disagreement: A Monte Carlo study |
| |
| Authors: | James A Penny Robert L Johnson[Author vitae] |
| |
| Institution: | aCASTLE Worldwide, United States;bUniversity of South Carolina – Columbia, United States |
| |
| Abstract: | When multiple raters score a writing sample, on occasion they will award discrepant scores. To report a single score to the examinee, some method of resolving those differences must be applied to the ratings before an operational score can be reported. Several forms of resolving score discrepancies have been described in the literature. Initial studies of the various methods, however, have demonstrated that decisions about student performance may differ depending on the resolution method applied. Thus, studies are needed to investigate the quality of the scores associated with each model. To study score quality associated with each model, we conducted a Monte Carlo study and varied the factors associated with scoring and resolution to determine the conditions under which a particular resolution method might be superior. |
| |
| Keywords: | Performance assessment Discrepant score resolution Monte Carlo study Resolution of rater disagreement |
| 本文献已被 ScienceDirect 等数据库收录！ |
