Addressing Reliability Problems in the Portfolio Assessment of College Writing |
| |
| Abstract: | In research and development designed to assess the writing skills of third-year college students, the University of Wisconsin Verbal Assessment Project developed and tested procedures for assessing writing portfolios from students in courses representing each college in the university. Following the work of Britton (1970) and 1:he National Assessment of Education Progress, we defined expository writing as sustained reflection in which the writer focuses and processes information to various degrees. Basing our work on this construct, we assessed writing samples in each portfolio in terms of both degree of reflection and extent of text elaboration. Results of two studies are presented. In Study 1, raters scored each text from a given portfolio before rating texts in the next portfolio. Reliability estimates were low to moderate for both scores. In follow-up Study 2, involving a comparable group of students, several changes were made to improve reliability: (a) Raters scored all texts written in response to a given prompt or assignment within a class before moving to the next set of texts; and (b) each time readers dealt with a new task, they read several examples together, coming to agreement about how various texts were to be rated. Estimates of reliability for both scores were somewhat higher and suggest that the modifications improved reliabilities. Results demonstrate that adequate reliability should be expected if texts are rated by task across portfolios within classes. Based on these findings, we contend that, because writing normally varies by topic, genre, and other variables, writing portfolios are better characterized by scores for each piece than by a single writing-skill score. |
| |
| Keywords: | |
|
|
