Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques |
| |
| Authors: | Fu-Yuan Hsu Hahn-Ming Lee Tao-Hsing Chang Yao-Ting Sung |
| |
| Institution: | 1. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, No.43, Sec.4, Keelung Rd., Da’an Dist., Taipei City 106, Taiwan;2. Department of Computer Science and Information Engineering, National Kaohsiung University of Applied Sciences, No.415, Jiangong Rd., Sanmin Dist., Kaohsiung City 807, Taiwan;3. Department of Educational Psychology and Counseling, National Taiwan Normal University, No.162, Sec.1, Heping E. Rd., Da’an Dist., Taipei City 106, Taiwan;4. Research Center for Psychological and Educational Testing, National Taiwan Normal University, No.162, Sec.1, Heping E. Rd., Da’an Dist., Taipei City 106, Taiwan;5. Institute of Information Science, Academia Sinica, Taiwan, No.128, Sec.2, Academia Rd., Nankang Dist., Taipei City 115, Taiwan |
| |
| Abstract: | Pretesting is the most commonly used method for estimating test item difficulty because it provides highly accurate results that can be applied to assessment development activities. However, pretesting is inefficient, and it can lead to item exposure. Hence, an increasing number of studies have invested considerable effort in researching the automated estimation of item difficulty. Language proficiency tests constitute the majority of researched test topics, while comparatively less research has focused on content subjects. This paper introduces a novel method for the automated estimation of item difficulty for social studies tests. In this study, we explore the difficulty of multiple-choice items, which consist of the following item elements: a question and alternative options. We use learning materials to construct a semantic space using word embedding techniques and project an item's texts into the semantic space to obtain corresponding vectors. Semantic features are obtained by calculating the cosine similarity between the vectors of item elements. Subsequently, these semantic features are sent to a classifier for training and testing. Based on the output of the classifier, an estimation model is created and item difficulty is estimated. Our findings suggest that the semantic similarity between a stem and the options has the strongest impact on item difficulty. Furthermore, the results indicate that the proposed estimation method outperforms pretesting, and therefore, we expect that the proposed approach will complement and partially replace pretesting in future. |
| |
| Keywords: | Multiple-choice item Item difficulty estimation Cognitive processing model Semantic similarity Word embedding Machine learning |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
