共查询到20条相似文献，搜索用时 15 毫秒
1.
Stefanie A. Wind 《Journal of Educational Measurement》2019,56(3):478-504
Numerous researchers have proposed methods for evaluating the quality of rater‐mediated assessments using nonparametric methods (e.g., kappa coefficients) and parametric methods (e.g., the many‐facet Rasch model). Generally speaking, popular nonparametric methods for evaluating rating quality are not based on a particular measurement theory. On the other hand, popular parametric methods for evaluating rating quality are often based on measurement theories such as invariant measurement. However, these methods are based on assumptions and transformations that may not be appropriate for ordinal ratings. In this study, I show how researchers can use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, to evaluate rating quality within the framework of invariant measurement without the use of potentially inappropriate parametric techniques. I use an illustrative analysis of data from a rater‐mediated writing assessment to demonstrate how one can use numeric and graphical indicators from MSA to gather evidence of validity, reliability, and fairness. The results from the analyses suggest that MSA provides a useful framework within which to evaluate rater‐mediated assessments for evidence of validity, reliability, and fairness that can supplement existing popular methods for evaluating ratings. 相似文献
2.
When cut scores for classifications occur on the total score scale, popular methods for estimating classification accuracy (CA) and classification consistency (CC) require assumptions about a parametric form of the test scores or about a parametric response model, such as item response theory (IRT). This article develops an approach to estimate CA and CC nonparametrically by replacing the role of the parametric IRT model in Lee's classification indices with a modified version of Ramsay's kernel‐smoothed item response functions. The performance of the nonparametric CA and CC indices are tested in simulation studies in various conditions with different generating IRT models, test lengths, and ability distributions. The nonparametric approach to CA often outperforms Lee's method and Livingston and Lewis's method, showing robustness to nonnormality in the simulated ability. The nonparametric CC index performs similarly to Lee's method and outperforms Livingston and Lewis's method when the ability distributions are nonnormal. 相似文献
3.
Randall David Penfield 《Educational Measurement》2014,33(1):36-48
A polytomous item is one for which the responses are scored according to three or more categories. Given the increasing use of polytomous items in assessment practices, item response theory (IRT) models specialized for polytomous items are becoming increasingly common. The purpose of this ITEMS module is to provide an accessible overview of polytomous IRT models. The module presents commonly encountered polytomous IRT models, describes their properties, and contrasts their defining principles and assumptions. After completing this module, the reader should have a sound understating of what a polytomous IRT model is, the manner in which the equations of the models are generated from the model's underlying step functions, how widely used polytomous IRT models differ with respect to their definitional properties, and how to interpret the parameters of polytomous IRT models. 相似文献
4.
Drawing valid inferences from item response theory (IRT) models is contingent upon a good fit of the data to the model. Violations of model‐data fit have numerous consequences, limiting the usefulness and applicability of the model. This instructional module provides an overview of methods used for evaluating the fit of IRT models. Upon completing this module, the reader will have an understanding of traditional and Bayesian approaches for evaluating model‐data fit of IRT models, the relative advantages of each approach, and the software available to implement each method. 相似文献
5.
Roy Levy 《Educational Measurement》2020,39(1):94-95
In this digital ITEMS module, Dr. Roy Levy describes Bayesian approaches to psychometric modeling. He discusses how Bayesian inference is a mechanism for reasoning in a probability-modeling framework and is well-suited to core problems in educational measurement: reasoning from student performances on an assessment to make inferences about their capabilities more broadly conceived, as well as fitting models to characterize the psychometric properties of tasks. The approach is first developed in the context of estimating a mean and variance of a normal distribution before turning to the context of unidimensional item response theory (IRT) models for dichotomously scored data. Dr. Levy illustrates the process of fitting Bayesian models using the JAGS software facilitated through the R statistical environment. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as education, psychology, sociology, political science, business, health, and other social sciences. It contains audio-narrated slides, diagnostic quiz questions, and data-based activities with video solutions as well as curated resources and a glossary. 相似文献
6.
The present study conducted a systematic review of the item response theory (IRT) literature in language assessment to investigate the conceptualization and operationalization of the dimensionality of language ability. Sixty-two IRT-based studies published between 1985 and 2020 in language assessment and educational measurement journals were first classified into two categories based on a unidimensional and multidimensional research framework, and then reviewed to examine language dimensionality from technical and substantive perspectives. It was found that 12 quantitative techniques were adopted to assess language dimensionality. Exploratory factor analysis was the primary method of dimensionality analysis in papers that had applied unidimensional IRT models, whereas the comparison modeling approach was dominant in the multidimensional framework. In addition, there was converging evidence within the two streams of research supporting the role of a number of factors such as testlets, language skills, subskills, and linguistic elements as sources of multidimensionality, while mixed findings were reported for the role of item formats across research streams. The assessment of reading, listening, speaking, and writing skills was grounded within both unidimensional and multidimensional framework. By contrast, vocabulary and grammar knowledge was mainly conceptualized as unidimensional. Directions for continued inquiry and application of IRT in language assessment are provided. 相似文献
7.
Investigating the fit of a parametric model plays a vital role in validating an item response theory (IRT) model. An area that has received little attention is the assessment of multiple IRT models used in a mixed-format test. The present study extends the nonparametric approach, proposed by Douglas and Cohen (2001), to assess model fit of three IRT models (three- and two-parameter logistic model, and generalized partial credit model) used in a mixed-format test. The statistical properties of the proposed fit statistic were examined and compared to S-X2 and PARSCALE’s G2. Overall, RISE (Root Integrated Square Error) outperformed the other two fit statistics under the studied conditions in that the Type I error rate was not inflated and the power was acceptable. A further advantage of the nonparametric approach is that it provides a convenient graphical inspection of the misfit. 相似文献
8.
《教育实用测度》2013,26(2):135-143
Over the past several years, item response theory (IRT) has had a dramatic impact on the field of educational measurement. At the theoretical level and for educational researchers, it has shown itself to have certain advantages over classical theory both in terms of the research questions it can address and in terms of the applications it can support. For these benefits to be realized by the measurement practitioner, however, two major difficulties must be confronted: (a) the enormous complexity of IRT theory and procedures and (b) the lack of robustness of IRT procedures to violation of assumptions. 相似文献
9.
Qingping He 《Educational research; a review for teachers and all concerned with progress in education》2013,55(1):89-112
Background:?Although on-demand testing is being increasingly used in many areas of assessment, it has not been adopted in high stakes examinations like the General Certificate of Secondary Education (GCSE) and General Certificate of Education Advanced level (GCE A level) offered by awarding organisations (AOs) in the UK. One of the major issues with on-demand testing is that some of the methods used for maintaining the comparability of standards over time in conventional testing are no longer available and the development of new methods is required. Purpose:?This paper proposes an item response theory (IRT) framework for implementing on-demand testing and maintaining the comparability of standards over time for general qualifications, including GCSEs and GCE A levels, in the UK and discusses procedures for its practical implementation. Sources of evidence:?Sources of evidence include literature from the fields of on-demand testing, the design of computer-based assessment, the development of IRT, and the application of IRT in educational measurement. Main argument:?On-demand testing presents many advantages over conventional testing. In view of the nature of general qualifications, including the use of multiple components and multiple question types, the advances made in item response modelling over the past 30 years, and the availability of complex IRT analysis software systems, coupled with increasing IRT expertise in awarding organisations, IRT models could be used to implement on-demand testing in high stakes examinations in the UK. The proposed framework represents a coherent and complete approach to maintaining standards in on-demand testing. The procedures for implementing the framework discussed in the paper could be adapted by people to suit their own needs and circumstances. Conclusions:?The use of IRT to implement on-demand testing could prove to be one of the viable approaches to maintaining standards over time or between test sessions for UK general qualifications. 相似文献
10.
Competence assessment can support evidence-based teaching and learning. Depending on its specific purpose, competence assessment can be applied at the individual, the classroom or school level as well as on a national level. Large-scale-assessments are typical examples for system monitoring. Competence assessment on the individual level can provide important information with respect to individual qualification and learning outcomes. Typically, competence assessment uses psychometric models from Item-Response-Theory (IRT). However, until present, competence assessment still struggles to tap its full potential regarding feedback and individual support. Teachers often have difficulties to interpret results from competence assessments that are based on IRT. Thus, it is crucial to support teachers in using these results to improve classroom teaching. One of the main challenges of educational measurement is to align everyday classroom teaching with formative and summative assessments and to establish competence models as a common ground. Process-orientated, formative assessment can inform criterion-based feedback and, consequently, promote teaching on the basis of educational standards that were defined on the system level. However, more work needs to be done with respect to research on assessments as well as their communication and implementation into teaching practice. 相似文献
11.
12.
13.
Detection of Differential Item Functioning with Nonlinear Regression: A Non‐IRT Approach Accounting for Guessing 
下载免费PDF全文
下载免费PDF全文 In this article we present a general approach not relying on item response theory models (non‐IRT) to detect differential item functioning (DIF) in dichotomous items with presence of guessing. The proposed nonlinear regression (NLR) procedure for DIF detection is an extension of method based on logistic regression. As a non‐IRT approach, NLR can be seen as a proxy of detection based on the three‐parameter IRT model which is a standard tool in the study field. Hence, NLR fills a logical gap in DIF detection methodology and as such is important for educational purposes. Moreover, the advantages of the NLR procedure as well as comparison to other commonly used methods are demonstrated in a simulation study. A real data analysis is offered to demonstrate practical use of the method. 相似文献
14.
Sandip Sinharay 《Educational Measurement》2016,35(3):38-54
Data mining methods for classification and regression are becoming increasingly popular in various scientific fields. However, these methods have not been explored much in educational measurement. This module first provides a review, which should be accessible to a wide audience in education measurement, of some of these methods. The module then demonstrates using three real‐data examples that these methods may lead to an improvement over traditionally used methods such as linear and logistic regression in educational measurement. 相似文献
15.
This inquiry is an investigation of item response theory (IRT) proficiency estimators’ accuracy under multistage testing (MST). We chose a two‐stage MST design that includes four modules (one at Stage 1, three at Stage 2) and three difficulty paths (low, middle, high). We assembled various two‐stage MST panels (i.e., forms) by manipulating two assembly conditions in each module, such as difficulty level and module length. For each panel, we investigated the accuracy of examinees’ proficiency levels derived from seven IRT proficiency estimators. The choice of Bayesian (prior) versus non‐Bayesian (no prior) estimators was of more practical significance than the choice of number‐correct versus item‐pattern scoring estimators. The Bayesian estimators were slightly more efficient than the non‐Bayesian estimators, resulting in smaller overall error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for low‐ and high‐performing examinees. 相似文献
16.
Stephen Case 《Assessment & Evaluation in Higher Education》2007,32(3):285-299
A reconfigured and realigned system of assessment feedback was implemented with undergraduates taking criminology modules at Swansea University. The reformulated system integrated explicit engagement with assessment criteria in feedback given on an electronic template form with the use of a statement bank and the offer of follow‐up, feedback consolidation meetings with the tutor. Student module evaluation feedback questionnaires demonstrated a positive qualitative impact upon students' ratings of their awareness and understanding of module learning outcomes, assessment criteria and required levels of learning; moving the focus of assessment feedback from teaching to (higher‐order) learning. Substantive qualitative impact has also been evidenced by departmental adoption of the reconfigured assessment process as standard. Quantitative impact was evidenced in the form of improvements in assessment performance within‐modules and within students from 2004–2005 to 2005–2006. The educational implications of the study findings are discussed. 相似文献
17.
As item response theory has been more widely applied, investigating the fit of a parametric model becomes an important part of the measurement process. There is a lack of promising solutions to the detection of model misfit in IRT. Douglas and Cohen introduced a general nonparametric approach, RISE (Root Integrated Squared Error), for detecting model misfit. The purposes of this study were to extend the use of RISE to more general and comprehensive applications by manipulating a variety of factors (e.g., test length, sample size, IRT models, ability distribution). The results from the simulation study demonstrated that RISE outperformed G2 and S‐X2 in that it controlled Type I error rates and provided adequate power under the studied conditions. In the empirical study, RISE detected reasonable numbers of misfitting items compared to G2 and S‐X2, and RISE gave a much clearer picture of the location and magnitude of misfit for each misfitting item. In addition, there was no practical consequence to classification before and after replacement of misfitting items detected by three fit statistics. 相似文献
18.
Item analysis is an integral part of operational test development and is typically conducted within two popular statistical frameworks: classical test theory (CTT) and item response theory (IRT). In this digital ITEMS module, Hanwook Yoo and Ronald K. Hambleton provide an accessible overview of operational item analysis approaches within these frameworks. They review the different stages of test development and associated item analyses to identify poorly performing items and effective item selection. Moreover, they walk through the computational and interpretational steps for CTT‐ and IRT‐based evaluation statistics using simulated data examples and review various graphical displays such as distractor response curves, item characteristic curves, and item information curves. The digital module contains sample data, Excel sheets with various templates and examples, diagnostic quiz questions, data‐based activities, curated resources, and a glossary. 相似文献
19.
Patrick O. Monahan Won-Chan Lee Robert D. Ankenmann 《Journal of Educational Measurement》2007,44(3):211-225
A Monte Carlo simulation technique for generating dichotomous item scores is presented that implements (a) a psychometric model with different explicit assumptions than traditional parametric item response theory (IRT) models, and (b) item characteristic curves without restrictive assumptions concerning mathematical form. The four-parameter beta compound-binomial (4PBCB) strong true score model (with two-term approximation to the compound binomial) is used to estimate and generate the true score distribution. The nonparametric item-true score step functions are estimated by classical item difficulties conditional on proportion-correct total score. The technique performed very well in replicating inter-item correlations, item statistics (point-biserial correlation coefficients and item proportion-correct difficulties), first four moments of total score distribution, and coefficient alpha of three real data sets consisting of educational achievement test scores. The technique replicated real data (including subsamples of differing proficiency) as well as the three-parameter logistic (3PL) IRT model (and much better than the 1PL model) and is therefore a promising alternative simulation technique. This 4PBCB technique may be particularly useful as a more neutral simulation procedure for comparing methods that use different IRT models. 相似文献
20.
Tarah Temen Nadia Jaramillo Cherrez Shannon Coleman 《Journal of Food Science Education》2020,19(3):173-182
Understanding safe food practice is important for home‐based food operators to prevent foodborne illness. Earlier work has found that home‐based food operators lack food safety knowledge and may benefit from training that is specifically tailored to their needs. Unfortunately, home‐based food operators may be deterred from enrolling in traditional educational formats due to their busy schedules. The objective of this study was to pilot and evaluate the effectiveness of an online food safety education module for home‐based food operators in Iowa through three learning assessments. Twenty‐one participants enrolled in a blended workshop in which participants completed one online module before attending a face‐to‐face session where they completed the remaining five modules. The effectiveness of the online module was measured by examining the first‐attempt average scores on learning assessments, the number of assessment attempts required to achieve 100%, and the first‐attempt performance by question type. The three learning assessment tools resulted in first‐attempt averages of approximately 86%, 90%, and 83%, surpassing our standard of effectiveness of 75% and showing good potential for the online format. The learning assessment attempt numbers of 4.65, 1.67, and 3.81 showed difficulty with knowledge transfer for some topics. Comprehension and analysis‐style questions had first‐attempt success rates of approximately 85% and 88%, respectively. Scores on knowledge and application‐style questions were lower with first‐attempt success rates of approximately 80% and 75%, respectively. These findings were used to improve the first online module and guide the transition of the remaining five modules to the online format. 相似文献