Psychological Test and Assessment Modeling, Volume 53, 2011 (4)
Bipolar items for the measurement of personal optimism instead of unipolar items
Karl Schweizer, Wolfgang Rauch & Andreas Gold
Abstract | 
PDF of the full article
Item banking for C-tests: A polytomous Rasch modeling approach
Thomas Eckes
Abstract | PDF of the full article
A comparison of item response theory-based methods for examining differential item functioning in object naming test by language of assessment among older Latinos
Frances M. Yang, Kevin C. Heslin, Kala M. Mehta, Cheng-Wu Yang, Katja Ocepek-Welikson, Marjorie Kleinman, Leo S. Morales, Ron D. Hays, Anita L. Stewart, Dan Mungas, Richard N. Jones, Jeanne A. Teresi
Abstract | PDF of the full article
Special Topic - Part 2:
Methodological advances in psychological and educational testing
Guest Editor: Matthias von Davier
Standard setting in complex performance assessments: An approach aligned with cognitive diagnostic models
Robert W. Lissitz & Feifei Li
Abstract | PDF of the full article
Investigating the saltus model as a tool for setting standards
Karen Draney & Minjeong Jeon
Abstract | PDF of the full article
Exploring levels of performance using the mixture Rasch model for
standard setting
Hong Jiao, Robert W. Lissitz, George Macready, Shudong Wang & Senfeng Liang
Abstract | PDF of the full article
Bipolar items for the measurement of personal optimism instead of unipolar items
Karl Schweizer, Wolfgang Rauch & Andreas Gold
Abstract
A personal optimism scale composed of unipolar items is transformed into a scale composed of bipolar items. The transformation is performed because of the item wording problem that denotes the lack of homogeneity in scales composed of positively and negatively worded items. The original optimism scale for the assessment of personal optimism comprises eight unipolar items. It is transformed into a scale with four bipolar items by merging pairs of items. In confirmatory factor analysis the scale based on bipolar items showed an acceptable degree of good model fit. Furthermore, the personal optimism latent variables of the two types of items showed to be equivalent when investigating trait-specific equivalence.
Key words: optimism, dimensionality, item-wording problem, bipolar items, unipolar items
Karl Schweizer, PhD
Department of Psychology
Goethe University Frankfurt
Mertonstr. 17
60054 Frankfurt a. M., Germany
K.Schweizer@psych.uni-frankfurt.de
Item banking for C-tests: A polytomous Rasch modeling approach
Thomas Eckes
Abstract
C-tests are gap-filling tests widely used to assess general language proficiency for purposes of placement, screening, or provision of feedback to language learners. C-tests consist of several short texts in which parts of words are missing. Development, administration, and scoring of C-tests are particularly efficient when use is made of a calibrated item bank. Rasch measurement provides a powerful approach to item bank construction. Based on construing C-test texts as superitems, where item values correspond to the number of gaps in the text filled in correctly, two polytomous Rasch models were applied to analyze and evaluate a large set of texts: Andrichs (1978) rating scale model and Müllers (1987, 1999b) continuous rating scale model. The test construction phase comprised a total of 218 texts trialled in 27 independent samples, covering a total of 5,927 participants. Across samples, reliability indices ranged from .94 to .98. Texts showing unsatisfactory model fit or DIF were eliminated. The remaining texts were put on the same difficulty scale through a concurrent estimation procedure. Results clearly attested to the suitability of polytomous Rasch models to calibrate texts for purposes of item banking.
Key words: polytomous Rasch models, item banking, language testing, C-test, local item dependence
Thomas Eckes, PhD
TestDaF Institute
Massenbergstr. 13 b
44787 Bochum, Germany
thomas.eckes@testdaf.de
A comparison of item response theory-based methods for examining differential item functioning in object naming test by language of assessment among older Latinos
Frances M. Yang, Kevin C. Heslin, Kala M. Mehta, Cheng-Wu Yang, Katja Ocepek-Welikson, Marjorie Kleinman, Leo S. Morales, Ron D. Hays4 Anita L. Stewart, Dan Mungas, Richard N. Jones, Jeanne A. Teresi
Abstract
Object naming tests are commonly included in neuropsychological test batteries. Differential item functioning (DIF) in these tests due to cultural and language differences may compromise the validity of cognitive measures in diverse populations. We evaluated 26 object naming items for DIF due to Spanish and English language translations among Latinos (n=1,159), mean age of 70.5 years old (Standard Deviation (SD)±7.2), using the following four item response theory-based ap-proaches: Mplus/Multiple Indicator, Multiple Causes (Mplus/MIMIC; Muthén & Muthén, 1998-2011), Item Response Theory Likelihood Ratio Differential Item Functioning (IRTLRDIF/MULTILOG; Thissen, 1991, 2001), difwithpar/Parscale (Crane, Gibbons, Jolley, & van Belle, 2006; Muraki & Bock, 2003), and Differential Functioning of Items and Tests/MULTILOG (DFIT/MULTILOG; Flowers, Oshima, & Raju, 1999; Thissen, 1991). Overall, there was moderate to near perfect agreement across methods. Fourteen items were found to exhibit DIF and 5 items observed consistently across all methods, which were more likely to be answered correctly by individuals tested in Spanish after controlling for overall ability.
Key words: Item response theory, differential item functioning, object naming test, Hispanic/Latinos, Spanish
Frances M. Yang, PhD
Institute for Aging Research
Hebrew SeniorLife
Harvard Medical School
Department of Medicine Beth Israel Deaconess Medical Center
Division of Gerontology
1200 Centre St.
Boston, MA 02131, USA
francesyang@hsl.harvard.edu
Standard setting in complex performance assessments: An approach aligned with cognitive diagnostic models
Robert W. Lissitz & Feifei Li
Abstract
With the increased interest in student-level diagnostic information from multiple performance assessments, it becomes possible to create multivariate classifications of knowledge, skills and abilities (KSAs). In this paper, a systematic, multivariate and non-compensating standard setting approach, called the cognitive analytical approach (CAA), is proposed for performance assessment with complex tasks.
CAA is based on the framework of evidence-centered design (Mislevy, Steinberg, & Almond, 2003) that supports a chain of reasoning from design and development to delivery of an assessment. In CAA, the performance standards are established simultaneously with domain-modeling, test specifications, and item writing rather than after the assessment has been completed; the cut scores are evaluated iteratively along with the test design and development phases. CAA has the benefits of ensuring the validity of the performance standards, reducing the cognitive load of standard setting, including the complexity of the tasks, and facilitating the vertical articulation of KSAs. In this paper, we elucidate the theoretical and practical rationale of CAA and demonstrate its proce-dures and results with an illustrative example.
Key words: Standard setting, cognitive diagnostic models, analytical approach, evidence centered design, performance centered, multidimensional standards
Robert W. Lissitz, PhD
1229 Benjamin Building
University of Maryland
College Park, MD 20742, USA
RLissitz@umd.edu
Investigating the saltus model as a tool for setting standards
Karen Draney & Minjeong Jeon
Abstract
The saltus model (Wilson, 1989; Draney, Wilson, Glück, & Spiel, 2008) is a latent variable mix-ture model originally designed for detecting developmental discontinuities.
One difference between this model and many other latent class models is that a clear ordering of the latent classes is hypothesized, from lowest to highest proficiency. At higher classes, the relative difficulty of performance on associated sets of items decreases compared to lower sets of items and lower groups of students. Such an ordering could potentially prove useful in the process of setting standards.
Although this model has previously been used primarily for the exploration of developmental differences, this paper will investigate the possibility of using the saltus model as a tool to assist in standard setting. Illustrative analysis will be conducted with data that were used in a recent study of the Bookmark method for standard setting, and results will be compared.
Key words: saltus model, latent class models, standard settings
Karen Draney, PhD
Graduate School of Education
University of California
Berkeley, CA 94720, USA
kdraney@berkeley.edu
Exploring levels of performance using the mixture Rasch model for standard setting
Hong Jiao, Robert W. Lissitz, George Macready, Shudong Wang & Senfeng Liang
Abstract
This study proposes a method using the mixture Rasch model to set performance standards. The mixture Rasch model classifies examinees into qualitatively distinct latent groups based on the information in the response patterns and quantifies individual differences within-group utilizing a continuous latent trait.The goals of this study are twofold. One is to demonstrate an application of the mixture Rasch model to statistically validate the performance proficiency levels set by policy makers and reviewed by content experts. The other is to demonstrate how performance cut scores can be obtained based on the results from data analysis using the mixture Rasch model. In general, this study presents a mixture Rasch model-based approach for setting performance standards which is expected to facilitate the standard setting process by providing the data driven information re-lated to the policy specified performance levels and the performance cut scores set by the panelists.
Key words: mixture Rasch model, standard setting, proficiency classification, model-based classi-fication, performance standards
Hong Jiao, PhD
Department of Measurement, Statistics and Evaluation
1230B Benjamin Building
University of Maryland
College Park, MD 20742, USA
hjiao@umd.edu
