Psychological Test and Assessment Modeling, Volume 55, 2013 (2)
Extension of an iterative hybrid ordinal logistic regression/item response theory approach to detect and account for differential item functioning in longitudinal data
Shubhabrata Mukherjee, Laura E. Gibbons, Elizabeth Kristjansson & Paul K. Crane
Abstract | 
PDF of the full article
Increasing unidimensional measurement precision using a multidimensional item response model approach
Steffen Brandt & Brent Duckor
Abstract | PDF of the full article
Estimation of composite score classification accuracy using compound probability distributions
Chris Wheadon & Ian Stockford
Abstract | PDF of the full article
Linking PISA 2000 and PISA 2009: Implications of instrument design on measurement invariance
Eunike Wetzel & Claus H. Carstensen
Abstract | PDF of the full article
Multiattributive decision theory: who are the best?
Hartmann H. Scheiblechner
Abstract | PDF of the full article
In memoriam Hartmann H. Scheiblechner
Klaus D. KubingerPDF of the full article
Extension of an iterative hybrid ordinal logistic regression/item response theory approach to detect and account for differential item functioning in longitudinal data
Shubhabrata Mukherjee, Laura E. Gibbons, Elizabeth Kristjansson & Paul K. Crane
Abstract
Many constructs are measured using multi-item data collection instruments. Differential item functioning (DIF) occurs when construct-irrelevant covariates interfere with the relationship between construct levels and item responses. DIF assessment is an active area of research, and several techniques are available to identify and account for DIF in cross-sectional settings. Many studies include data collected from individuals over time; yet appropriate methods for identifying and accounting for items with DIF in these settings are not widely available. We present an approach to this problem and apply it to longitudinal Modified Mini-Mental State Examination (3MS) data from English speakers in the Canadian Study of Health and Aging. We analyzed 3MS items for DIF with respect to sex, birth cohort and education. First, we focused on cross-sectional data from a subset of Canadian Study of Health and Aging participants who had complete data at all three data collection periods. We performed cross-sectional DIF analyses at each time point using an iterative hybrid ordinal logistic regression/item response theory (OLR/IRT) framework. We found that item-level findings differed at the three time points. We then developed and applied an approach to detecting and accounting for DIF using longitudinal data in which covariation within individuals over time is accounted for by clustering on person. We applied this approach to data for the "entire dataset of English speaking participants including people who later dropped out or died. Accounting for longitudinal DIF modestly attenuated differences between groups defined by educational attainment. We conclude with a discussion of further directions for this line of research.
Key words: item response theory, differential item functioning, ordinal logistic Regression
Shubhabrata Mukherjee, PhD
Department of Medicine
University of Washington
325 9th Avenue, Box 359780
Seattle, WA 98121, USA
smukherj@u.washington.edu
Increasing unidimensional measurement precision using a multidimensional item response model approach
Steffen Brandt & Brent Duckor
Abstract
In recent years the estimation of unidimensional abilities for instruments with subtests has been a focus of item response theory. Different hierarchical models, which assume a common unidimensional latent trait and several subtest specific latent traits, have been proposed in order to cope with local item dependencies due to subtests. In contrast to these models, the generalized subdimension model (GSM) allows for the estimation of a latent mean ability based on multidimensional latent traits. Examining a small data set (n=72) this article examines the implicit weighting of the unidimensional model in contrast to the explicit weighting of the GSM to improve measurement precision.
Key words: item response theory, local item dependence, generalized subdimension model, multidimensionality, hierarchical models, instrument Validation
Steffen Brandt, PhD
Ebereschenweg 28
24161 Altenholz, Germany
steffen.brandt@artofreduction.com
Estimation of composite score classification accuracy using compound probability distributions
Chris Wheadon & Ian Stockford
Abstract
Presented is a demonstration of an intuitively simple, flexible and computationally inexpensive approach to estimating classification accuracy indices for composite score scales formed from the aggregation of performance on two or more assessments. This approach uses a two stage application of the polytomous extension of the Lord-Wingersky recursive algorithm and can be driven by any IRT model with desired simplicity or required complexity to best represent the properties of the tests. The approach is demonstrated using operational data from a high stakes mathematics qualification which is formed from two tests administered on distinct occasions. To provide the simplest representation of a test containing both dichotomous and polytomous items, the partial credit model is applied to model behaviour on the two tests. As an extension to this, a testlet model is applied to allow joint calibration of parameters from both tests. This model provides more information to the calibration process at the expense of some added computational complexity. Further to this, the potential application of this approach in the absence of operational data is investigated using a comparison of simulated data to the observed data.
Key words: Classification accuracy, IRT, composite scores
Chris Wheadon, PhD
Centre for Education Research and Policy
AQA, Stag Hill House, Guildford
Surrey, GU2 7XJ, UK
cwheadon@aqa.org.uk
Linking PISA 2000 and PISA 2009: Implications of instrument design on measurement invariance
Eunike Wetzel & Claus H. Carstensen
Abstract
An important pre-requisite of trend analyses in large scale educational assessments is the measurement invariance of the testing instruments across cycles. This paper investigates the measurement invariance of the PISA 2000 and PISA 2009 reading instruments using Item Response Theory models. Links between the PISA 2000 and PISA 2009 instruments were analyzed using data from a sample tested in 2009 which took both the PISA 2000 and PISA 2009 instruments and additionally using part of the German PISA 2000 sample. Model fit comparisons showed that the instruments are not measurement invariant and that some link items show large differences in item difficulty. Position effects may explain some of these differences and may also influence the size of the link error.
Key words: PISA, measurement invariance, linking, link error, position effects
Eunike Wetzel, Dipl.-Psych.
Department of Psychology and Methods of Educational Research
Otto-Friedrich-University Bamberg
96045 Bamberg, Germany
eunike.wetzel@uni-bamberg.de
Multiattributive decision theory: who are the best?
Hartmann H. Scheiblechner†
Abstract
A rank ordering of preference of multiattributive choice alternatives is suggested. The choice alternatives are characterised by several attributes (dimensions) which themselves are assumed to be given by strict partial orders (rank orders with possibly ties, or "rating scales). A dominance relation is defined on the alternatives: an alternative dominates another if it is at least as good as the other in all dimensions and strictly superior in at least one dimension. The result is a multidimensional partial order. The problem is to choose a single best or a given number of best alternatives from the choice set. The solution must not involve comparisons of ranks of different dimensions, if the decision maker is a single individual, or of rank orders of different individuals, if the decision maker is a social group (social choice function). The (modified) percentile rank score (Scheiblechner, 2002, 2003) is suggested as scaling function. The performance of the (modified) percentile score is illustrated by the examples of the results of the competitors of the decathlon of Olympic Games at Beijing 2007 and World Championships at Berlin 2009.
Key words: Decision theory; ranking; scoring; d-dimensional isotonic probabilistic model
Correspondence concerning this article should be addressed to:
Klaus D. Kubinger, Ph.D.
c/o Division of Psychological Assessment and Applied Psychometrics
University of Vienna
Faculty of Psychology, Germany
klaus.kubinger@univie.ac.at
