NEWSBÜCHERJOURNALESHOP



 

Sie befinden sich hier: JOURNALE » Psychological Test and Assessment Modeling » Currently available » Inhalt lesen

« zurück

Psychological Test and Assessment Modeling

» Psychological Test and Assessment Modeling Online-Shop...


2018-1

PTAM 1-2018

Contents: Testing psychometric properties of the CFT 1-R for students with special educational needs

Jörg-Henrik Heine, Markus Gebhard, Susanne Schwab,
Phillip Neumann, Julia Gorges & Elke Wild

Contents  PDF of the full article

Sammlung  PDF of the full article

Testing psychometric properties of the CFT 1-R for students with special educational needs

Jörg-Henrik Heine, Markus Gebhard, Susanne Schwab, Phillip Neumann , Julia Gorges & Elke Wild
Abstract | PDF of the full article

Guest Editorial
Rater effects: Advances in item response modeling of human ratings  Part II

Thomas Eckes
 PDF of the full article

A tale of two models: Psychometric and cognitive perspectives on rater-mediated assessments using accuracy ratings
George Engelhard, Jr, Jue Wang, & Stefanie A. Wind
Abstract | PDF of the full article

 

Modeling rater effects using a combination of Generalizability Theory and IRT
Jinnie Choi & Mark R. Wilson
Abstract | PDF of the full article

 

Comparison of human rater and automated scoring of test takers? speaking ability and classification using Item Response Theory
Zhen Wang & Yu Sun
Abstract | PDF of the full article 

 

Item response models for human ratings: Overview, estimation methods, and implementation in R
Alexander Robitzsch & Jan Steinfeld
Abstract | PDF of the full article

 

 


Testing psychometric properties of the CFT 1-R for students with special educational needs
Jörg-Henrik Heine, Markus Gebhard, Susanne Schwab, Phillip Neumann, Julia Gorges & Elke Wild

Abstract

The Culture Fair Intelligence Test CFT 1-R (Weiß & Osterland, 2013) is one of the most used tests in Germany when diagnosing learning disabilities (LD). The test is constructed according to the classical test theory and provides age specific norms for students with LD in special schools. In our study, we analyzed the test results of 138 students in special schools and 166 students with LD in inclusive settings in order to test the measurement invariance between students with LD, who are educated in these two different educational settings. Data were analyzed within an IRT framework using a non-iterative approach for (item) parameter recovery. This approach parallels with the principle of limited information estimation, which allows for IRT analyses based on small datasets. Analyses for Differential Item Functioning (DIF) as well as a test for global and local model violations with regard to both subgroups were conducted. The results confirmed the assumption of measurement invariance across inclusive and exclusive educational settings for students with LD.

Key words: Measurement invariance, Rasch model, item parameter recovery, limited information estimation, learning disabilities

Jörg-Henrik Heine
Technical University of Munich
TUM School of Education
Centre for International Student Assessment (ZIB)
Arcisstr. 21
D-80333 München, Germany

joerg.heine@tum.de

top


Guest Editorial Rater effects: Advances in item response modeling of human ratings ? Part II
Thomas Eckes


Thomas Eckes, PhD
TestDaF Institute
University of Bochum
Universitätsstr. 134
44799 Bochum, Germany

thomas.eckes@testdaf.de

top


A tale of two models: Psychometric and cognitive perspectives on rater-mediated assessments using accuracy ratings
George Engelhard, Jr, Jue Wang, & Stefanie A. Wind
 

Abstract

The purpose of this study is to discuss two perspectives on rater-mediated assessments: psychometric and cognitive perspectives. In order to obtain high quality ratings in rater-mediated assessments, it is essential to be guided by both perspectives. It is also important that the specific models selected are congruent and complementary across perspectives. We discuss two measurement models based on Rasch measurement theory (Rasch, 1960, 1980) to represent the psychometric perspective, and we emphasize the Rater Accuracy Model (Engelhard, 1996, 2013). We build specific judgment models to reflect the cognitive perspective of rater scoring processes based on Brunswik's Lens model frame- work. We focus on differential rater functioning in our illustrative analyses. Raters who possess in- consistent perceptions may provide different ratings, and this may cause various types of inaccuracy. We use a data set that consists of the ratings of 20 operational raters and three experts of 100 essays written by Grade 7 students. Student essays were scored using an analytic rating rubric for two do- mains: (1) idea, development, organization, and cohesion; as well as (2) language usage and conven- tion. Explicit consideration of both psychometric and cognitive perspectives has important implica- tions for rater training and maintaining the quality of ratings obtained from human raters.

Keywords: Rater-mediated assessments, Rasch measurement theory, Lens model, Rater judgment, Rater accuracy


George Engelhard, Jr., Ph.D.
Professor of Educational Measurement and Policy
Quantitative Methodology Program, Department of Educational Psychology
325W Aderhold Hall
The University of Georgia Athens
Georgia 30602, U.S.A.

gengelh@uga.edu

top


Modeling rater effects using a combination of Generalizability Theory and IRT
Jinnie Choi & Mark R. Wilson

Abstract

Motivated by papers on approaches to combine generalizability theory (GT) and item response theory (IRT), we suggest an approach that extends previous research to more complex measure- ment situations, such as those with multiple human raters. The proposed model is a logistic mixed model that contains the variance components needed for the multivariate generalizability coeffi- cients. Once properly set-up, we can estimate the model by straightforward maximum likelihood estimation. We illustrate the use of the proposed method with a real multidimensional polytomous item response data set from classroom assessment that involved multiple human raters in scoring.
 

Keywords: generalizability theory, item response theory, rater effect, generalized linear mixed model


Jinnie Choi
Research Scientist at Pearson
221 River Street
Hoboken, NJ 07030

jinnie.choi@pearson.com

top


Comparison of human rater and automated scoring of test takers? speaking ability and classification using Item Response Theory
Zhen Wang & Yu Sun

Abstract

Automated scoring has been developed and has the potential to provide solutions to some of the obvious shortcomings in human scoring. In this study, we investigated whether SpeechRaterSM and a series of combined SpeechRater SM and human scores were comparable to human scores for an English language assessment speaking test. We found that there were some systematic patterns in the
five tested scenarios based on item response theory.

Keywords: SpeechRater, human scoring, item response theory, ability estimation and classification


Zhen (Jane) Wang
Educational Testing
Service, Senior Psychometrician
Psychometrics, Statistics and Data Sciences (PSDS)
Rosedale-Anrig,
Princeton, NJ, U.S.A

jwang@ets.org

 

 

top


Item response models for human ratings: Overview, estimation methods, and implementation in R
Alexander Robitzsch & Jan Steinfeld

Abstract

Item response theory (IRT) models for human ratings aim to represent item and rater characteristics by item and rater parameters. First, an overview of different IRT models (many-facet rater models, covariance structure models, and hierarchical rater models) is presented. Next, different estimation methods and their implementation in R software are discussed. Furthermore, suggestions on how
to choose an appropriate rater model are made. Finally, the application of several rater models in R is illustrated by a sample dataset.

Keywords: multiple ratings, many-facet rater model, hierarchical rater model, R packages, parameter estimation, item response models


Leibniz Institute for Science and Mathematics Education (IPN) at Kiel University
Kiel Germany


Alexander Robitzsch, PhD
IPN
Olshausenstraße 62
D-24118 Kiel, Germany

robitzsch@ipn.uni-kiel.de

 

 

 

 

top

 


 

 

 

 

 

 

 

 

 

» Psychological Test and Assessment Modeling Online-Shop...





alttext