NEWSBÜCHERJOURNALEONLINE-SHOP



 

Sie befinden sich hier: JOURNALE » Psychological Test and Assessment Modeling » Currently available » Inhalt lesen

« zurück

Psychological Test and Assessment Modeling

» Psychological Test and Assessment Modeling Online-Shop...


2021-1

Editorial: Necessity for in-depth research work on Psychological Test and Assessment Modeling
Klaus D. Kubinger
PDF of the full article


Cognitive Components of Writing in a Second Language:An Analysis with the Linear Logistic Test Model
Farshad Effatpanah, Purya Baghaei
PDF of the full article


Determining the Number of Factors in Exploratory Factor Analysis – The Performance of Tests of Relative Model Fit
Kay Brauer, Jochen Ranger
PDF of the full article


Measuring Rater Centrality Effects in Writing Assessment: A Bayesian Facets Modeling Approach
Thomas Eckes & Kuan-Yu Jin
PDF of the full article


On the Performance of Multiple-Indicator Correlated Traits-Correlated (Methods – 1) Models
Christian Geiser & Trenton G. Simmons
PDF of the full article


Structural Validity of Overclaiming Scores: analysing PISA 2012 data
Marek Muszyński, Artur Pokropek, Tomasz Żółtak
PDF of the full article


 

Editorial: Necessity for in-depth research work on Psychological Test and Assessment Modeling

 

Klaus D. Kubinger
c/o Faculty of Psychology
University of Vienna
Liebiggasse 5
A-1010 Vienna (Austria)
klaus.kubinger@univie.ac.at


 


Cognitive Components of Writing in a Second Language: An Analysis with the Linear Logistic Test Model

 

Abstract
Writing in a second/foreign language (L2) is a demanding task for L2 writers because it calls for multiple language abilities and (meta)cognitive knowledge. Research investigating the (meta)cognitive processes involved in composing in L2 have emphasized the complex and multidimensional nature of L2 writing with many underlying (meta)cognitive components. However, it is still
unclear what factors or components are involved in composing in L2. Employing correlational and qualitative approaches and through the modeling of L2 writing proficiency, previous studies could not offer adequate evidence for the exact nature of such components. This study aimed at examining the underlying cognitive operations of L2 writing performance using an IRT-based cognitive
processing model known as linear logistic test model (LLTM). To achieve this, the performance of 500 English as a foreign language (EFL) students on a writing task was analyzed. Five cognitive processes underlying L2 writing were postulated: content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. The results of the likelihood ratio test
showed that the Rasch model fits significantly better than the LLTM. The correlation coefficient between LLTM and Rasch model item parameters was .85 indicating that about 72 % of variance in item difficulties can be explained by the five postulated cognitive operations. LLTM analyses also revealed that vocabulary and content are the most difficult processes to use and grammar is
the easiest. More importantly, the results showed that it is possible to envisage a model for L2 writing with reference to a set of subskills or attributes.

Keywords: L2 Writing Attributes, Q-matrix, Linear Logistic Test Model (LLTM)

Farshad Effatpanah, 
English Department,
Islamic Azad University of Mashhad, 
Ostad Yusofi St., 
Mashhad, 91871, Iran 
farshadefp@gmail.com


 

Determining the Number of Factors in Exploratory Factor Analysis – The Performance of Tests of Relative Model Fit

 

Abstract
The number of factors in exploratory factor analysis is often determined with tests of model fit. Such tests can be employed in two different ways: Tests of global fit are used to compare factor models with increasing number of factors against a saturated model whereas tests of relative fit compare factor models against models with one additional factor. In both approaches, the number of factors is determined by choosing the simplest model that is not rejected by the test of model fit. Hayashi, Bentler, and Yuan (2007) recommend using tests of global fit because the tests of relative model fit tend to overfactoring. We investigate the performance of the tests of relative model fit. Overfactoring is prevented by using either a bootstrap implementation or a modification of the standard tests. The modification consists in testing each model against a restricted alternative that is identified under the null hypothesis. Simulation studies suggest that our tests of relative model fit perform well. Both implementations adhere to the nominal Type-I error rate closely and are more powerful than the tests of global fit. The application of the tests is illustrated in an empirical
example.

Keywords: Exploratory Factor Analysis, Factor model, Fit tests; Model fit

Kay Brauer, 
Psychological Assessment and Differential Psychology, 
Department of Psychology, 
Martin-Luther-University Halle-Wittenberg, 
Halle, Germany

kay.brauer@psych.uni-halle.de


 

Measuring Rater Centrality Effects in Writing Assessment: A Bayesian Facets Modeling Approach

 

Abstract
Rater effects such as severity/leniency and centrality/extremity have long been a concern for researchers and practitioners involving human raters in performance assessments. In the present research, a facets modeling approach advanced by Jin and Wang (2018) was adopted to account for both rater severity and centrality effects in a writing assessment context. In two separate studies, raters scored examinees’ writing performances on a set of criteria using a four-category rating scale. Rater severity and centrality parameters were estimated building on Bayesian Markov chain Monte
Carlo methods implemented in the freeware JAGS run from within the R environment. The findings revealed that (a) raters differed in their severity and centrality estimates, (b) rater severity and centrality estimates were only moderately correlated (Study 1) or uncorrelated with each other (Study 2), (c) centrality effects had a demonstrable impact on examinee rank orderings, and (d) statistical
indices of rater centrality derived from severity-only facets models (rater infit, residual-expected correlation, and standard deviation of rater-specific thresholds) correlated with centrality estimates much as predicted. The discussion focuses on implications for the analysis of rating quality in performance assessments.

Keywords: rater effects, rater centrality, facets models, rater-mediated assessment, Bayesian statistics, MCMC estimation

Thomas Eckes, PhD, 
TestDaF Institute,
University of Bochum, 
Universitätsstr. 134, 
44799 Bochum, Germany
thomas.eckes@gast.de


 

On the Performance of Multiple-Indicator Correlated Traits-Correlated (Methods – 1) Models

 

Abstract
We examined the performance of two versions of the multiple-indicator correlated traits-correlated (methods – 1) [CT-C(M – 1)] model (Eid et al., 2008) in terms of convergence, improper solutions, parameter bias, standard error bias, and power to detect misspecified models. We also studied whether Yuan et al.’s (2015) correction procedure for the maximum likelihood chi-square model fit test yields accurate Type-I error rates and adequate power for these models. The models performed well except for underestimated standard errors for some parameters in specific small-sample conditions. Yuan et al.’s (2015) chi-square correction worked well for correctly specified models but showed limited power to detect misspecified models in small-sample, low-reliability conditions. We recommend that researchers using these models in smaller samples select highly reliable indicators.

Keywords: method effects, multitrait-multimethod (MTMM) analysis, multiple-indicator CFAMTMM models, CT-C(M – 1) model, model-size effect.

Christian Geiser, 
Department of Psychology,
Utah State University, 
2810 Old Main Hill, 
Logan, UT 84322-2810 
christian.geiser@usu.edu


 

Structural Validity of Overclaiming Scores: analysing PISA 2012 data

 

Abstract
Overclaiming technique is a promising tool that can account for self-assessment imprecision, improve cross-country comparability and screen for fakers in high-stakes contexts. Despite the rising popularity of the overclaiming technique - evidenced by an increasing number of papers, citations and versions of the method - very little is known about the internal structure of scales designed to measure overclaiming tendencies. It is especially worrisome, as internal structure is one of the main sources of construct validity and its coherence to the assumed theory vouches for scores’ interpretability and validity. We aim to fill in this research gap and use the obtained results to comment on the validity of using overclaiming technique’s scores in research practice, where a two-factorial structure of the tool is assumed. To this end, we analyse the PISA 2012 overclaiming scale’s internal structure by applying confirmatory multilevel factor analysis. Our results suggest that items in the PISA overclaiming scale cannot be simply interpreted as reals (construct variance) and foils (bias variance) as both types of items measure both types of variance. We also show that the simple ontic status of an item is not enough to guarantee intended measurement characteristics, namely that real items only measure genuine knowledge and that foil items solely capture a tendency to overclaim. The obtained results are used not only to reflect on overclaiming technique’s internal structure but also to give advice on constructing such scales in the future.

Keywords: overclaiming technique, internal structure, factor analysis, construct validity, PISA.

Dr. Marek Muszyński, 
Institute of Philosophy and Sociology, 
Polish Academy of Sciences, ul. 
Nowy Świat 72, 
00-330 Warszawa, Poland
marek.muszynski@ifispan.edu.pl


 


Psychological Test and Assessment Modeling
Volume 63 · 2021 · Issue 1

Pabst, 2021
ISSN 2190-0493 (Print)
ISSN 2190-0507 (Internet)

» Psychological Test and Assessment Modeling Online-Shop...





alttext