Psychological Test and Assessment Modeling, Volume 54, 2012 (3)
The relative impact of persons, items, subtests, and academic background on performance on a language proficiency test
Hossein Karami
Abstract | 
PDF of the full article
Developing and validating an Academic Listening Questionnaire
Vahid Aryadoust, Christine C. M. Goh & Lee Ong Kim
Abstract | PDF of the full article
Examinee-centered standard setting for large-scale assessments: The prototype group method
Thomas Eckes
Abstract | PDF of the full article
Special topic:
Configural Frequency Analysis (CFA) and other non-parametrical statistical methods - Part II
Guest Editors: Mark Stemmler & Alexander von Eye
Configural Frequency Anlysis (CFA) and other non-parametrical statistical methods: Introduction to Special Topic (Part II)
Mark Stemmler & Alexander von Eye (Guest Editors)PDF of the full article
Bayesian Predictive Configural Frequency Analysis
Eduardo Gutiérrez-Peña
Abstract | PDF of the full article
Non-parametric methods - Tree and P-CFA - for the ecological evaluation and assessment of suitable aquatic habitats: A contribution to fish psychology
Andreas H. Melcher, Erwin Lautsch & Stefan Schmutz
Abstract | PDF of the full article
Continuity and patterns of externalizing and internalizing behavior problems in girls: A variable- and person-oriented study from preschool to youth age
Friedrich Lösel & Mark Stemmler
Abstract | PDF of the full article
The relative impact of persons, items, subtests, and academic background on performance on a languageproficiency test
Hossein Karami
Abstract
This study exploited generalizability theory to explore the impact of persons, items, subtests, and academic background on the dependability of the scores from a high-stakes language proficiency test, the University of Tehran English Proficiency Test (UTEPT). To this end and following Brown (1999), three questions were posed: 1. What are the distributional characteristics and CTT reliability of UTEPT test scores? 2. What are the relative contributions of persons, items, and subtests to the dependability of scores for each group and for all the groups combined? 3. What are the relative contributions of persons, items, subtests, academic background as well as their various interactions to the dependability of the scores when all groups are combined? To investigate the issues, 5795 examinees from four different academic backgrounds were selected from among all the participants who had taken the test in 2004. The results of the study indicated that the relative contributions of the facets were not stable across all groups, though highly similar. In addition, with academic background added as a facet, there was no significant interaction between items and fields, and the dependability of the scores did not decrease either. This result shows that background knowledge does not lead to bias in the UTEPT. This use of G-theory could be extended profitably to other measuring situations.
Key words: Generalizability theory, academic background, UTEPT, dependability, test bias
Hossein Karami, PhD
Department of English Language and Literature
Faculty of Foreign Language and Literatures
University of Tehran
Kargar Shomali Avenue
Tehran, Postcode/ZIP code: 14155-6553, Iran
hkarami@ut.ac.ir
Developing and validating an Academic Listening Questionnaire
Vahid Aryadoust, Christine C. M. Goh & Lee Ong Kim
Abstract
This article reports on the development and administration of the Academic Listening Self-rating Questionnaire (ALSA). The ALSA was developed on the basis of a proposed model of academic listening comprising six related components. The researchers operationalized the model, subjected items to iterative rounds of content analysis, and administered the finalized questionnaire to international ESL (English as a second language) students in Malaysian and Australian universities. Structural equation modeling and rating scale modeling of data provided content-related, substantive, and structural validity evidence for the instrument. The researchers explain the utility of the questionnaire for educational and assessment purposes.
Key words: academic listening, language testing, Rating Scale Model, structural equation modeling
Vahid Aryadoust
Centre for English Language Communication
10 Architecture Drive
Singapore 117511, National University of Singapore
elcsva@nus.edu.sg, vahidaryadoust@gmail.com
Examinee-centered standard setting for large-scale assessments: The prototype group method
Thomas Eckes
Abstract
This paper presents the prototype group method (PGM) of standard setting within the context of a large-scale language assessment project. The PGM combines a Rasch measurement approach to the analysis of examinee proficiency with the concept of prototypes drawn from research on human judgment and categorization. Experts first identify learners typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Based on the distributions of proficiency estimates for learner prototypes belonging to adjacent levels, cut scores are computed by means of a logistic regression procedure. These cut scores define the language proficiency level a particular examinee has achieved. Data from 39 independent samples of examinees (total N = 8,721) covering a range of German language proficiency levels are used to illustrate the PGM. Rasch analysis and logistic regression results corroborate the adequacy of this approach. The discussion focuses on the methods distinctive features, practical requirements of its implementation, and issues of cut-score validation.
Key words: standard setting, Rasch measurement, prototypes, large-scale assessment, language proficiency
Thomas Eckes, PhD
TestDaF Institute
Massenbergstr. 13 b
44787 Bochum, Germany
thomas.eckes@testdaf.de
Bayesian predictive Configural Frequency Analysis
Eduardo Gutiérrez-Peña
Abstract
Configural Frequency Analysis is a method for cell-wise inspection of cross-classifications. CFA searches for patterns of variable categories that occur either more often or less often than expected from a given base model. In this paper, we propose and discuss an alternative notion of types and antitypes that focuses on the likely values of the cell frequencies in future experiments, as opposed to the average values of such frequencies. The idea is developed from a Bayesian point of view.
Key words: Bayesian methods, Configural Frequency Analysis, predictive distribution
Eduardo Gutiérrez-Peña, PhD
Departamento de Probabilidad y Estadística
IIMAS-UNAM
Apartado Postal 20-726
01000 México, D.F., Mexico
eduardo@sigma.iimas.unam.mx
Non-parametric methods - Tree and P-CFA - for the ecological evaluation and assessment of suitable aquatic habitats: A contribution to fish psychology
Andreas H. Melcher, Erwin Lautsch & Stefan Schmutz
Abstract
This study analyses multidimensional spawning habitat suitability of the fish species "Nase (latin: Chondrostoma nasus). This is the first time non-parametric methods were used to better understand biotic habitat use in theory and practice. In particular, we tested (1) the Decision Tree technique, Chi-squared Automatic Interaction Detectors (CHAID), to identify specific habitat types and (2) Prediction-Configural Frequency Analysis (P-CFA) to test for statistical significance. The combination of both non-parametric methods, CHAID and P-CFA, enabled the identification, prediction and interpretation of most typical significant spawning habitats, and we were also able to determine non-typical habitat types, e.g., types in contrast to antitypes. The gradual combination of these two methods underlined three significant habitat types: shaded habitat, fine and coarse substrate habitat depending on high flow velocity. The study affirmed the importance for fish species of shading and riparian vegetation along river banks. In addition, this method provides a weighting of interactions between specific habitat characteristics. The results demonstrate that efficient river restoration requires re-establishing riparian vegetation as well as the open river continuum and hydro-morphological improvements to habitats.
Key words: River, fish, habitat, spawning, non-parametric analyses, Decision Tree technique (CHAID), Prediction-Configural Frequency Analysis (P-CFA)
Andreas H. Melcher, PhD
University of Natural Resources and Life Sciences Vienna
Max Emanuel-Strasse 17
1180 Vienna, Austria
andreas.melcher@boku.ac.at
Continuity and patterns of externalizing and internalizing behavior problems in girls: A variable- and person-oriented study from preschool to youth age
Friedrich Lösel & Mark Stemmler
Abstract
This article addresses the continuity and patterns of externalizing and internalizing behavior problems in girls from kindergarten to secondary school age and also relations to offending in adolescence. It is a sequel to a similar investigation of boys at the same age (Stemmler & Lösel, 2012). The sample consisted of 294 girls from the Erlangen-Nuremberg Development and Prevention Study. Behavior problems were measured by the Social Behavior Questionnaire at approximate child ages of 4.5 years (kindergarten educators as informants), 10.5 years (school teachers information) and 13.5 years (mothers information). The third assessment also contained a self report on juvenile delinquency. Both person-oriented and variable-oriented methods of data analysis were applied (i.e. correlations and Prediction-Configural Frequency Analysis).
The correlations between externalizing and internalizing problems at preschool age and in youth were mainly small, but in the same range as in the boys study. In contrast to the boys study no significant type of externalizing only problem behavior was observed. The externalizing problems at youth age were more related to internalizing problems than in the boys sample. There were no significant differences in juvenile delinquency between the various patterns of externalizing and internalizing problems in adolescence except for status offenses (e.g. truancy, substance misuse). Overall the results suggest similarities as well as differences between both genders, whereby internalizing problems in girls seem to play a stronger role for antisocial behavior than in boys. Potential content and methodological explanations for these findings are discussed.
Key words: Behavior problems in girls, externalizing behavior, internalizing behavior, juvenile delinquency, longitudinal research, Prediction-Configural Frequency Analysis
Friedrich Lösel, PhD
Institute of Criminology
University of Cambridge
Sidgwick Avenue
Cambridge, United Kingdom
fal23@cam.ac.uk
