With the increased interest in student-level diagnostic information from multiple performance assessments, it becomes possible to create multivariate classifications of knowledge, skills and abilities. Dres. Robert W. Lissitz and Feifei Li (University od Maryland/USA) propose a systematic, multivariate and non-compensating standard setting approach for performance assessment with complex tasks. The paper reg. "cognitive analytical approach" (CAA) appeared in Psychological Test and Assessment Modeling (formerly Psychology Science/Psychologische Beiträge).
"Cognitive analytical approach is based on the framework of evidence-centered design that supports a chain of reasoning from design and development to delivery of an assessment. In CAA, the performance standards are established simultaneously with domain-modeling, test specifications, and item writing rather than after the assessment has been completed; the cut scores are evaluated iteratively along with the test design and development phases. CAA has the benefits of ensuring the validity of the performance standards, reducing the cognitive load of standard setting, including the complexity of tasks, and facilitating the vertical articulation of knowledge, skills and abilities."
The authors elucidate the theoretical and practical rationale of CAA and demonstrate its procedures and results with an illustrative example.
"CAA can be regarded as complementary to the probabilistic diagnostic approach. They are both based on a certain kind of cognitive diagnostic framework, but through different classification procedures. However, it would be interesting to compare the results of the standard setting by human judgment with the model-driven classifications," Lissitz and Li remark.
Psychological Test and Assessment Modeling 4-2011
