Files
Abstract
Cognitive diagnostic models (CDMs) provide pedagogically relevant information in the form of a student profile of multiple binary categorizations of students into mastery or nonmastery statuses on latent traits called attributes. Federal educational accountability requires accountability measures to designate students into one of at least three ordinal levels of proficiency. To bridge the gap between educational accountability testing and pedagogical relevance, CDMs have been proposed for use as state accountability measures. This study examines aspects of mapping the multiple binary categorizations of the students profiles onto ordinal levels of proficiency. The setting of the ordinal levels takes place in a process called a standard setting, in which panels of experts are asked to categorize student profiles into levels of proficiency. We manipulated the number of profiles panelists viewed, the variability of the set of profiles panelists viewed, and the range of attributes mastered across the sets of profiles panelists viewed to determine if these factors impact the resulting cut point. Results indicate that the methods of standard setting are resilient to changes in the number of profiles that panelists view or the variability of profiles panelists view but are effected by the range of profiles panelistsview, indicated that while standard settings produce consistent results they lack objectivity and as arbitrary distinctions are difficult to justify their continued use in educational measurement.