Learners and educators in the health professions have called for more fine-grained information (subscores) from assessments, beyond a single overall test score. However, due to concerns over reliability, there have been limited uses of subscores in practice. Recent advances in latent class analysis have made contributions in subscore reporting by using diagnostic classification models (DCMs), which allow reliable classification of examinees into fine-grained proficiency levels (subscore profiles). This study examines the innovative and practical application of DCM framework to health professions educational assessments using retrospective large-scale assessment data from the basic and clinical sciences: National Board of Medical Examiners Subject Examinations in pathology (n = 2,006) and medicine (n = 2,351). DCMs were fit and analyzed to generate subscores and subscore profiles of examinees. Model fit indices, classification (reliability), and parameter estimates indicated that DCMs had good psychometric properties including consistent classification of examinees into subscore profiles. Results showed a range of useful information including varying levels of subscore distributions. The DCM framework can be a promising approach to report subscores in health professions education. Consistency of classification was high, demonstrating reliable results at fine-grained subscore levels, allowing for targeted and specific feedback to learners.
Keywords: latent variable; psychometric models; reliability; score reporting; subscores; validity.