Generalizability Theory for the Perplexed: A Practical Introduction and Guide: AMEE Guide No. 68

Med Teach. 2012;34(11):960-92. doi: 10.3109/0142159X.2012.703791.

Abstract

Background: Generalizability theory (G theory) is a statistical method to analyze the results of psychometric tests, such as tests of performance like the Objective Structured Clinical Examination, written or computer-based knowledge tests, rating scales, or self-assessment and personality tests. It is a generalization of classical reliability theory, which examines the relative contribution of the primary variable of interest, the performance of subjects, compared to error variance. In G theory, various sources of error contributing to the inaccuracy of measurement are explored. G theory is a valuable tool in judging the methodological quality of an assessment method and improving its precision.

Aim: Starting from basic statistical principles, we gradually develop and explain the method. We introduce tools to perform generalizability analysis, and illustrate the use of generalizability analysis with a series of common, practical examples in educational practice.

Conclusion: We realize that statistics and mathematics can be either boring or fearsome to many physicians and educators, yet we believe that some foundations are necessary for a better understanding of generalizability analysis. Consequently, we have tried, wherever possible, to keep the use of equations to a minimum and to use a conversational and slightly "off-serious" style.

MeSH terms

  • Data Interpretation, Statistical*
  • Educational Measurement / methods*
  • Guidelines as Topic
  • Humans
  • Reproducibility of Results
  • Research Design