Selected issues regarding the application of the kappa statistic in assessing the reliability of measures to classify patients are discussed. Data are used to illustrate how the methods used to estimate reliability can influence the resulting coefficient. Suggestions for a more comprehensive approach to the study of the psychometric properties of measures of patient classification are then offered.