To evaluate the reliability and validity of examiner ratings of simulated clinical encounters (SCEs), 94 subjects representing four distinct groups in training, experience, and ability to deliver emergency medical care were administered SCEs by two examiners. Inter-rater reliabilities of performance on specific SCEs varied from 0.61 to 0.89, with an average of 0.79 for all SCEs in the library. Identified sources of examiner error in determining clinical competence included case-specific variability of candidate performance, differences in standards used by different examiners, a halo effect several different ratings were made, and random errors in determining exact rating scores. However, this research indicates that examiner ratings clearly discriminate groups with known differences in clinical competence and correlated well (0.83) with objective test scores.