A study of the variability between raters in scoring an oral clinical examination in psychiatry in the format of the certification by the Royal College of Physicians and Surgeons of Canada is reported. A video-taped examination from another centre was rated independently by academic psychiatrists, nonacademic psychiatrists and residents. Considerable inter-rater differences between and within these groups were found. In particular, the averaging of the marks of pairs of raters as occurs in the actual certification resulted in the outcome depending to a considerable degree on the chance pairings of raters. These findings support a number of previous studies and emphasize the need to train examiners and to develop clearer rating criteria.