Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists

Abstract

Introduction: Dentists educated outside the EU, EEA and Switzerland have the option to undergo proficiency testing to practice dentistry in Sweden. A standardised evidence-based procedure for proficiency tests is crucial. This paper explores if the standard setting of the theoretical examination in the Swedish proficiency test for foreign dentists is fit to differentiate candidates who meet the acceptable standard for licensure from those who do not, by analysis of inter-rater reliability and credibility of Angoff panels and the test's ability to differentiate candidates regarding learning outcomes and dental disciplines.

Materials and methods: In a cohort study design, test results were collected from seven examinations during 2018-2019 (n = 316), combined with Angoff judgements and demographic data from The National Board of Health and Welfare. Intraclass correlation, Pearson correlation and linear regression were calculated.

Results: All exams demonstrated substantial judge inter-rater reliability using the Angoff procedure (0.65-0.82). Item difficulty predicted by the expert panels showed significant positive correlations with the item difficulty by candidate results (0.69-0.77). Candidates who did not meet the requirements for pass mark in the theoretical test revealed a significantly reduced performance overall in professional qualifications (i.e., learning outcomes) and reduced performance within dental disciplines.

Conclusions: The results indicate that a modified Angoff method is a feasible, simple, and defensible method for setting the pass mark for the theoretical examinations. The standard setting appears to differentiate candidates who do not meet the requirements of the professional qualifications of the dental degree. Adjustments could be undertaken to further discriminate for competence within single dental disciplines.

Keywords: assessment; dentists; educational measurement; foreign-trained dentists; licensure and certification; standard setting.