Purpose: Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process.
Method: Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance.
Results: A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01).
Conclusions: Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.
Underrepresented Minorities are Underrepresented Among General Surgery Applicants Selected to Interview.J Surg Educ. 2019 Nov-Dec;76(6):e15-e23. doi: 10.1016/j.jsurg.2019.05.018. Epub 2019 Jun 5. J Surg Educ. 2019. PMID: 31175064
Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test.Teach Learn Med. 2020 May 19:1-14. doi: 10.1080/10401334.2020.1760104. Online ahead of print. Teach Learn Med. 2020. PMID: 32427496
Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents.JAMA Surg. 2018 May 1;153(5):409-416. doi: 10.1001/jamasurg.2017.5013. JAMA Surg. 2018. PMID: 29282462 Free PMC article.
The Learning Preferences of Applicants Who Interview for General Surgery Residency: A Multiinstitutional Study.J Surg Educ. 2016 Nov-Dec;73(6):e136-e141. doi: 10.1016/j.jsurg.2016.06.013. Epub 2016 Aug 8. J Surg Educ. 2016. PMID: 27424095 Review.
Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100.Med Teach. 2016;38(1):3-17. doi: 10.3109/0142159X.2015.1072619. Epub 2015 Aug 27. Med Teach. 2016. PMID: 26313700 Review.