Objectives: Direct observation of procedural skills (DOPS) are competence-assessment tools in endoscopy. Formative paediatric gastroscopy DOPS were implemented into the UK curriculum in 2016 but lack validity evidence; we aimed to assess validity evidence using a recognised contemporary validity framework.
Methods: We performed a prospective UK-wide analysis of formative paediatric gastroscopy DOPS submitted to the e-Portfolio over 1 year. Internal structure validity was assessed using interitem correlations between DOPS items, average domain, and skillset scores and with the overall competency rating. Overall competence scores and mean DOPS scores were compared by trainee seniority and procedure count (discriminative validity). Receiver operating characteristic curve analysis was performed to explore if DOPS scores could be used to delineate procedural competency (consequential validity).
Results: A total of 157 DOPS assessments were completed by 20 trainers for 17 trainees. Strengths of correlations varied between DOPS components, with overall competency correlating most with technical-predominant items, domains and skillsets. Both the overall assessor's rating and mean DOPS scores increased with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Overall competency could be delineated using mean DOPS scores (area under receiver operating characteristic curve 0.95, P < 0.001), with a threshold of 3.9 providing optimal sensitivity (94.4%) and specificity (89.7%).
Conclusions: Competencies in paediatric gastroscopy, as assessed using DOPS, vary in their correlation with overall competence and increase with trainee experience. Formative DOPS thresholds could be used to indicate readiness for summative assessment. Our study therefore provides evidence of internal structure, discriminative, and consequential validity in support of formative paediatric gastroscopy DOPS.