Development of a learning health system science competency assessment to guide training and proficiency assessment

Abstract

Introduction: Learning health systems (LHS) science is fundamentally a transdisciplinary field. To capture the breadth of the competencies of an LHS scientist, AHRQ and national experts defined a series of 42 competencies across seven domains that support success. Clinicians, researchers, and leaders who are new to the LHS field can identify and prioritize proficiency development among these domains. In addition, existing leaders and researchers will assemble teams of experts who together represent the LHS science domains. To serve LHS workforce development and proficiency assessment, the AHRQ-funded ACCELERAT K12 training program recruited domain experts and trainees to define and operationalize items to include in an LHS Competency Assessment to support emerging and existing LHS scientists in prioritizing and monitoring proficiency development.

Methods: Sequential interviews with 18 experts iteratively defined skills and tasks to illustrate stage in proficiency, and its progression, for each of 42 competencies in the seven LHS expertise domains: systems science; research questions and standards of scientific evidence; research methods; informatics; ethics of research and implementation in health systems; improvement and implementation science; and engagement, leadership, and research management. An educational assessment expert and LHS scientist refined the assessment criteria at each stage to use parallel language across domains. Last, current trainees reviewed and pilot tested the assessment and the LHS Competency Assessment was further refined using their feedback. The assessment framework was informed by Bloom's revised taxonomy of educational objectives (Anderson and Krathwohl, A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives, 2001) where learning progresses from recalling, defining, understanding, and awareness at the lower levels of the taxonomy, to applying and adopting and finally to creating, designing, and critiquing at the upper levels of the taxonomy. We also developed assessment criteria that could be used for longer term assessment of direct performance. Van der Vleuten et al. (Best Pract Res Clin Obstetr Gynaecol. 2010;24(6):703-719) define longer term direct assessment methods as assessment that occurs over a period ranging from weeks to even years and involves multiple assessment methods and exposure to the learner's work over an extended period.

Results: This experience report describes the content of the LHS Competency Assessment. For each domain and competency, the assessment lists examples of evidence to support expertise at each level of proficiency: no exposure; foundational (awareness/understanding); emerging (early application); and proficient (application with a high level of skill). Trainees begin with baseline standard assessment tables, where they can indicate no exposure or mark the foundational and emerging skills with which they have competence. For domains where foundational and emerging skills have been achieved, users can move on to assessment tables that list evidence of proficiency.

Conclusion: The LHS Competency Assessment offers consistent, graded criteria across the seven LHS domains to guide trainees and mentors to evaluate progress from no experience to foundational knowledge, emerging proficiency, and proficiency. The assessment can also be used to design training and mentoring for those newly exposed to LHS science and for those with key expertise who wish to expand LHS expertise.

Keywords: assessment; competency; curriculum; learning health systems.

FIGURE 1

Illustrative sources of evidence for trainee proficiency self, mentor, or program assessment

FIGURE 2

Using learning health systems competency assessment