Setting Standards for Teaching Evaluation Data: An Application of the Contrasting Groups Method

Teach Learn Med. Apr-Jun 2009;21(2):82-6. doi: 10.1080/10401330902791222.

Abstract

Background: Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached.

Purpose: Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions.

Methods: Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels.

Results: For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges.

Conclusions: Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.

MeSH terms

  • Confidence Intervals
  • Education, Medical, Graduate / standards*
  • Education, Medical, Undergraduate / standards*
  • Educational Measurement
  • Faculty, Medical / standards*
  • Humans
  • Pennsylvania
  • Professional Competence / standards*
  • Schools, Medical / standards*
  • Statistics as Topic
  • Teaching / methods
  • Teaching / standards*
  • United States