Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Aug;18(3):375-96.
doi: 10.1007/s10459-012-9376-x. Epub 2012 May 17.

Workplace-based Assessment: Raters' Performance Theories and Constructs

Affiliations
Free PMC article

Workplace-based Assessment: Raters' Performance Theories and Constructs

M J B Govaerts et al. Adv Health Sci Educ Theory Pract. .
Free PMC article

Abstract

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Figures

Fig. 1
Fig. 1
Rating form; 1-dimensional overall performance rating (VC procedure step 2)
Fig. 2
Fig. 2
Aggregate performance theory, presenting the performance dimensions used by supervisor-raters in general practice when assessing trainee performance in GP patient encounters

Similar articles

See all similar articles

Cited by 25 articles

See all "Cited by" articles

References

    1. Albanese MA. Challenges in using rater judgments in medical education. Journal of Evaluation in Medical Practice. 2000;6(3):305–319. doi: 10.1046/j.1365-2753.2000.00253.x. - DOI - PubMed
    1. Archer J, McGraw M, Davies H. Assuring validity of multisource feedback in a national programme. Archives of Disease in Childhood. 2010;95:330–335. doi: 10.1136/adc.2008.146209. - DOI - PubMed
    1. Arts JARM, Gijselaers WH, Boshuizen HPA. Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology. 2006;31(4):387–410. doi: 10.1016/j.cedpsych.2006.05.005. - DOI
    1. Barbour RS. Checklists for improving rigour in qualitative research. A case of the tail wagging the dog? British Medical Journal. 2001;322:1115–1117. doi: 10.1136/bmj.322.7294.1115. - DOI - PMC - PubMed
    1. Barnes-Farrell JL. Performance appraisal: Person perception processes and challenges. In: London M, editor. How people evaluate others in organizations. Mahwah, NJ: Lawrence Erlbaum Associates, Inc; 2001. pp. 135–153.

MeSH terms

LinkOut - more resources

Feedback