The development, content validity and inter-rater reliability of the SMART-Goal Evaluation Method: A standardised method for evaluating clinical goals

Aust Occup Ther J. 2015 Dec;62(6):420-7. doi: 10.1111/1440-1630.12218. Epub 2015 Aug 19.

Abstract

Introduction: Goal setting is a complex skill. The use of formal goal writing procedures (including the use of the SMART goal model) has been advocated. However, a standardised method of writing and evaluating SMART goals is currently lacking. This study comprised of two phases. The aims of phase one was to (i) develop the SMART Goal Evaluation Method (SMART-GEM) based on a SMART goal model; and (ii) investigate the content validity of the SMART-GEM. The aim of phase two of the study was to test the inter-rater reliability of the SMART-GEM.

Methods: Development of the SMART- GEM involved defining and constructing evaluation criteria suitable for auditing goal statements. A content validity assessment was conducted using an expert panel of Occupational Therapists (n = 10). Inter-rater reliability of the SMART-GEM was examined using a purposive sample of multiple raters (n = 24).

Results: The SMART- GEM was rated as having good content validity (individual items CVI ranged from 0.90 to 1.00; total SMART- GEM CVI = 0.99, ρ = 0.05). Agreement between raters on individual items ranged from poor (κ = 0.254) to excellent (κ = 0.965) and agreement of overall grades was fair to good (κ = 0.582). Inter-rater agreement on total scores was found to be very good (ICC = 0.895, 95% CI = 0.743 to 0.986, ρ = 0.001) with excellent internal consistency (α = 0.995).

Conclusion: The SMART-GEM demonstrated good construct validity and very good inter-rater reliability on total score and shows promise as a standardised method to writing and evaluating clinical goals.

Keywords: goal; occupational therapy; outcome assessment (health care); rehabilitation.

MeSH terms

  • Humans
  • Observer Variation
  • Occupational Therapy / methods*
  • Occupational Therapy / standards*
  • Outcome Assessment, Health Care / methods*
  • Outcome Assessment, Health Care / standards*
  • Patient Care Planning / standards*
  • Reproducibility of Results