Validity and reliability of the novel three-item occupational violence patient risk assessment tool

Abstract

Aim: To develop and psychometrically test an occupational violence (OV) risk assessment tool in the emergency department (ED).

Design: Three studies were conducted in phases: content validity, predictive validity and inter-rater reliability from June 2019 to March 2021.

Methods: For content validity, ED end users (mainly nurses) were recruited to rate items that would appropriately assess for OV risk. Subsequently, a risk assessment tool was developed and tested for its predictive validity and inter-rater reliability. For predictive validity, triage notes of ED presentations in a month with the highest OV were assessed for presence of OV risk. Each presentation was then matched with events recorded in the OV incident register. Sensitivity and specificity values were calculated. For inter-rater reliability, two assessors-trained and untrained-independently assessed the triage notes for presence of OV risk. Cohen's kappa was calculated.

Results: Two rounds of content validity with a total of N = 81 end users led to the development of a three-domain tool that assesses for OV risk using aggression history, behavioural concerns (i.e., angry, clenched fist, demanding, threatening language or resisting care) and clinical presentation concerns (i.e., alcohol/drug intoxication and erratic cognition). Recommended risk ratings are low (score = 0 risk domain present), moderate (score = 1 risk domain present) and high (score = 2-3 risk domains present), with an area under the curve of 0.77 (95% confidence interval 0.7-0.81, p < .01). Moderate risk rating had a 61% sensitivity and 91% specificity, whereas high risk rating had 37% sensitivity and 97% specificity. Inter-rater reliability ranged from 0.67 to 0.75 (p < .01), suggesting moderate agreement.

Conclusions: The novel three-domain OV risk assessment tool was shown to be appropriate and relevant for application in EDs. The tool, developed through a rigorous content validity process, demonstrates acceptable predictive validity and inter-rater reliability.

Impact: The developed tool is currently piloted in a single hospital ED, with a view to extend to inpatient settings and other hospitals.

Keywords: emergency nursing; instrument development; psychometrics; risk assessment; risk factors; workplace violence.