Inter-rater reliability of the neonatal adverse event severity scale using real-world Neonatal clinical trial data

Abstract

Objective: The Neonatal Adverse Event Severity Scale (NAESS) was developed to improve scoring of neonatal adverse events (AEs) and accelerate neonatal drug development. This is the first validation study of the novel tool.

Study design: Retrospective validation study assessing the inter-rater reliability (IRR) of the NAESS. Reviewers used real-world AE data from a neonatal trial. Intra-class correlation (ICC) statistical analysis was performed.

Result: Sixty AEs were randomly assigned to twelve reviewers for a total of 240 severity scores. Generic and AE-specific NAESS tables were assessed. The ICC was 0.63 (95% confidence interval 0.51 to 0.73). Percent variation due to reviewer and residual error was 0.03 and 0.34, respectively.

Conclusion: In this first study of the NAESS tool, an ICC of 0.63 indicates moderate reliability. Results highlight the need for improved data collection on neonatal AE forms, augmented training on the NAESS tool, and will inform the prospective validation studies.