Calibration of multisite raters for prospective visual reads of amyloid PET scans

Abstract

Introduction: In multicenter Alzheimer's disease studies, amyloid positron emission tomography (PET) visual reads are typically performed centrally by a few experts. Incorporating a broader reader network enhances scalability and generalizability.

Methods: Ten neuroimaging experts from eight Alzheimer's Disease Research Centers (ADRCs) visually read 180 amyloid PET scans (30 scans and 15 duplicate scans for each of four tracers, imaged across a wide variety of scanners), using preferred reading software without anatomical imaging or quantitation. Scans were classified as elevated or non-elevated per tracer-specific criteria. Inter- and intra-rater agreement was assessed.

Results: Inter-rater agreement was substantial (Fleiss' κ = 0.78), with full consensus on 69% of scans. Inter-rater reliability was substantial to perfect across tracers (Fleiss' κ = 0.70-0.87). Intra-rater agreement was substantial to perfect (Cohen's κ = 0.79-1). Scans with intermediate (10-40 Centiloid) quantitation had lower reader agreement.

Discussion: A multicenter expert network achieved substantial agreement classifying amyloid PET scans. These scans provide a standard for reader training and reliability assurance in future studies.

Highlights: Calibration methods ensure reliable amyloid positron emission tomography (PET) visual reads across multiple raters. Substantial agreement is possible across readers using their preferred tools. Agreement is also substantial regardless of the amyloid PET tracer used. Scans with intermediate (10-40 Centiloid) quantitation have lower reader agreement. The calibration set will become a training tool for amyloid PET visual read studies.

Keywords: Amyloid positron emission tomography; inter‐rater reliability; intra‐rater reliability; visual reader calibration; visual reads.