Reliability and usability of the RIGHT-COI&F reporting guideline: user survey and test of the checklist

Abstract

Objectives: Recently, we published a reporting checklist for conflicts of interest (COI) and funding in practice guidelines and guideline organizations' policy documents, RIGHT for Conflicts of Interest and Funding (RIGHT-COI&F). In this study, we examined the reliability and usability of this new tool in practice to assess reporting.

Study design and setting: We extracted a sample of guideline development organizations' COI and funding policies, and all guidelines developed by the World Health Organization (WHO) since 2019. We evaluated the adherence of these documents to the applicable items of RIGHT-COI&F. External reliability was assessed with the Kappa coefficient. We also surveyed key interest-holders on the ease of understanding of the tool who were asked to score each item on a scale from 1 (very difficult to understand) to 7 (very easy to understand).

Results: We evaluated the COI and funding policies of 28 guideline development organizations and 62 WHO guidelines. The mean Kappa value between the evaluators was 0.90 (full range 0.82-0.95), demonstrating good external reliability. Guideline policies adhered on average to half of the applicable items. The majority (77%) of WHO guidelines adhered to most applicable items. The mean times to evaluate policies and guidelines using the applicable items of the RIGHT-COI&F checklist were 28 and 30 minutes, respectively. The survey was completed by 157 individuals. The mean scores for ease of understanding were greater than five for all items.

Conclusion: RIGHT-COI&F checklist demonstrated good reliability and usability when used to assess reporting of COI and funding in practice guidelines and guideline organizations' policy documents.

Keywords: Conflicts of interest; Funding; Guidelines; Reliability; Reporting checklist; Usability.