Objective: To develop an explicit scheme for calculating a 5-star quality rating for consumer health information, and to test if there is good agreement be tween this and the final DISCERN quality question.
Design: A sample of 15 consumers and health professionals rated 26 health information leaflets covering a broad range of conditions and treatments using two new 5-star-rating schemes and the existing DISCERN final quality rating. Each scheme is based on the 15 DISCERN criteria, but the 5-star schemes provide more explicit methods for summarizing overall quality. The level of agreement between the three different rating systems was compared using Kappa scores with quadratic weights. Participants were also asked to complete a brief questionnaire that was designed to elicit their views on using a visual summary of the quality of health information.
Results: The level of agreement between each 5-star-rating system and the existing DISCERN quality rating question was high (kappa = 0.86, 95% CI 0.83-0.89 in both instances). Seventy-seven per cent of the sample preferred the second star-rating scheme, and 80% reported they would use such a scheme.
Conclusion: Assigning a single quality score using an explicit scoring scheme (represented by stars) based on answers to the DISCERN questionnaire is a reliable and valid way of rating consumer health information.