No short-cut in assessing trial quality: a case study

Trials. 2009 Jan 7;10:1. doi: 10.1186/1745-6215-10-1.


Background: Assessing the quality of included trials is a central part of a systematic review. Many check-list type of instruments for doing this exist. Using a trial of antibiotic treatment for acute otitis media, Burke et al., BMJ, 1991, as the case study, this paper illustrates some limitations of the check-list approach to trial quality assessment.

Results: The general verdict from the check list type evaluations in nine relevant systematic reviews was that Burke et al. (1991) is a good quality trial. All relevant meta-analyses extensively used its data to formulate therapeutic evidence. My comprehensive evaluation, on the other hand, brought to the surface a series of serious problems in the design, conduct, analysis and report of this trial that were missed by the earlier evaluations.

Conclusion: A check-list or instrument based approach, if used as a short-cut, may at times rate deeply flawed trials as good quality trials. Check lists are crucial but they need to be augmented with an in-depth review, and where possible, a scrutiny of the protocol, trial records, and original data. The extent and severity of the problems I uncovered for this particular trial warrant an independent audit before it is included in a systematic review.

MeSH terms

  • Access to Information
  • Acute Disease
  • Adolescent
  • Amoxicillin / therapeutic use*
  • Anti-Bacterial Agents / therapeutic use*
  • Biomedical Research / standards*
  • Child
  • Child, Preschool
  • Double-Blind Method
  • Evidence-Based Medicine*
  • Humans
  • Internet
  • Meta-Analysis as Topic*
  • Otitis Media / drug therapy*
  • Peer Review, Research
  • Periodicals as Topic
  • Publication Bias
  • Quality Control
  • Randomized Controlled Trials as Topic / standards*
  • Reproducibility of Results
  • Research Design / standards*
  • Scientific Misconduct
  • Treatment Outcome


  • Anti-Bacterial Agents
  • Amoxicillin