The undue influence of significant p-values on the perceived importance of study results

Acta Orthop. 2005 Jun;76(3):291-5.

Abstract

Background: Statistically significant differences between treatments (i.e., results typically associated with p < 0.05) may not always correspond to important differences upon which to base orthopedic practice. If the hypothesis that p < 0.05 unduly influences the perception of importance of study results were true, we would expect that presenting such a p-value would lead to 1) greater agreement among clinicians about the importance of a study result, and 2) greater perceived importance of a study result, when compared with presenting the same results omitting the p-value.

Methods: The participants were 3 orthopedics residents, 5 fellows, and 4 attending orthopedic surgeons at a university hospital. We constructed a 40-item questionnaire with the comparison groups, primary outcome of interest, and the results from each of 40 studies. These studies represent a variety of interventions across orthopedic surgery assessed in 2-group comparative intervention studies (randomized trials, observational studies) and were published between 2000 and 2002 in the Journal of Bone and Joint Surgery--American Volume. For each question, respondents were asked to rate the importance of the study results. Participants answered the questionnaire first without p-values and then, 8 weeks later, with p-values (and a random sample of items without p-values). An intra-class correlation quantified agreement between clinicians when answering items with and without p-values. The difference in mean importance scores between the two presentations was also estimated.

Results: Of 40 eligible clinical comparative studies, 30 reported p < 0.05 for their primary comparison. Without presenting p-values, overall agreement regarding clinical significance among reviewers was fair (ICC = 0.35, 95% CI: 0.25-0.49). In the 30 studies with p < 0.05, mean importance scores (1 = low; 3 = high) were greater when p-values were presented (difference 0.6, CI 0.1-1.1). 10 of 12 reviewers perceived results to be more important when presented with significant p-values.

Interpretation: When significant, p-values unduly influence the perception of clinicians regarding the importance of study results.

Publication types

  • Comparative Study
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Biomedical Research / standards
  • Biomedical Research / statistics & numerical data*
  • Clinical Trials as Topic / standards*
  • Data Interpretation, Statistical*
  • Evidence-Based Medicine
  • Humans
  • Orthopedics / standards*
  • Probability
  • Surveys and Questionnaires