The efficiency of simulation-based multiple comparisons

Biometrics. 1987 Dec;43(4):913-28.


A frequently encountered problem in practice is that of simultaneous interval estimation of p linear combinations of a parameter beta in the setting of (or equivalent to) a univariate linear model. This problem has been solved adequately only in a few settings when the covariance matrix of the estimator is diagonal; in other cases, conservative solutions can be obtained by the methods of Scheffé, Bonferroni, or Sidák (1967, Journal of the American Statistical Association 62, 626-633). Here we investigate the efficiency of using a simulated critical point for exact intervals, which has been suggested before but never put to serious test. We find the simulation-based method to be completely reliable and essentially exact. Sample size savings are substantial (in our settings): 3-19% over the Sidák method, 4-37% over the Bonferroni method, and 27-33% over the Scheffé method. We illustrate the efficiency and flexibility of the simulation-based method with case studies in physiology and marine ecology.

Publication types

  • Comparative Study
  • Research Support, U.S. Gov't, Non-P.H.S.
  • Research Support, U.S. Gov't, P.H.S.

MeSH terms

  • Analysis of Variance
  • Animals
  • Capillary Permeability
  • Cats
  • Computer Simulation
  • Decapoda
  • Fishes
  • Mathematics
  • Models, Biological*
  • Predatory Behavior
  • Probability
  • Research Design*