Expected p-values in light of an ROC curve analysis applied to optimal multiple testing procedures

Abstract

Many statistical studies report p-values for inferential purposes. In several scenarios, the stochastic aspect of p-values is neglected, which may contribute to drawing wrong conclusions in real data experiments. The stochastic nature of p-values makes their use to examine the performance of given testing procedures or associations between investigated factors to be difficult. We turn our focus on the modern statistical literature to address the expected p-value (EPV) as a measure of the performance of decision-making rules. During the course of our study, we prove that the EPV can be considered in the context of receiver operating characteristic (ROC) curve analysis, a well-established biostatistical methodology. The ROC-based framework provides a new and efficient methodology for investigating and constructing statistical decision-making procedures, including: (1) evaluation and visualization of properties of the testing mechanisms, considering, e.g. partial EPVs; (2) developing optimal tests via the minimization of EPVs; (3) creation of novel methods for optimally combining multiple test statistics. We demonstrate that the proposed EPV-based approach allows us to maximize the integrated power of testing algorithms with respect to various significance levels. In an application, we use the proposed method to construct the optimal test and analyze a myocardial infarction disease dataset. We outline the usefulness of the "EPV/ROC" technique for evaluating different decision-making procedures, their constructions and properties with an eye towards practical applications.

Keywords: AUC; Benjamini–Hochberg procedure; Bonferroni procedure; Bootstrap tilting method; P-value; ROC curve; best combination; confidence region; expected p-value; multiple testing; partial AUC.

Figure 1.

The ROC curves related to the Student’s t-test (“______”) and the Welch’s t-test (“----”), where panel (a) represents the case of n = 10, m = 50, ${\sigma }_1^2=1$, ${\sigma }_2^2=1$, δ = 0.7; graph (b) represents the case of n = 20, m = 40, ${\sigma }_1^2=1$, ${\sigma }_2^2=1$, δ = 0.7; graph (c) represents the case of n = 40, m = 20, ${\sigma }_1^2=4$, ${\sigma }_2^2=1$, δ = 0.7; graph (d) represents the case of n = 40, m = 20, ${\sigma }_1^2=9$, ${\sigma }_2^2=1$, δ = 0.7 ; graph (e) represents the case of n = 10, m = 50, ${\sigma }_1^2=4$, ${\sigma }_2^2=1$, δ = 1 and graph (f) represents the case of n = 20, m = 40, ${\sigma }_1^2=9$, ${\sigma }_2^2=1$, δ = 0.1.

Figure 2.

The average power of the approximated BLC of the test statistics, the Bonferroni and BH procedures plotted against the sample sizes.

Figure 3.

The EPVs of the approximated BLC the test statistics, Bonferroni and BH procedures plotted against the sample sizes.