Type I error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective

J Exp Psychol Learn Mem Cogn. 2014 Sep;40(5):1287-306. doi: 10.1037/a0036914. Epub 2014 Jun 9.

Abstract

In order to examine metacognitive accuracy (i.e., the relationship between metacognitive judgment and memory performance), researchers often rely on by-participant analysis, where metacognitive accuracy (e.g., resolution, as measured by the gamma coefficient or signal detection measures) is computed for each participant and the computed values are entered into group-level statistical tests such as the t test. In the current work, we argue that the by-participant analysis, regardless of the accuracy measurements used, would produce a substantial inflation of Type I error rates when a random item effect is present. A mixed-effects model is proposed as a way to effectively address the issue, and our simulation studies examining Type I error rates indeed showed superior performance of mixed-effects model analysis as compared to the conventional by-participant analysis. We also present real data applications to illustrate further strengths of mixed-effects model analysis. Our findings imply that caution is needed when using the by-participant analysis, and recommend the mixed-effects model analysis.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Cognition / physiology*
  • Computer Simulation
  • Humans
  • Judgment / physiology*
  • Memory / physiology*
  • Models, Psychological*
  • Signal Detection, Psychological / physiology*