Psychometric methods for accurate and timely detection of item compromise have been a long-standing topic. While Bayesian methods can incorporate prior knowledge or expert inputs as additional information for item compromise detection, they have not been employed in item compromise detection itself. The current study proposes a two-phase Bayesian change-point framework for both stationary and real-time detection of changes in each item's compromise status. In Phase I, a stationary Bayesian change-point model for compromise detection is fitted to the observed responses over a specified time-frame. The model produces parameter estimates for the change-point of each item from uncompromised to compromised, as well as structural parameters accounting for the post-change response distribution. Using the post-change model identified in Phase I, the Shiryaev procedure for sequential testing is employed in Phase II for real-time monitoring of item compromise. The proposed methods are evaluated in terms of parameter recovery, detection accuracy, and detection efficiency under various simulation conditions and in a real data example. The proposed method also showed superior detection accuracy and efficiency compared to the cumulative sum procedure.
Keywords: Bayesian change-point detection; Shiryaev procedure; computerized tests; item compromise; test security.
© 2022 The Authors. British Journal of Mathematical and Statistical Psychology published by John Wiley & Sons Ltd on behalf of British Psychological Society.