A cautionary note on the use of probability values to evaluate interobserver agreement

J Appl Behav Anal. 1982 Spring;15(1):189-90. doi: 10.1901/jaba.1982.15-189.

Abstract

Proposed methods of assessing the statistical significance of interobserver agreements provide erroneous probability values when conducted on serially correlated data. Investigators who wish to evaluate interobserver agreements by means of statistical significance can do so by limiting the analysis to every k(th) interval of data, or by using Markovian techniques which accommodate serial correlations.