Considerations in the choice of interobserver reliability estimates

J Appl Behav Anal. 1977 Spring;10(1):103-16. doi: 10.1901/jaba.1977.10-103.

Abstract

Two types of interobserver reliability values may be needed in treatment studies in which observers constitute the primary data-acquisition system: trial reliability and the reliability of the composite unit or score which is subsequently analyzed, e.g., daily or weekly session totals. Two approaches to determining interobserver reliability are described: percentage agreement and "correlational" measures of reliability. The interpretation of these estimates, factors affecting their magnitude, and the advantages and limitations of each approach are presented.