In chromosome breakage assays, validated, universal criteria for selection of cells and classification of chromosome aberrations may enhance their utility for cancer susceptibility screening. To standardize a fluorescence in situ hybridization (FISH) modification of the mutagen sensitivity assay (MSA), scoring criteria were evaluated by web-based validation. Two hundred digital FISH images were assigned random identification numbers. With this set of images, criteria for inclusion of cells and measurement of the frequency of abnormal cells were evaluated by eight observers, all of whom had five or more years of experience. Observers included doctoral and MS/BS level cytogeneticists, and were drawn from a randomized pool of 54 volunteers. Questions addressed were: (1) how uniformly were criteria applied to analysis of a standard digital FISH image set and (2) did concordance vary with educational level? These data suggest inter-rater agreement within a factor of 2 for average breakage frequency, but revealed greater variability in cell selection. These results aid in estimating the components of assay variance due to definitions, technical parameters and biological variables.