In situ hybridization (ISH) techniques on interphase cells, or interphase cytogenetics, have powerful potential clinical and biological applications, such as detection of minimal residual disease, early relapse, and the study of clonal evolution and expansion in neoplasia. Much attention has been paid to issues related to ISH data acquisition, i.e., the numbers, colors, intensities, and spatial relationships of hybridization signals. The methodology concerning data analysis, which is of prime importance for clinical applications, however, is less well investigated. We have studied the latter for the detection of small monosomic and trisomic cell populations using various mixtures of human female and male cells. With a chromosome X specific probe, the male cells stimulated monosomic subpopulations of 0, 1, 5, 10, 50, 90, 95, 99, and 100%. Analogously, when a (7 + Y) specific probe combination was used, containing a mixture of chromosome No. 7 and Y-specific DNA, the male cells simulated trisomic cell populations. Probes specific for chromosomes Nos. 1, 7, 8, and 9 were used for estimation of ISH artifacts. Three statistical tests, the Kolmogorov-Smirnov test, the multiple-proportion test, and the z'-max test, were applied to the empirical data using the control data as a reference for ISH artifacts. The Kolmogorov-Smirnov test was found to be inferior for discrimination of small monosomic or trisomic cell populations. The other two tests showed that when 400 cells were evaluated, and using selected control probes, monosomy X could be detected at a frequency of 5% aberrant cells, and trisomy 7 + Y at a frequency of 1%.(ABSTRACT TRUNCATED AT 250 WORDS)