Compounds with similar target specificities and modes of inhibition cause similar cellular phenotypes. Based on this observation, we hypothesized that we could quantitatively classify compounds with diverse mechanisms of action using cellular phenotypes and identify compounds with unintended cellular activities within a chemical series. We have developed Cytometrix technologies, a highly automated image-based system capable of quantifying, clustering, and classifying changes in cellular phenotypes for this purpose. Using this system, 45 out of 51 known compounds were accurately classified into 12 distinct mechanisms of action. We also demonstrate microtubule-binding activity in one of seven related cytochalasin actin poisons. This technology can be used for a variety of drug discovery applications, including high-throughput primary screening of chemical and siRNA libraries and as a secondary assay to detect unintended activities and toxicities.