Intratumoral heterogeneity greatly complicates the study of molecular mechanisms driving cancer progression and our ability to predict patient outcomes. Here we have developed an automated high-throughput cell-imaging platform (htCIP) that allows us to extract high-content information about individual cells, including cell morphology, molecular content and local cell density at single-cell resolution. We further develop a comprehensive visually-aided morpho-phenotyping recognition (VAMPIRE) tool to analyze irregular cellular and nuclear shapes in both 2D and 3D microenvironments. VAMPIRE analysis of ~39,000 cells from 13 previously sequenced patient-derived pancreatic cancer samples indicate that metastasized cells present significantly lower heterogeneity than primary tumor cells. We found the same morphological signature for metastasis for a cohort of 10 breast cancer cell lines. We further decipher the relative contributions to heterogeneity from cell cycle, cell-cell contact, cell stochasticity and heritable morphological variations.