While it has been established that a number of microenvironment components can affect the likelihood of metastasis, the link between microenvironment and tumor cell phenotypes is poorly understood. Here we have examined microenvironment control over two different tumor cell motility phenotypes required for metastasis. By high-resolution multiphoton microscopy of mammary carcinoma in mice, we detected two phenotypes of motile tumor cells, different in locomotion speed. Only slower tumor cells exhibited protrusions with molecular, morphological, and functional characteristics associated with invadopodia. Each region in the primary tumor exhibited either fast- or slow-locomotion. To understand how the tumor microenvironment controls invadopodium formation and tumor cell locomotion, we systematically analyzed components of the microenvironment previously associated with cell invasion and migration. No single microenvironmental property was able to predict the locations of tumor cell phenotypes in the tumor if used in isolation or combined linearly. To solve this, we utilized the support vector machine (SVM) algorithm to classify phenotypes in a nonlinear fashion. This approach identified conditions that promoted either motility phenotype. We then demonstrated that varying one of the conditions may change tumor cell behavior only in a context-dependent manner. In addition, to establish the link between phenotypes and cell fates, we photoconverted and monitored the fate of tumor cells in different microenvironments, finding that only tumor cells in the invadopodium-rich microenvironments degraded extracellular matrix (ECM) and disseminated. The number of invadopodia positively correlated with degradation, while the inhibiting metalloproteases eliminated degradation and lung metastasis, consistent with a direct link among invadopodia, ECM degradation, and metastasis. We have detected and characterized two phenotypes of motile tumor cells in vivo, which occurred in spatially distinct microenvironments of primary tumors. We show how machine-learning analysis can classify heterogeneous microenvironments in vivo to enable prediction of motility phenotypes and tumor cell fate. The ability to predict the locations of tumor cell behavior leading to metastasis in breast cancer models may lead towards understanding the heterogeneity of response to treatment.