Cancer-associated fibroblasts (CAF) comprise the majority of stromal cells in breast cancers, yet their precise origins and relative functional contributions to malignant progression remain uncertain. Local invasion leads to the proximity of cancer cells and adipocytes, which respond by phenotypical changes to generate fibroblast-like cells termed as adipocyte-derived fibroblasts (ADF) here. These cells exhibit enhanced secretion of fibronectin and collagen I, increased migratory/invasive abilities, and increased expression of the CAF marker FSP-1 but not α-SMA. Generation of the ADF phenotype depends on reactivation of the Wnt/β-catenin pathway in response to Wnt3a secreted by tumor cells. Tumor cells cocultivated with ADFs in two-dimensional or spheroid culture display increased invasive capabilities. In clinical specimens of breast cancer, we confirmed the presence of this new stromal subpopulation. By defining a new stromal cell population, our results offer new opportunities for stroma-targeted therapies in breast cancer.