Younger age and obesity increase the incidence and metastasis of triple-negative breast cancer (TNBC), an aggressive subtype of breast cancer. The extracellular matrix (ECM) promotes tumor invasion and metastasis. We characterized the effect of age and obesity on the ECM of mammary fat pads, lungs, and liver using a diet-induced obesity (DIO) model. At 4 week intervals, we either injected the mammary fat pads with allograft tumor cells to characterize tumor growth and metastasis or isolated the mammary fat pads and livers to characterize the ECM. Age had no effect on tumor growth but increased lung and liver metastasis after 16 weeks. Obesity increased tumor growth starting at 12 weeks, increased liver metastasis only at 4 weeks, and weight gain correlated to increased lung but not liver metastasis. Utilizing whole decellularized ECM coupled with proteomics, we found that early stages of obesity were sufficient to induce changes in the ECM composition and invasive potential of mammary fat pads with increased abundance of pro-invasive ECM proteins Collagen IV and Collagen VI. We identified cells of stromal vascular fraction and adipose stem and progenitor cells as primarily responsible for secreting Collagen IV and VI, not adipocytes. We characterized the changes in ECM in the lungs and liver, and determined that older age decreases the metastatic potential of lung and liver ECM while later-stage obesity increases the metastatic potential. These data implicate ECM changes in the primary tumor and metastatic microenvironment as mechanisms by which age and obesity contribute to breast cancer progression.