Cancer-Associated Fibroblasts (CAFs) are the most prominent stromal cell type in breast tumors. CAFs promote tumor growth and metastasis by multiple mechanisms, including by mediating tumor-promoting inflammation. Immune modulation in the tumor microenvironment plays a central role in determining disease outcome. However, the functional interactions of CAFs with immune cells are largely unknown. Here we report a novel signaling axis between fibroblasts, cancer cells and immune cells in breast tumors that drives an immunosuppressive microenvironment, mediated by CAF-derived Chi3L1. We demonstrate that Chi3L1 is highly upregulated in CAFs isolated from mammary tumors and pulmonary metastases of transgenic mice, and in the stroma of human breast carcinomas. Genetic ablation of Chi3L1 in fibroblasts in vivo attenuated tumor growth, macrophage recruitment and reprogramming to an M2-like phenotype, enhanced tumor infiltration by CD8+ and CD4+ T cells and promoted a Th1 phenotype. These results indicate that CAF-derived Chi3L1 promotes tumor growth and shifts the balance of the immune milieu towards type 2 immunity. Taken together, our findings implicate fibroblast-derived Chi3L1 as a novel key player in the complex reciprocal interactions of stromal cells that facilitate tumor progression and metastasis, and suggest that targeting Chi3L1 may be clinically beneficial in breast cancer.