Introduction Physical activity is associated not only with a decreased risk of developing colorectal cancer but also with improved survival. One putative mechanism is the infiltration of immune cells in the tumor microenvironment. Experimental findings suggest that physical activity may mobilize immune cells to the tumor. We hypothesized that higher levels of physical activity prior to colorectal cancer diagnosis are associated with higher densities of tumor-infiltrating T-lymphocytes in colorectal cancer patients. Method The study setting was a northern Swedish population-based cohort, including 109792 participants with prospectively collected health- and lifestyle-related data. For 592 participants who later developed colorectal cancer, archival tumor tissue samples were used to assess the density of CD3+ and CD8+ cytotoxic T-cells by immunohistochemistry. Odds ratios for associations between self-reported, pre-diagnostic recreational physical activity and immune-cell infiltration were estimated by ordinal logistic regression. Results Recreational physical activity >3 times per week was associated with a higher density of CD8+ T-cells in the tumor front and center compared to participants reporting no recreational physical activity. Odds ratios were 2.77 (95% CI 1.21-6.35) and 2.85 (95% CI 1.28-6.33) for the tumor front and center, respectively, after adjustment for sex, age at diagnosis, and tumor stage. The risk estimates were consistent after additional adjustment for several potential confounders. For CD3 no clear associations were found. Conclusion Physical activity may promote the infiltration of CD8+ immune cells in the tumor microenvironment of colorectal cancer. Impact The study provides some evidence on how physical activity may alter the prognosis in colorectal cancer.