Purpose There is limited information on how the COVID-19 pandemic has changed health behaviors among cancer patients. We examined the impact of the pandemic on changes in exercise behaviors and identified characteristics associated with these changes among cancer patients. Methods Cancer patients (n = 1,361) completed a survey from August-September 2020 to assess COVID-19 pandemic-related changes in health behaviors and psychosocial factors. Patients were categorized into 3 groups: exercising less, exercising did not change, and exercising more. Patient characteristics were compared by exercise groups. Results One-third of the patients reported a decreased amount of regular exercise, while 11% reported exercising more during the pandemic. Patients who exercised less were more likely to be unemployed/retired, undergoing active treatment, and had increased pandemic-related alcohol consumption and psychosocial stressors such as loneliness and financial stress (all p < 0.05). In contrast, patients who exercised more were younger, female, full-time employed, did not consume alcohol, and had good health status and more social interactions (all p < 0.05). Patients who were living in rural areas and did not experience changes in daily life, were also more likely not to experience changes in exercise habits (all p < 0.05). Conclusion Our results indicate that a significant proportion of cancer patients experienced changes in exercise habits during the first 6 months of the COVID-19 pandemic. Age, sex, employment status, health status, alcohol consumption, and psychosocial factors were associated with changes in exercise behaviors. Providers should monitor for changes in health behaviors, such as exercise, because of their importance in improving cancer survivorship.