The 2003-2004 and 2005-2006 cycles of the National Health and Nutrition Examination Survey (NHANES) were among the first population-level studies to incorporate objectively measured physical activity and sedentary behavior, allowing for greater understanding of these behaviors. However, there has yet to be a comprehensive examination of these data in cancer survivors, including short- and long-term survivors of all cancer types. Therefore, the purpose of this analysis was to use these data to describe activity behaviors in short- and long-term cancer survivors of various types. A secondary aim was to compare activity patterns of cancer survivors to that of the general population. Cancer survivors (n = 508) and age-matched individuals not diagnosed with cancer (n = 1,016) were identified from a subsample of adults with activity measured by accelerometer. Physical activity and sedentary behavior were summarized across cancer type and demographics; multivariate regression was used to evaluate differences between survivors and those not diagnosed with cancer. On average, cancer survivors were 61.4 (95% CI: 59.6, 63.2) years of age; 57% were female. Physical activity and sedentary behavior patterns varied by cancer diagnosis, demographic variables, and time since diagnosis. Survivors performed 307 min/day of light-intensity physical activity (95% CI: 295, 319), 16 min/day of moderate-vigorous intensity activity (95% CI: 14, 17); only 8% met physical activity recommendations. These individuals also reported 519 (CI: 506, 532) minutes of sedentary time, with 86 (CI: 84, 88) breaks in sedentary behavior per day. Compared to non-cancer survivors, after adjustment for potential confounders, survivors performed less light-intensity activity (P = 0.01), were more sedentary (P = 0.01), and took fewer breaks in sedentary time (P = 0.04), though there were no differences in any other activity variables. These results suggest that cancer survivors are insufficiently active. Relative to adults of similar age not diagnosed with cancer, they engage in more sedentary time with fewer breaks. As such, sedentary behavior and light-intensity activity may be important intervention targets, particularly for those for whom moderate-to-vigorous activity is not well accepted.