The objective of this study was to examine whether lifestyle factors were associated with 5-year change in abdominal fat measured by computed tomography (CT) in the Insulin Resistance and Atherosclerosis (IRAS) Family Study. We obtained abdominal CT scans at baseline and at 5 years, from African Americans (AA) (N = 339) and Hispanic Americans (N = 775), aged 18-81 years. Visceral (VAT) and subcutaneous (SAT) adipose tissue was measured at the L4/L5 vertebral level. Physical activity was documented by self-report of vigorous activity and a 1-year recall instrument. Dietary intake was assessed at follow-up using a semi-quantitative food frequency questionnaire referencing the previous year. Generalized linear models, accounting for family structure, were used to assess the associations between percent change in fat accumulation and smoking, physical activity, total calories, polyunsaturated, monounsaturated, protein, and saturated fat intake, percent of calories from sweets, and soluble and insoluble fiber. Soluble fiber intake and participation in vigorous activity were inversely related to change in VAT, independent of change in BMI. For each 10 g increase in soluble fiber, rate of VAT accumulation decreased by 3.7% (P = 0.01). Soluble fiber was not associated with change in SAT (0.2%, P = 0.82). Moderately active participants had a 7.4% decrease in rate of VAT accumulation and a 3.6% decrease in rate of SAT accumulation versus less active participants (P = 0.003 and P = 0.01, respectively). Total energy expenditure was also inversely associated with accumulation of VAT. Soluble fiber intake and increased physical activity were related to decreased VAT accumulation over 5 years.