Obesity elicits an immune response characterized by myeloid cell recruitment to key metabolic organs, including adipose tissue. However, the response of immune cells to nonpathologic metabolic stimuli has been less well studied, and the factors that regulate the metabolic-dependent accumulation of immune cells are incompletely understood. Here we characterized the response of adipose tissue macrophages (ATMs) to weight loss and fasting in mice and identified a role for lipolysis in ATM recruitment and accumulation. We found that the immune response to weight loss was dynamic; caloric restriction of high-fat diet-fed mice led to an initial increase in ATM recruitment, whereas ATM content decreased following an extended period of weight loss. The peak in ATM number coincided with the peak in the circulating concentrations of FFA and adipose tissue lipolysis, suggesting that lipolysis drives ATM accumulation. Indeed, fasting or pharmacologically induced lipolysis rapidly increased ATM accumulation, adipose tissue chemoattractant activity, and lipid uptake by ATMs. Conversely, dietary and genetic manipulations that reduced lipolysis decreased ATM accumulation. Depletion of macrophages in adipose tissue cultures increased expression of adipose triglyceride lipase and genes regulated by FFA, and increased lipolysis. These data suggest that local lipid fluxes are central regulators of ATM recruitment and that once recruited, ATMs form lipid-laden macrophages that can buffer local increases in lipid concentration.