The complex relationship between metabolic disease risk and body fat distribution in humans involves cellular characteristics which are specific to body fat compartments. Here we show depot-specific differences in the stromal vascual fraction of visceral and subcutaneous adipose tissue by performing single-cell RNA sequencing of tissue specimen from obese individuals. We characterize multiple immune cells, endothelial cells, fibroblasts, adipose and hematopoietic stem cell progenitors. Subpopulations of adipose-resident immune cells are metabolically active and associated with metabolic disease status and those include a population of potential dysfunctional CD8+ T cells expressing metallothioneins. We identify multiple types of adipocyte progenitors that are common across depots, including a subtype enriched in individuals with type 2 diabetes. Depot-specific analysis reveals a class of adipocyte progenitors unique to visceral adipose tissue, which shares common features with beige preadipocytes. Our human single-cell transcriptome atlas across fat depots provides a resource to dissect functional genomics of metabolic disease.