In mammals, the male and female hormones androgen and estrogen act as endocrine regulators of energy metabolism. However, adipose tissue is also a site of androgen and estrogen synthesis; androgens convert to estrogens in these tissues, and adipose tissue is also a reservoir of steroids that act locally in a paracrine and intracrine manner. Thus, in adipose tissue, the local output of sex hormones is more complex than would be suggested by routine measurement of serum hormone concentrations. This review integrates studies on the effects of androgens and estrogens in the developmental programming of adipose tissue function in early life and addresses the contributions of local androgen and estrogen metabolism on adipose tissue function in adults.