The androgen receptor (AR) is required for male sex development and contributes to prostate cancer cell survival. In contrast to other nuclear receptors that bind the LXXLL motifs of coactivators, the AR ligand binding domain is preferentially engaged in an interdomain interaction with the AR FXXLF motif. Reported here are crystal structures of the ligand-activated AR ligand binding domain with and without bound FXXLF and LXXLL peptides. Key residues that establish motif binding specificity are identified through comparative structure-function and mutagenesis studies. A mechanism in prostate cancer is suggested by a functional AR mutation at a specificity-determining residue that recovers coactivator LXXLL motif binding. An activation function transition hypothesis is proposed in which an evolutionary decline in LXXLL motif binding parallels expansion and functional dominance of the NH(2)-terminal transactivation domain in the steroid receptor subfamily.