The structure-activity relationships of a tamoxifen (TAM) (Z-1-(4- beta-dimethylaminoethoxyphenyl)1,2-diphenylbut-1-ene) series have been investigated. The tamoxifen derivatives were assayed in vitro by their modulation of estradiol (E2)-stimulated prolactin synthesis in primary cultures of dispersed rat pituitary gland cells. Monohydroxylation of TAM in position 4 of the stilbene ring system was found to be the optimal substitution for binding to the estrogen receptor [relative binding affinity (RBA) = 234] and to inhibit E2 (1 nM)-stimulated prolactin synthesis (IC50 7 nM) by pituitary cells in primary culture. Substitution in positions 3 and 4 to form a catechol did not decrease affinity for the estrogen receptor (RBA = 252), and potency as an antiestrogen was maintained in the prolactin assay (IC50 20 nM) as long as oxidation of the catechol was prevented. All of the hydroxylated derivatives of tamoxifen tested were estrogen antagonists; however, removal of the alkylaminoethoxy side chain from TAM produced a full estrogen agonist with low potency (20 nM). In contrast, removal of the side chain from 4-hydroxytamoxifen (4-OH TAM) produced a partial agonist. A structural analogue of 4-OH TAM, 3-[beta-dimethylaminoethoxy]-11-ethyl-12-(4-hydroxyphenyl)5,6- dihydrodibenzo[a,e]-cyclooctene (7c) had a decreased potency (IC50 16 nM) compared with 4-OH TAM (IC50 4 nM in the same experiment) as an estrogen antagonist. If the side chain was changed from a dimethylaminoethoxy to glyceryl, antagonist activity was reduced (IC50 0.8 microM). An allyl side chain produced a compound with no antiestrogenic activity at concentrations up to 1 microM. An adaptation of Belleau's macromolecular perturbation theory is suggested to explain the interaction of agonists, antagonists, and partial agonists at the ligand binding site of the estrogen receptor.