Melatonin (5-methoxy-N-acetyltryptamine) at picomolar concentrations (IC50, 40 pM) inhibited the calcium-dependent release of [3H]dopamine elicited at 3 Hz (2 min, 20 mA, 2 msec) from rabbit retina through activation of a site possessing the pharmacological and functional characteristics of a receptor. The effect of melatonin shows biological specificity as this hormone does not modify [3H]dopamine release from striatum or olfactory tubercle. This paper describes the effects of small modifications of the melatonin structure on the inhibition of calcium-dependent release of [3H]dopamine from retina. The more active melatonin analogs were those possessing a 5-methoxy group on carbon 5 of the indole nucleus and an N-acetyl group on the same position as in melatonin. The potencies of 5-methoxy indoles compounds was as follows (IC50): melatonin (40 pM) = 6-chloromelatonin (40 pM) greater than 6-hydroxymelatonin (1.6 nM) greater than or equal to 6-methoxymelatonin (2 nM) greater than 5-methoxytryptamine (63 nM) greater than 5-methoxy-N,N-di-methyltryptamine (200 nM) much greater than 5-methoxytryptophol (4 microM). The structure activity relationships of melatonin and related indoles indicated that the efficacy of melatonin is determined by the moiety substituted on carbon 5 (i.e., 5-methoxy group), whereas the affinity for the receptor is determined primarily by the moiety substituted on carbon 3 (i.e., ethyl N-acetyl group) of the indole nucleus. N-acetyltryptamine competitively antagonized the inhibitory effect of melatonin in the chicken retina and appears to be a partial agonist in the rabbit retina.(ABSTRACT TRUNCATED AT 250 WORDS)