1. Radioligand binding with [125I]-cyanopindolol in the presence of isoproterenol was used to define the distribution of 5-HT1B receptors in the superior colliculus (SC) of adult hamsters. There was a high density of these receptors in the stratum griseum superficiale (SGS), and they were much less dense in other SC laminae. Enucleation of one eye produced a marked reduction in the density of these receptors in the contralateral SGS, suggesting that they are located primarily on retinotectal axon terminals. 2. Intracellular recording techniques were used to evaluate the effects of serotonin (5-HT) on the excitatory postsynaptic potentials (EPSPs) evoked in SC cells of adult hamsters by stimulation of the optic tract (OT) in vitro. Application of 5-HT produced a reduction of > or = 50% in OT-evoked EPSPs in 79% of the 67 cells tested. The average EPSP amplitude was 7.8 +/- 2.1 (SD) mV under control conditions and 2.7 +/- 1.9 mV in the presence of 5-HT (P < 0.01). For most of these neurons, application of 5-HT had little effect on their membrane potential or input resistance. The average percent change in membrane potential for cells tested with 5-HT was 0.5 +/- 6.0% and the average percent change in input resistance was 0.6 +/- 22.9%. 3. For four of six cells tested, application of 5-HT had no significant effects on the responses evoked by application of glutamate, either under normal bathing conditions or when the medium included low Ca2+ and high Mg2+. 4. Pharmacologic experiments indicated that the effects of 5-HT on retinotectal transmission were mimicked by the 5-HT1B agonists 1-[3-(trifluoromethyl)phenyl]-piperazine and 7-trifluoromethyl-4(4-methyl-1-piperazinyl) [1,2-a]-quinoxaline maleate and antagonized by the 5-HT1A/1B antagonists (-)-pindolol and methiothepin. The effects of 5-HT on the OT-evoked EPSP were not antagonized by either spiperone, ketanserin, 1-(2-methoxyphenyl)-4-[4-(2-phthalimido)butyl]-piperazine HBr, or [1-H-3 alpha-5 alpha-tropan-3-yl]-3,5-dichlorobenzoate. 5. Both the anatomic and physiological results are consistent with the conclusion that 5-HT presynaptically inhibits retinotectal transmission and that this effect is mediated by the 5-HT1B receptor.