Melatonin is reported to reduce proliferation in many cell types, but the effect is small and the results are inconsistent. Information on the mechanism by which melatonin exerts its antiproliferative effects might provide insight into the variability of the response. In an ovarian adenocarcinoma cell line (BG-1), we find that melatonin at concentrations of 10(-9)-10(-7) M caused a 20-25% reduction in cell number. Melatonin also resulted in a similar reduction in [3H]-thymidine incorporation with no significant increase in cell death as measured by trypan blue incorporation. The Kd for melatonin reduction in cell number was approximately 5 x 10(-10) M. Melatonin ML2 receptors have a Kd for melatonin binding in the low nM range and are linked to the production of the calcium mobilizing agent inositol-1,4,5-trisphosphate (IP3). To investigate whether melatonin signaling involves an increase in cytosolic-free calcium. BG-1 cells were loaded with the calcium sensitive indicator, fura-2. Acute addition of melatonin (10(-5)-10(-9) M) did not alter cytosolic calcium. Addition of the putative nuclear receptor agonist CGP52608 caused a dose-dependent inhibition of cell number with a Kd of approximately 2 x 10(-9) M. Addition of CGP52608 caused a similar reduction in [3H]-thymidine incorporation. Neither melatonin (10(-8) M-10(-5) M) nor CGP52608 at concentrations below 10(-7) M induced cell death associated with the inhibition of cell proliferation; however, addition of CGP52608 at a high dose (10(-7) M) caused an increase in cell death, consistent with apoptosis. Growth inhibition by melatonin or CGP52608 did not alter the percentage of cells in G1 versus S/G2/M.