The allosteric effects and subtype selectivity of methoctramine on neuronal muscarinic receptors in N1E-115 cells and two different rat brain regions (cerebral cortex and striatum) were assessed. Saturation isotherms of [3H]N-methylscopolamine binding, performed in N1E-115 cells and dissociated cerebral cortex, showed that methoctramine reduced the Bmax in a concentration-dependent manner. Furthermore, this compound slowed the rate of dissociation of bound [3H]N-methylscopolamine in the same tissue preparations. Low concentrations of methoctramine (less than or equal to 1 microM) antagonized the M1-linked phosphoinositide response in N1E-115 cells and dissociated cerebral cortex in an apparent competitive mechanism. However, methoctramine exhibited noncompetitive effects at higher concentrations in N1E-115 cells. Observation of a similar effect in cerebrocortical cells was precluded since methoctramine by itself, at concentrations higher than 1 microM, stimulated inositol phosphate formation. The stimulatory effect of methoctramine on phosphoinositide hydrolysis was not blocked by atropine. A solely competitive mode of antagonism by methoctramine was observed for the inhibition of cAMP formation (a noncardiac-M2 coupled response) in N1E-115 cells and dissociated striatum. This antagonism was evident even at concentrations of methoctramine that noncompetitively antagonized the M1 response. Anomalously, methoctramine alone inhibited cAMP formation in dissociated striatum at concentrations of greater than or equal to 30 microM. Atropine was ineffective at blocking this effect. Methoctramine failed to demonstrate muscarinic receptor subtype selectively in blocking these two second messenger responses. This nonselectivity was supported by indirect binding experiments involving methoctramine and [3H]N-methylscopolamine. The data presented here demonstrate that methoctramine binds to a secondary site(s) associated with neuronal muscarinic receptors. Furthermore, methoctramine exhibits different mechanisms of antagonism and displays poor selectivity for the M1-linked phosphoinositide and noncardiac-M2 linked cAMP responses.