Bile salts enhance secretion of cholesterol into bile and its subsequent solubilization with phosphatidylcholine in mixed micelles. Sphingomyelin, a major structural lipid of the hepatocyte canalicular membrane, and disaturated phosphatidylcholines are known to impede nucleation of solid cholesterol crystals in supersaturated model systems. To understand these effects physico-chemically, we compared the influence of bile salts on interactions of cholesterol with natural sphingomyelins, as well as with dipalmitoyl and egg yolk phosphatidylcholines using various in vitro systems. Submicellar bile salts enhanced significantly bidirectional transfer of dehydroergosterol (a fluorescent cholesterol analog) between sphingomyelin and egg yolk phosphatidylcholine vesicles in the rank order taurocholate < tauroursodeoxycholate < taurodeoxycholate. Quasielastic light scattering of serially diluted sphingomyelin-taurocholate mixtures (1:1 molar ratio, 3 g/dl) revealed metastable temperature-dependent transitions between globular micelles, rod-shaped micelles and vesicles, suggesting that phase transitions under these experimental conditions were metastable only at temperatures below 37 degrees C. Ternary phase diagrams of all sphingomyelins and dipalmitoyl phosphatidylcholine with cholesterol and taurocholate (37 degrees C, 3 g/dl, 0.15 M NaCl) were identical. Compared to systems containing egg yolk phosphatidylcholine, the 1-phase micellar zone and 2- and 3-phase solid cholesterol crystal-containing zones were reduced markedly while the 2-phase zone with stable cholesterol-sphingomyelin liquid crystals was greatly expanded. Our results suggest that the high affinity of cholesterol for sphingomyelin is lost in the presence of bile salts. Our findings may be relevant to secretion of cholesterol into bile and to its inability to crystallize in the hepatocyte canalicular lumen or its surrounding membranes.