Self- or concentration quenching of octadecylrhodamine B (C18-Rh) fluorescence increases linearly in egg phosphatidylcholine (PC) vesicles but exponentially in vesicles composed of egg PC:cholesterol, 1:1, as the probe concentration is raised to 10 mol%. Cholesterol-dependent enhancement of self-quenching also occurs when N-(lissamine-rhodamine-B-sulfonyl)dioleoylphosphatidylethanolamine is substituted for C18-Rh and resembles that in dipalmitoylphosphatidylcholine vesicles below, as opposed to above, the phase transition. These effects are not due to changes in dimer:monomer absorbance. Stern-Volmer plots indicate a dependence of quenching on nonfluorescent dimers both in the presence and absence of cholesterol. Decreases in fluorescence lifetimes with increasing probe concentration parallel decreases in residual fluorescence of C18-Rh with increasing probe concentration in PC and PC + cholesterol membranes, respectively. Decreases in the steady-state polarization of C18-Rh fluorescence as its concentration is raised to 10 mol% indicate energy transfer with emission between probe molecules in PC and to a lesser extent in PC + cholesterol membranes. The calculated R0 for 50% efficiency of energy transfer from excited state probe to monomer was 55-58 A and to dimer was 27 A. Since lateral diffusion of C18-Rh is probably too slow to permit collisional quenching during the lifetime of the probe, even if C18-Rh were concentrated in a separate phase, C18-Rh self-quenching appears to be due mainly to energy transfer without emission to nonfluorescent dimers.