Very little is known about the mechanisms that drive the alternation between the two views during binocular rivalry. A key property of the rivalry process is the rate at which the two views alternate. Understanding the factors that affect the rate of the alternation is critical to the final understanding of the underlying process. Using a circular and a radial grating as the rivalry stimuli, we observed a significantly faster binocular rivalry when stimuli were presented in the right visual field than that in the left visual field for the right-handed observers, and a reversed asymmetry for the left-handed observers. In both groups, rivalry was faster for stimuli presented in the lower visual field than that in the upper visual field. This pattern of results suggests that (1) rivalry is likely a locally driven process and (2) the visual brain in the left hemisphere may be the faster one of the two hemispheres in right-handed people.