Functional magnetic resonance imaging (fMRI) studies of early sensory cortex often measure stimulus-driven increases in the blood oxygenation level-dependent (BOLD) signal. However, these positive responses are frequently accompanied by reductions in the BOLD signal in adjacent regions of cortex. Although this negative BOLD response (NBR) is thought to result from neuronal suppression, the precise relationship between local activity, suppression, and perception remains unknown. By measuring BOLD signals in human primary visual cortex while varying the baseline contrast levels in the region affected by the NBR, we tested three physiologically plausible computational models of neuronal modulation that could explain this phenomenon: a subtractive model, a response gain model, and a contrast gain model. We also measured the ability of isoluminant contrast to generate an NBR. We show that the NBR can be modeled as a pathway-specific contrast gain modulation that is strongest outside the fovea. We found a similar spatial bias in a psychophysical study using identical stimuli, although these data indicated a response gain rather than a contrast gain mechanism. We reconcile these findings by proposing (1) that the NBR is associated with a long-range suppressive mechanism that hyperpolarizes a subset of magnocellularly driven neurons at the input to V1, (2) that this suppression is broadly tuned to match the spatial features of the mask region, and (3) that increasing the baseline contrast in the suppressed region drives all neurons in the input layer, reducing the relative contribution of the suppressing subpopulation in the fMRI signal.