We examined the limitations imposed by neural factors on spatial contrast sensitivity for both isochromatic and isoluminant gratings. We used two strategies to isolate these neural factors. First, we eliminated the effect of blurring by the dioptrics of the eye by using interference fringes. Second, we corrected our data for additional sensitivity losses up to and including the site of photon absorption by applying an ideal-observer analysis described by Geisler [J. Opt. Soc. Am. A 1, 775 (1984)]. Our measurements indicate that the neural visual system modifies the shape of the contrast-sensitivity functions for both isochromatic and isoluminant stimuli at high spatial frequencies. If we assume that the high-spatial-frequency performance of the neural visual system is determined by a low-pass spatial filter followed by additive noise, then the visual system has a spatial bandwidth 1.8 times lower for isoluminant red-green than for isochromatic stimuli. On the other hand, we find no difference in bandwidth or sensitivity of the neural visual system for isoluminant red-green and S-cone-isolated stimuli.