Sensitivity to luminance difference, or contrast sensitivity, is critical for animals to survive in and interact with the external world. The contrast sensitivity function (CSF), which measures visual sensitivity to spatial patterns over a wide range of spatial frequencies, provides a comprehensive characterization of the visual system. Despite its popularity and significance in both basic research and clinical practice, it hasn't been clear what determines the CSF and how the factors underlying the CSF change in different conditions. In the current study, we applied the external noise method and perceptual template model to a wide range of external noise and spatial frequency (SF) conditions, and evaluated how the various sources of observer inefficiency changed with SF and determined the limiting factors underlying the CSF. We found that only internal additive noise and template gain changed significantly with SF, while the transducer non-linearity and coefficient for multiplicative noise were constant. The 12-parameter model provided a very good account of all the data in the 200 tested conditions (86.5%, 86.2%, 89.5%, and 96.4% for the four subjects, respectively). Our results suggest a re-consideration of the popular spatial vision model that employs the CSF as the front-end filter and constant internal additive noise across spatial frequencies. The study will also be of interest to scientists and clinicians engaged in characterizing spatial vision deficits and/or developing rehabilitation methods to restore spatial vision in clinical populations.