Recent research on eye movements during scene viewing has primarily focused on where the eyes fixate. But eye fixations also differ in their durations. Here we investigated whether fixation durations in scene viewing are under the direct and immediate control of the current visual input. Subjects freely viewed photographs of scenes in preparation for a later memory test while their eye movements were recorded. Using a novel scene degradation paradigm based on a saccade-contingent display change method, scenes were reduced in luminance during saccades ending in critical fixations. Results from two experiments showed that the durations of the critical fixations were immediately affected by scene luminance, with a monotonic relationship between luminance reduction and fixation duration. The results are the first to demonstrate that fixation durations in scene viewing are immediately influenced by the ease of processing of the image currently in view. These results are consistent with the CRISP (a timer-Controlled Random-Walk with Inhibition for Saccade Planning) computational model of saccade generation in scenes, proposing that difficulty in moment-by-moment visual and cognitive processing of the scene modulates fixation durations.