The inferior frontal cortex, particularly the ventrolateral prefrontal cortex (VLPFC) in the right hemisphere, has been implicated to serve as a general inhibitory mechanism in the cognitive control of behavior. Because this notion was primarily based on studies of response inhibition in manual tasks, it has yet to be validated in other response modalities. We conducted a functional magnetic resonance imaging study to examine whether the VLPFC is commonly activated during inhibition of responses by hand and by eye within the same subjects. We used the stop-signal task, a relatively pure measure of response inhibition, as the behavioral paradigm. Results from 12 subjects showed that both the right and the left caudal VLPFC and anterior insula, rostral to the premotor area, are activated during inhibition of both manual and saccadic responses. Within the posterior VLPFC, activations overlapped to a significant extent across the two response modalities, although a weaker functionally differentiation was also found along the dorsoventral axis. Other areas such as medial superior frontal gyrus (pre-supplementary motor area/supplementary eye field), dorsolateral prefrontal cortex, and inferior parietal cortex were also activated during canceling both hand and eye movements. Our findings suggest that a common VLPFC network is involved in response inhibition, although the specific control of the different response modalities may be partially segregated within the lateral prefrontal cortex.