Using direct cortical stimulation to map language function during awake craniotomy is a well-described and useful technique. However, the optimum neuropsychological tasks to use have not been detailed. We used both functional MRI (fMRI) and direct cortical stimulation to compare the sensitivity of two behavioral paradigms, number counting and object naming, in the demonstration of eloquent cortical language areas. Fifteen patients with left hemisphere lesions and seven healthy control subjects participated. Patients had both preoperative fMRI at 3 T and direct cortical stimulation. Patients and controls performed object naming and number counting during fMRI at 3 T. Laterality indices were calculated from the fMRI maps for the Number-counting>Object-naming and Object-naming>Number-counting contrasts. The same number-counting and object-naming paradigms were tested during awake craniotomy and assessed for sensitivity to speech disruption. In all patients during intraoperative cortical stimulation, speech disruption occurred at more sites during object naming than during number counting. Subtle speech errors were only elicited with the object-naming paradigm, whereas only speech arrest and/or hypophonia were measured using the number counting paradigm. In both patients and controls, fMRI activation maps demonstrated greater left lateralization for object naming as compared to number counting in both frontal and temporal language areas. Number counting resulted in a more bihemispheric distribution of activations than object naming. Both cortical stimulation testing and fMRI suggest that automated speech tasks such as number counting may not fully engage putative language networks and therefore are not optimal for language localization for surgical planning.