The unique anatomical and electrophysiological features of the inferior olive and its importance to cerebellar function have been recognized for decades. However, understanding the exact function of the inferior olive has been limited by the general lack of correlation between its neural activity and specific behavioral states. Electrophysiological studies in animals showed that the inferior olive response to sensory stimuli is generally invariant to stimulus properties but is enhanced by unexpected stimuli. Using functional magnetic resonance imaging in humans, we have shown that the inferior olive is activated when subjects performed a task requiring perception of visual stimuli with unpredictable timing (Xu et al. J Neurosci 26(22):5990-5995, 2006, Liu et al. J Neurophysiol 100(3):1557-1561, 2008). In the current study, subjects were scanned while passively perceiving visual and tactile stimuli that were rendered unpredictable by continuously varying interstimulus intervals (ISIs). Sequences of visual stimuli and tactile stimuli to the right hand were presented separately within the same scanning session. In addition to the activation of multiple areas in the cerebellar cortex consistent with previous imaging studies, the results show that both tactile and visual stimulation with variable ISIs were effective in activating the inferior olive. Together with our previous findings, the current results are consistent with the electrophysiological studies in animals and further support the view that the inferior olive and the climbing fiber system primarily convey the temporal information of sensory input regardless of the modality.