By using blocked and rapid event-related functional MRI studies of memory, we explored the implications of using rest periods as a baseline condition in functional MRI studies. Activity in the medial temporal lobe (as well as in other brain regions) was substantially higher during rest than during several alternative baseline conditions. The effect of this elevated activity during rest was to reduce, eliminate, or even reverse the sign of the activity during task conditions relevant to memory functions. The results demonstrate that periods of rest are associated with significant cognitive activity and, therefore, provide a nonoptimal baseline for memory tasks. These results were observed not only when relatively long blocks of rest were used (experiment 1), but also when rest consisted of the short null trials typically used in rapid event-related designs (experiment 2). The findings have important implications for the design and interpretation of a wide range of fMRI studies of cognition.