The sensitivity and rate of neural coding along the early visual pathways adapt to changes in contrast of the retinal image caused by external motion or self-generated eye movements. To identify the functional mechanisms of fast and slow contrast adaptation at the level of the visual cortex, we randomly varied, over both short and long timescales, the contrast of optimal sinusoidal gratings flashed in the receptive field of simple cells. We found that fast contrast-dependent suppression lagged excitation by ~11 ms and controlled the spike's temporal precision. During slow adaptation to low contrasts, the gain and latency of excitation increased whereas suppression became less visible, resulting in more sensitive but slower and more variable responses. We conclude that delayed suppression controls the response dynamics during both fast and slow contrast adaptation. More generally, we propose that sensory adaptation trades neuronal sensitivity for processing speed by changing the balance between excitation and delayed inhibition.