In the visual cortex, inhibitory neurons alter the computations performed by target cells via combination of two fundamental operations, division and subtraction. The origins of these operations have been variously ascribed to differences in neuron classes, synapse location or receptor conductances. Here, by utilizing specific visual stimuli and single optogenetic probe pulses, we show that the function of parvalbumin-expressing and somatostatin-expressing neurons in mice in vivo is governed by the overlap of response timing between these neurons and their targets. In particular, somatostatin-expressing neurons respond at longer latencies to small visual stimuli compared with their target neurons and provide subtractive inhibition. With large visual stimuli, however, they respond at short latencies coincident with their target cells and switch to provide divisive inhibition. These results indicate that inhibition mediated by these neurons is a dynamic property of cortical circuits rather than an immutable property of neuronal classes.