To examine the neural mechanism for behavioral inhibition, we recorded single-cell activity in macaque ventrolateral prefrontal cortex, which is known to receive visual information directly from the inferotemporal cortex. In response to a moving random pattern of colored dots, monkeys had to make a go or no-go response. In the color condition, green indicated go, whereas red indicated no-go, regardless of the motion direction; in the motion condition, upward indicated go, whereas downward indicated no-go, regardless of the color. Approximately one-half of the visual cells were go/no-go differential. A majority of these cells (64/73) showed differential activity only in the color condition; they responded nondifferentially in the motion condition, although the same set of stimuli was used. We classified these cells as "go type" (n = 41) and "no-go type" (n = 23) depending on the color for which they showed a stronger response. Interestingly, in both types of cells, the differential effects were observed only for the no-go-indicating color. Compared with the nondifferential responses in the motion condition, go-type cells in the color condition showed weaker responses to the no-go-indicating color, whereas their responses to the go-indicating color were similar; in contrast, no-go type cells showed stronger responses to the no-go-indicating color, whereas their responses to the go-indicating color were similar. Both types of cells did not show any activity change during the actual execution of the go or no-go response. These results suggest that neurons in ventrolateral prefrontal cortex contribute to stimulus-response association in complex task situations by inhibiting behavioral responses on the basis of visual information from the ventral stream.