Experience-dependent changes in the response properties of ventral visual stream neurons are thought to underlie our ability to rapidly and efficiently recognize visual objects. How these neural changes are related to efficient visual processing during natural vision remains unclear. Here, we demonstrate a neurophysiological correlate of efficient visual search through highly familiar object arrays. Humans and monkeys are faster at locating the same target when it is surrounded by familiar compared with unfamiliar distractors. We show that this behavioral enhancement is driven by an increased sensitivity of target-selective neurons in inferior temporal cortex. This results from an increased "signal" for target representations and decreased "noise" from neighboring familiar distractors. These data highlight the dynamic properties of the inferior temporal cortex neurons and add to a growing body of evidence demonstrating how experience shapes neural processing in the ventral visual stream.