"Color constancy" refers to our ability to recognize the color of a surface despite changes in illumination. A range of cues and mechanisms, from receptoral adaptation to higher order cognitive cues, is thought to contribute to our color constancy ability. Here we used psychophysical adaptation to probe for an adaptable representation of surface color. We used stimuli that were matched for cone contrast when averaged over time but were consistent with either a constant scene under changing illumination or a changing scene. The color opponent aftereffect during adaptation to the constant scene was greater than that induced by the changing scene stimulus. Since the stimuli were matched for the responses they would elicit in receptoral mechanisms, the increased aftereffect in the constant scene condition cannot be wholly attributed to adaptation of receptors and neural mechanisms responsive to raw quantal catch. We interpret our result as most parsimoniously explained by the existence of adaptable mechanisms responsive to surface color, most likely located in early visual cortex.