The color context effects referred to as color contrast, constancy, and assimilation underscore the fact that color percepts do not correspond to the spectral characteristics of the generative stimuli. Despite a variety of proposed theories, these phenomena have resisted explanation in a single principled framework. Using a hyperspectral image database of natural scenes, we here show that color contrast, constancy, and assimilation are all predicted by the statistical organization of spectral returns from natural visual environments.