Synaptic interactions to extract information about wavelength, and thus color, begin in the vertebrate retina with three classes of light-sensitive cells: rod photoreceptors at low light levels, multiple types of cone photoreceptors that vary in spectral sensitivity, and intrinsically photosensitive ganglion cells that contain the photopigment melanopsin. When isolated from its neighbors, a photoreceptor confounds photon flux with wavelength and so by itself provides no information about color. The retina has evolved elaborate color opponent circuitry for extracting wavelength information by comparing the activities of different photoreceptor types broadly tuned to different parts of the visible spectrum. We review studies concerning the circuit mechanisms mediating opponent interactions in a range of species, from tetrachromatic fish with diverse color opponent cell types to common dichromatic mammals where cone opponency is restricted to a subset of specialized circuits. Distinct among mammals, primates have reinvented trichromatic color vision using novel strategies to incorporate evolution of an additional photopigment gene into the foveal structure and circuitry that supports high-resolution vision. Color vision is absent at scotopic light levels when only rods are active, but rods interact with cone signals to influence color perception at mesopic light levels. Recent evidence suggests melanopsin-mediated signals, which have been identified as a substrate for setting circadian rhythms, may also influence color perception. We consider circuits that may mediate these interactions. While cone opponency is a relatively simple neural computation, it has been implemented in vertebrates by diverse neural mechanisms that are not yet fully understood.