In everyday experience, perceived colors of objects remain approximately constant under changes in illumination. This constancy is helpful for identifying objects across viewing conditions. Studies on color constancy often employ monitor simulations of illumination and reflectance changes. Real scenes, however, have features that might be important for color constancy but that are in general not captured by monitor displays. Here, we investigate categorical color constancy employing real surfaces and real illuminants in a rich viewing context. Observers sorted 450 Munsell samples into the 11 basic color categories under a daylight and four filtered daylight illuminants. We additionally manipulated illuminant cues from the local surround. Color constancy as quantified both with a classification consistency index and a standard color constancy index was high in both cue conditions. Observers generally classified colors with the same precision across different illuminants as across repetitions for the daylight illuminant. Moreover, the pattern of classification consistency in terms of stimulus hue, value, and chroma was similar when comparing different observers for the daylight illuminant and when comparing individual observers across different illuminants. We conclude that color categorization is robust under illuminant changes as well as across observers, thus potentially serving both object identification and communication.