Recent studies have noted positive effects of red clothing on success in competitive sports, perhaps arising from an evolutionary predisposition to associate the color red with dominance status. Red may also enhance judgments of women's attractiveness by men, perhaps through a similar association with fertility. Here we extend these studies by investigating attractiveness judgments of both sexes and by contrasting attributions based on six different colors. Furthermore, by photographing targets repeatedly in different colors, we could investigate whether color effects are due to influences on raters or clothing wearers, by either withholding from raters information about clothing color or holding it constant via digital manipulation, while retaining color-associated variation in wearer's expression and posture. When color cues were available, we found color-attractiveness associations when males were judged by either sex, or when males judged females, but not when females judged female images. Both red and black were associated with higher attractiveness judgments and had approximately equivalent effects. Importantly, we also detected significant clothing color-attractiveness associations even when clothing color was obscured from raters and when color was held constant by digital manipulation. These results suggest that clothing color has a psychological influence on wearers at least as much as on raters, and that this ultimately influences attractiveness judgments by others. Our results lend support for the idea that evolutionarily-derived color associations can bias interpersonal judgments, although these are limited neither to effects on raters nor to the color red.