Another person's caress is one of the most powerful of all emotional social signals. How much the primary somatosensory cortices (SIs) participate in processing the pleasantness of such social touch remains unclear. Although ample empirical evidence supports the role of the insula in affective processing of touch, here we argue that SI might be more involved in affective processing than previously thought by showing that the response in SI to a sensual caress is modified by the perceived sex of the caresser. In a functional MRI study, we manipulated the perceived affective quality of a caress independently of the sensory properties at the skin: heterosexual males believed they were sensually caressed by either a man or woman, although the caress was in fact invariantly delivered by a female blind to condition type. Independent analyses showed that SI encoded, and was modulated by, the visual sex of the caress, and that this effect is unlikely to originate from the insula. This suggests that current models may underestimate the role played by SI in the affective processing of social touch.