Jealousy-related behaviors such as intimate partner violence and morbid jealousy are more common in males. Principal questionnaire studies suggest that men and women have different modules to process cues of sexual and emotional infidelity. We aimed to elucidate the neural response to sentences depicting sexual and emotional infidelity in men and women using functional magnetic resonance imaging. Although there was no sex difference in the self-rating score of jealousy for sexual and emotional infidelity, men and women showed different brain activation patterns in response to the two types of infidelity. During jealous conditions, men demonstrated greater activation than women in the brain regions involved in sexual/aggressive behaviors such as the amygdala and hypothalamus. In contrast, women demonstrated greater activation in the posterior superior temporal sulcus. Our fMRI results are in favor of the notion that men and women have different neuropsychological modules to process sexual and emotional infidelity. Our findings might contribute to a better understanding of the neural basis of the jealousy-related behaviors predominantly observed in males.