Stress has a critical impact on affective and cognitive processing. Based on rodent data suggesting that endocannabinoid signaling via CB1 receptors serves as an emotional buffer, we hypothesized that a common variant of the gene coding for the CB1 receptor modulates affective processing under stress (CNR1; rs1049353 A vs G allele). Therefore, 139 participants, genotyped for this polymorphism, underwent a stress or control manipulation before they viewed emotionally neutral and negative pictures in a magnetic resonance imaging scanner. The ventromedial prefrontal cortex, known for its crucial role in emotion regulation, was significantly more activated in AA/AG vs GG genotype carriers when viewing negative pictures after stress. Under no-stress conditions, AA/AG genotype carriers showed enhanced crosstalk between the ventrolateral prefrontal cortex and the amygdala. We further assessed participants' 24 h-delayed memory for the presented pictures and found that memory performance correlated with amygdala and hippocampus activity and connectivity in stressed carriers of the AA/AG but not the GG genotype. These findings underline the modulatory role of the endocannabinoid system in stress effects on emotion and cognition and provide insights into the neural mechanisms that may contribute to the suggested protective effect of the AA/AG genotype of the CB1 receptor polymorphism against stress-related psychopathologies.