Being recalled for further diagnostic procedures after an abnormal screening mammogram (ASM) can evoke a high state anxiety with lowered quality of life (QoL). We examined whether these adverse psychological consequences are found in all women with benign breast disease (BBD) or are particular to women referred after ASM. In addition, the influence of the anxiety as a personality characteristic (trait anxiety) was studied. Between September 2002 and February 2010 we performed a prospective longitudinal study in six Dutch hospitals. Women referred after ASM or with a palpable lump in the breast (PL), who were subsequently diagnosed with BBD, were included. Before diagnosis (at referral) and during follow-up, questionnaires were completed examining trait anxiety (at referral), state anxiety, depressive symptoms (at referral, one, three and 6 months after diagnosis), and QoL (at referral and 12 months). Women referred after ASM (N=363) were compared with women with PL (N=401). A similar state anxiety score was found in both groups, but a lower psychological QoL score at 12 months was seen in the ASM group. In women with not-high trait anxiety those in the ASM group were more anxious with more depressive symptoms at referral, and reported impaired psychological QoL at referral and at 12 months compared with the PL group. No differences were found between ASM and PL in women with high trait anxiety, but this group scored unfavorably on anxiety, depressive symptoms and QoL compared with women with not-high trait anxiety. ASM evokes more anxiety and depressive symptoms and lowered QoL compared with women referred with PL, especially in women who are not prone to anxiety. Women should be fully informed properly about the risks and benefits of breast cancer screening programs. We recommend identifying women at risk of reduced QoL using a psychometric test.