Both estrogen and testosterone insufficiency has been associated with reduced psychological well-being including fatigue. However, hormonal replacement studies on fatigue are rare. Therefore, we wanted to study the effect of testosterone and estrogen replacement therapy on cognitive fatigue and the relation between sex hormone levels and cognitive fatigue in oophorectomized women. Fifty women with surgically induced menopause (mean age: 54.0 ± 2.9 years) were randomly assigned to treatment with estradiol valerate in combination with testosterone undecanoate or placebo for 24 weeks in a double-blind cross-over study. Neuropsychological tests and questionnaires were used to assess cognitive fatigue and psychological well-being. Cognitive fatigue was significantly associated to poor self-rated health and higher body mass index but not to general psychological well-being or sex hormone levels. Treatment with testosterone + estrogen had no significant effect on cognitive fatigue but the results indicated a curvilinear relation for hormonal levels. The estrogen/testosterone ratio was more related to functions rather than high or low hormone levels per se. We found that cognitive fatigue is frequent in oophorectomized women and negatively associated to self-perceived health and positively associated to BMI. A well-balanced ratio between estrogen and testosterone levels may be important for cognitive fatigue.