Androgens, which are mediated via the androgen receptor (AR), play important roles in normal follicular development and female fertility. However, just like a double-edged sword, besides the positive effects of androgen on follicular development, abnormal androgen levels, especially as in hyperandrogenism, seriously suppress normal follicular development. A crucial balance exists between the importance of androgens in follicular development and their negative effects when in excess. As the first meiotic division and epigenetic reprogramming are two critical events in oogenesis, abnormal androgen levels or deficiency in androgen/AR signaling in the ovary may affect these vital events. Oocytes have a tendency to develop genomic instability, thus resulting in an increasing incidence of unpredictable adult diseases. Although many studies have explored the effects of androgens and AR on follicular development, the conclusions are controversial and there has been no thorough review of this topic. This review focuses on the roles of androgens in the physiological process of follicular development, summarizes new insights into the roles of androgens in the arrested development of follicles, and discusses the potential risk of adult diseases originating from abnormal follicular androgen levels or androgen receptor signals, which may determine areas for future studies.