HIV-infected infants are at an increased risk of progressing rapidly to AIDS in the first weeks of life. Here, we evaluated immunological and virological parameters in 25 SIV-infected infant rhesus macaques to understand the factors influencing a rapid disease outcome. Infant macaques were infected with SIVmac251 and monitored for 10 to 17 weeks post-infection. SIV-infected infants were divided into either typical (TypP) or rapid (RP) progressor groups based on levels of plasma anti-SIV antibody and viral load, with RP infants having low SIV-specific antibodies and high viral loads. Following SIV infection, 11 out of 25 infant macaques exhibited an RP phenotype. Interestingly, TypP had lower levels of total CD4 T cells, similar reductions in CD4/CD8 ratios and elevated activation of CD8 T cells, as measured by the levels of HLA-DR, compared to RP. Differences between the two groups were identified in other immune cell populations, including a failure to expand activated memory (CD21-CD27+) B cells in peripheral blood in RP infant macaques, as well as reduced levels of germinal center (GC) B cells and T follicular helper (Tfh) cells in spleens (4- and 10-weeks post-SIV). Reduced B cell proliferation in splenic germinal GCs was associated with increased SIV+ cell density and follicular type 1 interferon (IFN)-induced immune activation. Further analyses determined that at 2-weeks post SIV infection TypP infants exhibited elevated levels of the GC-inducing chemokine CXCL13 in plasma, as well as significantly lower levels of viral envelope diversity compared to RP infants. Our findings provide evidence that early viral and immunologic events following SIV infection contributes to impairment of B cells, Tfh cells and germinal center formation, ultimately impeding the development of SIV-specific antibody responses in rapidly progressing infant macaques.