Cognitive decline is a common problem of aging. Whereas multiple neural and glial mechanisms may account for these declines, microglial sensitization and/or dystrophy has emerged as a leading culprit in brain aging and dysfunction. However, glial activation is consistently observed in normal brain aging as well, independent of frank neuroinflammation or functional impairment. Such variability suggests the existence of additional vulnerability factors that can impact neuronal-glial interactions and thus overall brain and cognitive health. The goal of this review is to elucidate our working hypothesis that an individual's risk or resilience to neuroinflammatory disorders and poor cognitive aging may critically depend on their early life experience, which can change immune reactivity within the brain for the remainder of the lifespan. For instance, early-life infection in rats can profoundly disrupt memory function in young adulthood, as well as accelerate age-related cognitive decline, both of which are linked to enduring changes in glial function that occur in response to the initial infection. We discuss these findings within the context of the growing literature on the role of immune molecules and neuroimmune crosstalk in normal brain development. We highlight the intrinsic factors (e.g., chemokines, hormones) that regulate microglial development and their colonization of the embryonic and postnatal brain, and the capacity for disruption or "re-programming" of this crucial process by external events (e.g., stress, infection). An impact on glia, which in turn alters neural development, has the capacity to profoundly impact cognitive and mental health function at all stages of life.