Immune responses from prior severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection and COVID-19 vaccination mitigate disease severity, but they do not fully prevent subsequent infections, especially from genetically divergent strains. We examined the incidence of and immune differences against human endemic coronaviruses (eCoVs) as a proxy for response against future genetically heterologous coronaviruses (CoVs). We assessed differences in symptomatic eCoV and non-CoV respiratory disease incidence among those with known prior SARS-CoV-2 infection or previous COVID-19 vaccination but no documented SARS-CoV-2 infection or neither exposure. Retrospective cohort analyses suggest that prior SARS-CoV-2 infection, but not previous COVID-19 vaccination alone, associates with a lower incidence of subsequent symptomatic eCoV infection. There was no difference in non-CoV incidence, implying that the observed difference was eCoV specific. In a second cohort where both cellular and humoral immunity were measured, those with prior SARS-CoV-2 spike protein exposure had lower eCoV-directed neutralizing antibodies, suggesting that neutralization is not responsible for the observed decreased eCoV disease. The three groups had similar cellular responses against the eCoV spike protein and nucleocapsid antigens. However, CD8+ T cell responses to the nonstructural eCoV proteins nsp12 and nsp13 were higher in individuals with previous SARS-CoV-2 infection as compared with the other groups. This association between prior SARS-CoV-2 infection and decreased incidence of eCoV disease may therefore be due to a boost in CD8+ T cell responses against eCoV nsp12 and nsp13, suggesting that incorporation of nonstructural viral antigens in a future pan-CoV vaccine may improve vaccine efficacy.