Mucosal immunity is an important correlate of protection against respiratory infections such as SARS-CoV-2. Comparing B cell responses in the upper respiratory tract following vaccination and infection may offer unique insights into mucosal immunity. Here, we characterized antigen-specific B cells in the tonsils, adenoids, and peripheral blood of children who had been infected with SARS-CoV-2 or vaccinated with SARS-CoV-2 mRNA vaccines. SARS-CoV-2-specific switched memory B cells (BSM) and germinal center B cells were found in the blood and pharyngeal lymphoid tissues after vaccination or infection. However, infection generated a higher proportion of IgA+ BSM and CXCR3+CD21+ BSM, which showed distinct spatial localization, greater clonal expansion and increased propensity for plasma cell differentiation compared to their CXCR3- counterparts, accompanied by persistent activation of innate and T follicular helper cells in the tissues. Our data provide evidence for tissue-specific B cell memory after either SARS-CoV-2 vaccination or infection, but with distinct characteristics that can influence the quality, durability, and localization of immunity.