The durability of infection-induced SARS-CoV-2 immunity has major implications for reinfection and vaccine development. Here, we show a comprehensive profile of antibody, B cell and T cell dynamics over time in a cohort of patients who have recovered from mild-moderate COVID-19. Binding and neutralising antibody responses, together with individual serum clonotypes, decay over the first 4 months post-infection. A similar decline in Spike-specific CD4+ and circulating T follicular helper frequencies occurs. By contrast, S-specific IgG+ memory B cells consistently accumulate over time, eventually comprising a substantial fraction of circulating the memory B cell pool. Modelling of the concomitant immune kinetics predicts maintenance of serological neutralising activity above a titre of 1:40 in 50% of convalescent participants to 74 days, although there is probably additive protection from B cell and T cell immunity. This study indicates that SARS-CoV-2 immunity after infection might be transiently protective at a population level. Therefore, SARS-CoV-2 vaccines might require greater immunogenicity and durability than natural infection to drive long-term protection.