As of November 12, 2020, the mortality to incidence ratio (MIR) of COVID-19 was 5.8% in the US. A longitudinal model-based clustering system on the disease trajectories over time was used to identify "vulnerable" clusters of counties that would benefit from allocating additional resources by federal, state and county policymakers. County-level COVID-19 cases and deaths, together with a set of potential risk factors were collected for 3050 U.S. counties during the 1st wave of COVID-19 (Mar25-Jun3, 2020), followed by similar data for 1344 counties (in the "sunbelt" region of the country) during the 2nd wave (Jun4-Sep2, 2020), and finally for 1055 counties located broadly in the great plains region of the country during the 3rd wave (Sep3-Nov12, 2020). We used growth mixture models to identify clusters of counties exhibiting similar COVID-19 MIR growth trajectories and risk-factors over time. The analysis identifies "more vulnerable" clusters during the 1st, 2nd and 3rd waves of COVID-19. Further, tuberculosis (OR 1.3-2.1-3.2), drug use disorder (OR 1.1), hepatitis (OR 13.1), HIV/AIDS (OR 2.3), cardiomyopathy and myocarditis (OR 1.3), diabetes (OR 1.2), mesothelioma (OR 9.3) were significantly associated with increased odds of being in a more vulnerable cluster. Heart complications and cancer were the main risk factors increasing the COVID-19 MIR (range 0.08-0.52% MIR↑). We identified "more vulnerable" county-clusters exhibiting the highest COVID-19 MIR trajectories, indicating that enhancing the capacity and access to healthcare resources would be key to successfully manage COVID-19 in these clusters. These findings provide insights for public health policymakers on the groups of people and locations they need to pay particular attention while managing the COVID-19 epidemic.