SARS-CoV-2 uses the double-membrane vesicles as replication organelles. However, how virion assembly occurs has not been fully understood. Here we identified a SARS-CoV-2-driven membrane structure named the 3a dense body (3DB). 3DBs have unusual electron-dense and dynamic inner structures, and their formation is driven by the accessory protein ORF3a via hijacking a specific subset of the trans-Golgi network (TGN) and early endosomal membranes. 3DB formation is conserved in related bat and pangolin coronaviruses yet lost during the evolution to SARS-CoV. 3DBs recruit the viral structural proteins spike (S) and membrane (M) and undergo dynamic fusion/fission to facilitate efficient virion assembly. A recombinant SARS-CoV-2 virus with an ORF3a mutant specifically defective in 3DB formation showed dramatically reduced infectivity for both extracellular and cell-associated virions. Our study uncovers the crucial role of 3DB in optimal SARS-CoV-2 infectivity and highlights its potential as a target for COVID-19 prophylactics and therapeutics.