In off-axis digital holography, spatial filtering is a key problem limiting the quality of reconstructed image, especially in the case of spurious spectrum generated by coherent noise in the hologram spectrum. In this paper, a new spatial filtering method with spurious spectrum elimination is proposed. Side band centering judgment is firstly implemented to locate the center point of the +1 term in the hologram spectrum. Then by roughly recognizing the region of +1 term spectrum, most of the -1 term, 0 term and the spurious spectral components are eliminated. Finally, Butterworth filtering is performed to extract the +1 term spectrum as enough as possible without introducing the spurious spectrum. Simulated hologram of E-shaped specimen with the spurious spectrum is generated to evaluate the performance of the proposed method. Experimental data of USAF 1951 resolution target, ovarian slice and microlens array are adopted to verify the effectiveness of the proposed method. Simulation and experimental results demonstrated that the proposed method is able to accurately extract the +1 term spectrum with spurious spectrum elimination and achieve a relatively good balance between the structural detail characterization and noise suppression.