The sources of broad backgrounds in visible-near-IR absorption spectra of single-walled carbon nanotube (SWCNT) dispersions are studied through a series of controlled experiments. Chemical functionalization of nanotube sidewalls generates background absorption while broadening and red-shifting the resonant transitions. Extensive ultrasonic agitation induces a similar background component that may reflect unintended chemical changes to the SWCNTs. No major differences are found between spectral backgrounds in sample fractions with average lengths between 120 and 650 nm. Broad background absorption from amorphous carbon is observed and quantified. Overlapping resonant absorption bands lead to elevated backgrounds from spectral congestion in samples containing many SWCNT structural species. A spectral modeling method is described for separating the background contributions from spectral congestion and other sources. Nanotube aggregation increases congestion backgrounds by broadening the resonant peaks. Essentially no background is seen in sorted pristine samples enriched in a single semiconducting (n,m) species. By contrast, samples enriched in mixed metallic SWCNTs show broad intrinsic absorption backgrounds far from the resonant transitions. The shape of this metallic background component and its absorptivity coefficient are quantitatively assessed. The results obtained here suggest procedures for preparing SWCNT dispersions with minimal extrinsic background absorptions and for quantifying the remaining intrinsic components. These findings should allow improved characterization of SWCNT samples by absorption spectroscopy.