Vascular disease has been proposed as a contributing factor for presbyacusis (age-related hearing loss). While this hypothesis is supported by pathological evidence of vascular decline in post-mortem human and animal studies, evidence in human subjects has been mixed with associations typically reported between a measure of vascular health and low frequency hearing in older women. Given the difficulty of characterizing the in vivo health of the cochlear artery in humans, an estimate of cerebral small vessel disease was used to test the prediction that age-related change in low frequency hearing and not high frequency hearing is related to a global decline in vascular health. We examined the extent to which these associations were specific to women and influenced by a history of high blood pressure in 72 older adults (mean age 67.12 years, SD = 8.79). Probability estimates of periventricular white matter hyperintensities (WMH) from T1- and fluid attenuated T2-weighted magnetic resonance images were significantly associated with a low frequency hearing metric across the sample, which were independent of age, but driven by women and people with a history of high blood pressure. These results support the premise that vascular declines are one mechanism underlying age-related changes in low frequency hearing.