Higher levels of serum phosphorus that remain within the reference range are associated with increased risk of cardiovascular disease (CVD). However, the mechanisms by which higher serum phosphorus concentrations may contribute to the development of CVD remain unclear. Cross-sectional association between serum phosphorus levels and arterial stiffness as estimated by an ankle brachial pressure index (ABPI) >1.3 was examined in 581 participants in the Third National Health and Nutrition Examination Survey. Logistic regression analysis was performed to evaluate whether higher serum phosphorus levels were associated with high ABPI, independently of several potential confounders. Of 581 participants, 38% and 10% had a serum phosphorus levels >3.5 and >4.0 mg/dl, respectively. An ABPI >1.3 was present in 7.3% of participants. Higher quartiles of serum phosphorus levels were associated with a greater prevalence of high ABPI: 5.4%, 3.7%, 7.8%, and 12.9% for quartiles 1 (<3.1 mg/dl), 2 (3.1 to 3.4 mg/dl), 3 (3.4 to 3.7 mg/dl), and 4 (3.7 to 5.0 mg/dl), respectively. There was a strong, positive association between the highest quartile of serum phosphorus (3.7 to 5.0 mg/dl) and high ABPI compared to the reference group (3.1 to 3.4 mg/dl) after adjustment for demographics, traditional CVD risk factors, kidney function, C-reactive protein, serum calcium, and 25-hydroxyvitamin D levels (adjusted odds ratio 4.78, 95% confidence interval 1.73 to 13.2, p = 0.003). In conclusion, serum phosphorus levels, even within the reference range, are independently associated with high ABPI, a marker of arterial stiffness, in the US adult population.