In x-ray based imaging of the breast, contrast between fibroglandular (Fg) tissue and adipose (Ad) tissue is a source of anatomic noise. The goal of this work was to validate by simulation and experiment a mathematical framework for modelling the Fg component of anatomic noise in digital mammograpy (DM) and dual-energy (DE) DM. Our mathematical framework unifies and generalizes existing approaches. We compared mathematical predictions directly with empirical measurements of the anatomic noise power spectrum of the CIRS BR3D structured breast phantom using two clinical mammography systems and four beam qualities. Our simulation and experimental results showed agreement with mathematical predictions. As a demonstration of utility, we used our mathematical framework in a theoretical spectral optimization of DM for the task of detecting breast masses. Our theoretical optimization showed that the optimal tube voltage for DM may be higher than that based on predictions that do not account for anatomic noise, in agreement with recent theoretical findings. Additionally, our theoretical optimization predicts that filtering tungsten-anode x-ray spectra with rhodium has little influence on lesion detectability, in contrast with previous findings. The mathematical methods validated in this work can be incorporated easily into cascaded systems analysis of breast imaging systems and will be useful when optimizating novel techniques for x-ray-based imaging of the breast.