Specific intermolecular interactions are largely guided by electrostatics. However, the most common model for electrostatic interactions-atomic point charges-fails to reproduce anisotropic charge distributions, such as lone pairs and sigma holes. Although this has long been known, point charges are still widely used in chemical modeling and reasoning. In this contribution, we analyze the deficiency of atomic point charges in reproducing the electrostatic potential (ESP) around molecules and find that multipole moments up to quadrupoles can, with a much lower error than point charges, reproduce the relevant ESP for all cases. Mapping the surface to the closest atom allows to compare ESP errors between atom types and to identify cases with the most urgent need for atomic multipoles. Our analysis shows that almost all heteroatoms require multipoles to correctly describe their charge distribution, with the most serious cases being nitrogen, sulfur, and halogens. Comparison with small molecule crystallography data studies supports our findings and emphasizes the need for incorporating anisotropic charge descriptions in chemical models. The scheme introduced here can be used to identify anisotropic binding preferences for atom types where there is too little coverage in crystal structure databases.