A common metabotropic glutamate receptor 5 (mGlu5) allosteric site is known to accommodate diverse chemotypes. However, the structural relationship between compounds from different scaffolds and mGlu5 is not well understood. In an effort to better understand the molecular determinants that govern allosteric modulator interactions with mGlu5, we employed a combination of site-directed mutagenesis and computational modeling. With few exceptions, six residues (P654, Y658, T780, W784, S808, and A809) were identified as key affinity determinants across all seven allosteric modulator scaffolds. To improve our interpretation of how diverse allosteric modulators occupy the common allosteric site, we sampled the wealth of mGlu5 structure-activity relationship (SAR) data available by docking 60 ligands (actives and inactives) representing seven chemical scaffolds into our mGlu5 comparative model. To spatially and chemically compare binding modes of ligands from diverse scaffolds, the ChargeRMSD measure was developed. We found a common binding mode for the modulators that placed the long axes of the ligands parallel to the transmembrane helices 3 and 7. W784 in TM6 not only was identified as a key NAM cooperativity determinant across multiple scaffolds, but also caused a NAM to PAM switch for two different scaffolds. Moreover, a single point mutation in TM5, G747V, altered the architecture of the common allosteric site such that 4-nitro-N-(1,3-diphenyl-1H-pyrazol-5-yl)benzamide (VU29) was noncompetitive with the common allosteric site. Our findings highlight the subtleties of allosteric modulator binding to mGlu5 and demonstrate the utility in incorporating SAR information to strengthen the interpretation and analyses of docking and mutational data.