Src homology 3 (SH3) domains were initially characterized as a prevalent protein module that recognizes proline-rich sequences, in particular those containing a PxxP motif. Recent studies have shown that the specificity and cellular function of SH3 domains are far more diverse than previously appreciated. Despite lacking distinguishing features, the ligand-binding surface of an SH3 domain can be molded to accommodate a variety of peptide ligands. Moreover, certain SH3 domains are capable of using surfaces distinct from the canonical ligand-binding site to engage a peptide or protein. The identification of novel motifs and domains recognized by the SH3 domain greatly expands the ligand pool and cellular function for this family. However, this also imposes the question as to how the specificity of the hundreds of human SH3 domains is regulated in a cell to ensure their proper functions. Here we review literature on the specificity of SH3 domains, with an emphasis on the structural basis of ligand recognition, and discuss mechanisms employed by SH3 domain-containing proteins to execute defined cellular functions through highly regulated SH3-ligand interactions.