Hematopoietic tyrosine phosphatase (HePTP) is one of three members of the kinase interaction motif (KIM) phosphatase family which also includes STEP and PCPTP1. The KIM-PTPs are characterized by a 15 residue sequence, the KIM, which confers specific high-affinity binding to their only known substrates, the MAP kinases Erk and p38, an interaction which is critical for their ability to regulate processes such as T cell differentiation (HePTP) and neuronal signaling (STEP). The KIM-PTPs are also characterized by a unique set of residues in their PTP substrate binding loops, where 4 of the 13 residues are differentially conserved among the KIM-PTPs as compared to more than 30 other class I PTPs. One of these residues, T106 in HePTP, is either an aspartate or asparagine in nearly every other PTP. Using multiple techniques, we investigate the role of these KIM-PTP specific residues in order to elucidate the molecular basis of substrate recognition by HePTP. First, we used NMR spectroscopy to show that Erk2-derived peptides interact specifically with HePTP at the active site. Next, to reveal the molecular details of this interaction, we solved the high-resolution three-dimensional structures of two distinct HePTP-Erk2 peptide complexes. Strikingly, we were only able to obtain crystals of these transient complexes using a KIM-PTP specific substrate-trapping mutant, in which the KIM-PTP specific residue T106 was mutated to an aspartic acid (T106D). The introduced aspartate side chain facilitates the coordination of the bound peptides, thereby stabilizing the active dephosphorylation complex. These structures establish the essential role of HePTP T106 in restricting HePTP specificity to only those substrates which are able to interact with KIM-PTPs via the KIM (e.g., Erk2, p38). Finally, we describe how this interaction of the KIM is sufficient for overcoming the otherwise weak interaction at the active site of KIM-PTPs.