The receptor-like protein tyrosine phosphatases (RPTP) mu and RPTP kappa have a modular ectodomain consisting of four fibronectin type III-like repeats, a single Ig-like domain, and a newly identified N-terminal MAM domain. The function of the latter module, which comprises about 160 amino acids and is found in diverse transmembrane proteins, is not known. We previously reported that both RPTP mu and RPTP kappa can mediate homophilic cell interactions when expressed in insect cells. Here we show that despite their striking structural similarity, RPTP mu and RPTP kappa fail to interact in a heterophilic manner. To examine the role of the MAM domain in homophilic binding, we expressed a mutant RPTP mu lacking the MAM domain in insect Sf9 cells. Truncated RPTP mu is properly expressed at the cell surface but fails to promote cell-cell adhesion. Homophilic cell adhesion is fully restored in a chimeric RPTP mu molecule containing the MAM domain of RPTP kappa. However, this chimeric RPTP mu does not interact with either RPTP mu or RPTP kappa. These results indicate that the MAM domain of RPTP mu and RPTP kappa is essential for homophilic cell-cell interaction and helps determine the specificity of these interactions.