Many plasma membrane proteins in yeast are ubiquitinated and endocytosed, but how they are recognized for modification has remained unknown. Here, we show that the manganese transporter Smf1 is endocytosed when cells are exposed to cadmium ions, that this endocytosis depends on Rsp5-dependent ubiquitination of specific lysines and that it also requires phosphorylation at nearby sites. This phosphorylation is, however, constitutive rather than stress-induced. Efficient ubiquitination requires Ecm21 or Csr2, two members of a family of arrestin-like yeast proteins that contain several PY motifs and bind to Rsp5. Ecm21 also binds to phosphorylated Smf1, providing a link between Rsp5 and its substrate. PY motif-containing arrestin-like proteins are found in many species, including humans, and might have a general role as ubiquitin ligase adaptors.