In fat and skeletal muscle cells, insulin-responsive amino peptidase (IRAP) along with glucose transporter 4 (Glut4) and sortilin, represents a major component protein of the insulin-responsive vesicles (IRVs). Here, we show that IRAP, similar to Glut4 and sortilin, is retrieved from endosomes to the trans-Golgi network by retromer. Unlike Glut4, retrograde transport of IRAP does not require sortilin, as retromer can directly bind to the cytoplasmic tail of IRAP. Ablation of IRAP in 3T3-L1 adipocytes shifts the endosomal pool of Glut4 to more acidic endosomes, but does not affect IRV targeting, stability, and insulin responsiveness of Glut4.