Musashi1 (Msi1) is an RNA-binding protein that is highly expressed in neural stem cells. We previously reported that Msi1 contributes to the maintenance of the immature state and self-renewal activity of neural stem cells through translational repression of m-Numb. However, its translation repression mechanism has remained unclear. Here, we identify poly(A) binding protein (PABP) as an Msi1-binding protein, and find Msi1 competes with eIF4G for PABP binding. This competition inhibits translation initiation of Msi1's target mRNA. Indeed, deletion of the PABP-interacting domain in Msi1 abolishes its function. We demonstrate that Msi1 inhibits the assembly of the 80S, but not the 48S, ribosome complex. Consistent with these conclusions, Msi1 colocalizes with PABP and is recruited into stress granules, which contain the stalled preinitiation complex. However, Msi1 with mutations in two RNA recognition motifs fails to accumulate into stress granules. These results provide insight into the mechanism by which sequence-specific translational repression occurs in stem cells through the control of translation initiation.