The intestinal intraepithelial lymphocytes (IELs) are mostly T cells dispersed as single cells within the epithelial cell layer that surrounds the intestinal lumen. IELs are, therefore, strategically located at the interface between the antigen-rich outside world and the sterile core of the body. The intestine of higher vertebrates has further evolved to harbor numerous commensal bacteria that carry out important functions for the host, and while defensive immunity can effectively protect against the invasion of pathogens, similar immune reactions against food-derived antigens or harmless colonizing bacteria can result in unnecessary and sometimes damaging immune responses. Probably as a result of this unique dilemma imposed by the gut environment, multiple subsets of IEL have differentiated, which all display characteristics of 'activated yet resting' immune cells. Despite this common feature, IELs are heterogeneous with regard to their phenotype, ontogeny, and function. In this review, we discuss the different subtypes of IELs and highlight the distinct pathways they took that led to their unique differentiation into highly specialized effector memory T cells, which provide the most effective immune protection yet in a strictly regulated fashion to preserve the integrity and vital functions of the intestinal mucosal epithelium.