In most cells the endocytic system is organized following a common concept that allows for the integrative handling of a variety of housekeeping functions. In addition, variations on the general scheme provide for specialized endosome-based pathways that occur only in specific cell types. The diversity of endosomal functions is not only reflected by characteristic molecular compositions, but also mirrored in their morphological organization. In this review we will first describe the general outline of the endocytic system by combining kinetic, morphological, molecular, and functional definitions. In the second part, adaptations of endosomes that allow their functioning in specialized processes, such as antigen presentation, synaptic vesicle formation, and glucose transport, will be addressed.