Endocytosis comprises several routes of internalization. An outstanding question is whether the caveolar and endosomal pathways intersect. Following transport of the caveolar protein Caveolin-1 and two cargo complexes, Simian Virus 40 and Cholera toxin, in live cells, we uncovered a Rab5-dependent pathway in which caveolar vesicles are targeted to early endosomes and form distinct and stable membrane domains. In endosomes, the low pH selectively allowed the toxin to diffuse out of the caveolar domains into the surrounding membrane, while the virus remained trapped. Thus, we conclude that, unlike cyclic assembly and disassembly of coat proteins in vesicular transport, oligomeric complexes of caveolin-1 confer permanent structural stability to caveolar vesicles that transiently interact with endosomes to form subdomains and release cargo selectively by compartment-specific cues.