Numerous biological processes rely on endocytosis. The construction of endocytic pits is achieved by a bewildering complexity of biochemical factors that function in clathrin-dependent and -independent pathways. In this review, we argue that this complexity can be conceptualized by a deceptively small number of physical principles that fall into two broad categories: passive mechanisms, such as asymmetric transbilayer stress, scaffolding, line tension, and crowding, and active mechanisms driven by mechanochemical enzymes and/or cytoskeleton. We illustrate how the functional identity of biochemical modules depends on system parameters such as local protein density on membranes, thus explaining some of the controversy in the field. Different modules frequently operate in parallel in the same step and often are shared by apparently divergent uptake processes. The emergence of a novel endocytic classification system may thus be envisioned in which functional modules are the elementary bricks.