New functional materials can in principle be created using colloids that self-assemble into a desired structure by means of a programmable recognition and binding scheme. This idea has been explored by attaching 'programmed' DNA strands to nanometre- and micrometre- sized particles and then using DNA hybridization to direct the placement of the particles in the final assembly. Here we demonstrate an alternative recognition mechanism for directing the assembly of composite structures, based on particles with complementary shapes. Our system, which uses Fischer's lock-and-key principle, employs colloidal spheres as keys and monodisperse colloidal particles with a spherical cavity as locks that bind spontaneously and reversibly via the depletion interaction. The lock-and-key binding is specific because it is controlled by how closely the size of a spherical colloidal key particle matches the radius of the spherical cavity of the lock particle. The strength of the binding can be further tuned by adjusting the solution composition or temperature. The composite assemblies have the unique feature of having flexible bonds, allowing us to produce flexible dimeric, trimeric and tetrameric colloidal molecules as well as more complex colloidal polymers. We expect that this lock-and-key recognition mechanism will find wider use as a means of programming and directing colloidal self-assembly.