Photonic circuits, in which beams of light redirect the flow of other beams of light, are a long-standing goal for developing highly integrated optical communication components. Furthermore, it is highly desirable to use silicon--the dominant material in the microelectronic industry--as the platform for such circuits. Photonic structures that bend, split, couple and filter light have recently been demonstrated in silicon, but the flow of light in these structures is predetermined and cannot be readily modulated during operation. All-optical switches and modulators have been demonstrated with III-V compound semiconductors, but achieving the same in silicon is challenging owing to its relatively weak nonlinear optical properties. Indeed, all-optical switching in silicon has only been achieved by using extremely high powers in large or non-planar structures, where the modulated light is propagating out-of-plane. Such high powers, large dimensions and non-planar geometries are inappropriate for effective on-chip integration. Here we present the experimental demonstration of fast all-optical switching on silicon using highly light-confining structures to enhance the sensitivity of light to small changes in refractive index. The transmission of the structure can be modulated by up to 94% in less than 500 ps using light pulses with energies as low as 25 pJ. These results confirm the recent theoretical prediction of efficient optical switching in silicon using resonant structures.