Semiconductor fabrication is a mainstay of modern civilization, enabling the myriad applications and technologies that underpin everyday life. However, while sub-10 nanometer devices are already entering the mainstream, the end of the Moore's law roadmap still lacks tools capable of bulk semiconductor fabrication on sub-nanometer and atomic levels, with probe-based manipulation being explored as the only known pathway. Here we demonstrate that the atomic-sized focused beam of a scanning transmission electron microscope can be used to manipulate semiconductors such as Si on the atomic level, inducing growth of crystalline Si from the amorphous phase, reentrant amorphization, milling, and dopant front motion. These phenomena are visualized in real-time with atomic resolution. We further implement active feedback control based on real-time image analytics to automatically control the e-beam motion, enabling shape control and providing a pathway for atom-by-atom correction of fabricated structures in the near future. These observations open a new epoch for atom-by-atom manufacturing in bulk, the long-held dream of nanotechnology.