Individuals in social insect colonies cooperate to perform collective work. While colonies often respond to changing environmental conditions by flexibly reallocating workers to different tasks, the factors determining which workers switch and why are not well understood. Here, we use an automated tracking system to continuously monitor nest behavior and foraging activity of uniquely identified workers from entire bumble bee (Bombus impatiens) colonies foraging in a natural outdoor environment. We show that most foraging is performed by a small number of workers and that the intensity and distribution of foraging is actively regulated at the colony level in response to forager removal. By analyzing worker nest behavior before and after forager removal, we show that spatial fidelity of workers within the nest generates uneven interaction with relevant localized information sources, and predicts which workers initiate foraging after disturbance. Our results highlight the importance of spatial fidelity for structuring information flow and regulating collective behavior in social insect colonies.