The physical nature of the bacterial chromosome has important implications for its function. Using high-resolution dynamic tracking, we observe the existence of rare but ubiquitous 'rapid movements' of chromosomal loci exhibiting near-ballistic dynamics. This suggests that these movements are either driven by an active machinery or part of stress-relaxation mechanisms. Comparison with a null physical model for subdiffusive chromosomal dynamics shows that rapid movements are excursions from a basal subdiffusive dynamics, likely due to driven and/or stress-relaxation motion. Additionally, rapid movements are in some cases coupled with known transitions of chromosomal segregation. They do not co-occur strictly with replication, their frequency varies with growth condition and chromosomal coordinate, and they show a preference for longitudinal motion. These findings support an emerging picture of the bacterial chromosome as off-equilibrium active matter and help developing a correct physical model of its in vivo dynamic structure.