The chromosomes-DNA polymers and their binding proteins-are compacted into a spatially organized, yet dynamic, three-dimensional structure. Recent genome-wide chromatin conformation capture experiments reveal a hierarchical organization of the DNA structure that is imposed, at least in part, by looping interactions arising from the activity of loop extrusion factors. The dynamics of chromatin reflects the response of the polymer to a combination of thermal fluctuations and active processes. However, how chromosome structure and enzymes acting on chromatin together define its dynamics remains poorly understood. To gain insight into the structure-dynamics relationship of chromatin, we combine high-precision microscopy in living Schizosaccharomyces pombe cells with systematic genetic perturbations and Rouse model polymer simulations. We first investigated how the activity of two loop extrusion factors, the cohesin and condensin complexes, influences chromatin dynamics. We observed that deactivating cohesin, or to a lesser extent condensin, increased chromatin mobility, suggesting that loop extrusion constrains rather than agitates chromatin motion. Our corresponding simulations reveal that the introduction of loops is sufficient to explain the constraining activity of loop extrusion factors, highlighting that the conformation adopted by the polymer plays a key role in defining its dynamics. Moreover, we find that the number of loops or residence times of loop extrusion factors influence the dynamic behavior of the chromatin polymer. Last, we observe that the activity of the INO80 chromatin remodeler, but not the SWI/SNF or RSC complexes, is critical for ATP-dependent chromatin mobility in fission yeast. Taking the data together, we suggest that thermal and INO80-dependent activities exert forces that drive chromatin fluctuations, which are constrained by the organization of the chromosome into loops.