The relationship between the particle mobility and magnetorheological effect was investigated for polyurethane magnetic elastomers containing carbonyl iron particles with various cross-linking densities or plasticizer concentrations. The storage modulus at 0 mT increased and the on-field modulus at 500 mT decreased with the cross-linking density. The critical magnetic field where the storage modulus starts to rise up increased with the cross-linking density, indicating that the movement of magnetic particles is depressed by the cross-linking points of the polyurethane network. Magnetic elastomers with various plasticizer concentrations revealed that the storage modulus at 0 mT decreased and the on-field modulus at 500 mT increased with the plasticizer concentration. The critical magnetic field decreased with increasing plasticizer concentration, indicating that a dense polyurethane network prevents magnetic particles from moving. It was found that the change in the modulus due to the magnetic field can be scaled by the storage modulus at 0 mT as well as the critical magnetic field. Thus, there is a certain correlation between the macroscopic modulus of elasticity (storage modulus at 0 mT) and the microscopic mobility of magnetic particles reflected in the critical magnetic field.