The coupling between spin and charge degrees of freedom in a crystal gives rise to magneto-optical effects with applications in the sensitive detection of local magnetic order, optical modulation and data storage. In two-dimensional magnets these effects manifest themselves in the large magneto-optical Kerr effect1,2, spontaneous helical light emission3,4 from ferromagnetic (FM) monolayers and electric-field induced Kerr rotation5-7 and giant second-order non-reciprocal optical effects8 in antiferromagnetic (AFM) bilayers. Here we demonstrate the tuning of inelastically scattered light through symmetry control in atomically thin chromium triiodide (CrI3). In monolayers, we found an extraordinarily large magneto-optical Raman effect from an A1g phonon mode due to the emergence of FM order. The linearly polarized, inelastically scattered light rotates by ~40°, more than two orders of magnitude larger than the rotation from the magneto-optical Kerr effect under the same experimental conditions. In CrI3 bilayers, the same phonon mode becomes Davydov-split into two modes of opposite parity, which exhibit divergent selection rules that depend on inversion symmetry and the underlying magnetic order. We demonstrate the magneto-electrical control over these selection rules by activating or suppressing Raman activity for the odd-parity phonon mode and the magneto-optical rotation of scattered light from the even-parity phonon mode. Our work underlines the unique opportunities provided by two-dimensional magnets to control the combined time-reversal and inversion symmetries to manipulate Raman optical selection rules and for exploring emergent magneto-optical effects and spin-phonon coupled physics.