We show that the observed transition from rock salt to orthorhombic P(nma) symmetry in CrN can be understood in terms of stress anisotropy. Using local spin density functional theory, we find that the imbalance between stress stored in spin-paired and spin-unpaired Cr nearest neighbors causes the rock-salt structure to be unstable against distortions and justifies the observed antiferromagnetic ordering. This stress has a purely magnetic origin and may be important in any system where the coupling between spin ordering and structure is strong.