The quantum spin Hall (QSH) state, observed in a zero magnetic field in HgTe quantum wells, respects the time-reversal symmetry and is distinct from quantum Hall (QH) states. We show that the QSH state persists in strong quantizing fields and is identified by counterpropagating (helical) edge channels with nonlinear dispersion inside the band gap. If the Fermi level is shifted into the Landau-quantized conduction or valence band, we find a transition between the QSH and QH regimes. Near the transition the longitudinal conductance of the helical channels is strongly suppressed due to the combined effect of the spectrum nonlinearity and enhanced backscattering. It shows a power-law decay B(-2N) with magnetic field B, determined by the number of backscatterers on the edge N. This suggests a rather simple and practical way to probe the quality of quasiballistic QSH devices using magnetoresistance measurements.