We provide a topological understanding of the phonon Hall effect in dielectrics with Raman spin-phonon coupling. A general expression for phonon Hall conductivity is obtained in terms of the Berry curvature of band structures. We find a nonmonotonic behavior of phonon Hall conductivity as a function of the magnetic field. Moreover, we observe a phase transition in the phonon Hall effect, which corresponds to the sudden change of band topology, characterized by the altering of integer Chern numbers. This can be explained by touching and splitting of phonon bands.