First-order Raman scattering and multiphonons are studied in RVO3 (R = Ho and Y) as a function of temperature in the orthorhombic and monoclinic phases. Raman spectra of HoVO3 and YVO3 unveil similar features since both compounds have nearly identical R-radii. However, the most important difference lies in the transition temperature involving the V(3+) orbitals, the V(3+) magnetic moments as well as the crystallographic structure. Particularly, the magnetic and orbital reorientations occur at T N2 = 40 K for HoVO3 instead T N2 =77 K in the case of YVO3. For both systems, anomalous phonon shifts which are related to spin-phonon coupling are observed below the V(3+) magnetic ordering temperature (T N1 ≈ 110 K) while additional phonon anomalies are exclusively observed in HoVO3 around T (*) ≈ 15 K. On the other hand, infrared (IR) transmittance measurements as a function of temperature reveal Ho(3+5)I8 → (5)I7 excitations and additional excitations assigned as vibronics. These latter combined with drastic changes in Ho(3+5)I8 → (5)I7 excitations at T N2, are indicative of a strong coupling between the Ho(3+) ions and the ligand field. This could explain the large magnetocaloric capacity shown by HoVO3.