Viscoelasticity is an important diagnostic parameter to investigate physiological dysfunctions in biological tissues. This Letter reports the quantification of viscoelastic parameters by Rayleigh wave tracing on the surface of tissue-mimicking phantoms using holographic imaging. The Rayleigh wave is induced by an electromechanical actuator on the surface of oil-in-gelatin phantoms and a biological tissue sample followed by holographic imaging and reconstruction of the wave. The frequency-dependent velocity dispersion is fitted to a Voigt model for the quantification of viscous and elastic moduli. The viscoelastic parameters calculated by the proposed method are validated by comparing the results from a conventional mechanical rheometer.