We investigate the time-dependent variance of the fidelity with which an initial narrow wave packet is reconstructed after its dynamics is time reversed with a perturbed Hamiltonian. In the semiclassical regime of perturbation, we show that the variance first rises algebraically up to a critical time t(c) , after which it decays. To leading order in the effective Planck's constant Planck's(eff) , this decay is given by the sum of a classical term approximately same as exp [-2lambdat] , a quantum term approximately same as 2Planck's(eff) exp [-Gamma t] , and a mixed term approximately 2 exp [- (Gamma+lambda) t] . Compared to the behavior of the average fidelity, this allows for the extraction of the classical Lyapunov exponent lambda in a larger parameter range. Our results are confirmed by numerical simulations.