A novel time-resolved diagnostic is used to record the critical surface motion during picosecond-scale relativistic laser interaction with a solid target. Single-shot measurements of the specular light show a redshift decreasing with time during the interaction, corresponding to a slowing-down of the hole boring process into overdense plasma. On-shot full characterization of the laser pulse enables simulations of the experiment without any free parameters. Two-dimensional particle-in-cell simulations yield redshifts that agree with the data, and support a simple explanation of the slowing-down of the critical surface based on momentum conservation between ions and reflected laser light.