Ultrafast dynamics of surface-enhanced Raman scattering (SERS) was investigated at cleaved graphite surfaces bearing deposited gold (Au) nanostructures (∼10 nm in diameter) by using sensitive pump-probe reflectivity spectroscopy with ultrashort (7.5 fs) laser pulses. We observed enhancement of phonon amplitudes (C═C stretching modes) in the femtosecond time domain, considered to be due to the enhanced electromagnetic (EM) field around the Au nanostructures. Finite-difference time-domain (FDTD) calculations confirmed the EM enhancement. The enhancement causes drastic increase of coherent D-mode (40 THz) phonon amplitude and nanostructure-dependent changes in the amplitude and dephasing time of coherent G-mode (47 THz) phonons. This methodology should be suitable to study the basic mechanism of SERS and may also find application in nanofabrication.