The surface plasmon polaritons (SPPs) of graphene reflect the microscopic spatial variations of underlying electronic structure and dynamics. Here, we excite and image the graphene SPP response in phase and amplitude by near-field interferometry. We develop an analytic cavity model that can self-consistently describe the SPP response function for edge, grain boundary, and defect SPP reflection and scattering. The derived SPP wave vector, damping, and carrier mobility agree with the results from more complex models. Spatial variations in the Fermi level and associated variations in dopant concentration reveal a nanoscale spatial inhomogeneity in the reduced conductivity at internal boundaries. The additional SPP phase information thus opens a new degree of freedom for spatial and spectral graphene SPP tuning and modulation for optoelectronics applications.