We report on the passive measurement of time-dependent Green's functions in the optical frequency domain with low-coherence interferometry. Inspired by previous studies in acoustics and seismology, we show how the correlations of a broadband and incoherent wave field can directly yield the Green's functions between scatterers of a complex medium. Both the ballistic and multiple scattering components of the Green's function are retrieved. This approach opens important perspectives for optical imaging and characterization in complex scattering media.