Harmonizing diffusion MRI (dMRI) images across multiple sites is imperative for joint analysis of the data to significantly increase the sample size and statistical power of neuroimaging studies. In this work, we develop a method to harmonize diffusion MRI data across multiple sites and scanners that incorporates two main novelties: i) we take into account the spatial variability of the signal (for different sites) in different parts of the brain as opposed to existing methods, which consider one linear statistical covariate for the entire brain; ii) our method is model-free, in that no a-priori model of diffusion (e.g., tensor, compartmental models, etc.) is assumed and the signal itself is corrected for scanner related differences. We use spherical harmonic basis functions to represent the signal and compute several rotation invariant features, which are used to estimate a regionally specific linear mapping between signal from different sites (and scanners). We validate our method on diffusion data acquired from four different sites (including two GE and two Siemens scanners) on a group of healthy subjects. Diffusion measures such fractional anisotropy, mean diffusivity and generalized fractional anisotropy are compared across multiple sites before and after the mapping. Our experimental results demonstrate that, for identical acquisition protocol across sites, scanner-specific differences can be accurately removed using the proposed method.