The crystallographic preferred orientation (CPO) of olivine produced during dislocation creep is considered to be the primary cause of elastic anisotropy in Earth's upper mantle and is often used to determine the direction of mantle flow. A fundamental question remains, however, as to whether the alignment of olivine crystals is uniquely produced by dislocation creep. Here we report the development of CPO in iron-free olivine (that is, forsterite) during diffusion creep; the intensity and pattern of CPO depend on temperature and the presence of melt, which control the appearance of crystallographic planes on grain boundaries. Grain boundary sliding on these crystallography-controlled boundaries accommodated by diffusion contributes to grain rotation, resulting in a CPO. We show that strong radial anisotropy is anticipated at temperatures corresponding to depths where melting initiates to depths where strongly anisotropic and low seismic velocities are detected. Conversely, weak anisotropy is anticipated at temperatures corresponding to depths where almost isotropic mantle is found. We propose diffusion creep to be the primary means of mantle flow.