The Earth's mantle beneath ocean ridges is widely thought to be depleted by previous melt extraction, but well homogenized by convective stirring. This inference of homogeneity has been complicated by the occurrence of portions enriched in incompatible elements. Here we show that some refractory abyssal peridotites from the ultraslow-spreading Gakkel ridge (Arctic Ocean) have very depleted 187Os/188Os ratios with model ages up to 2 billion years, implying the long-term preservation of refractory domains in the asthenospheric mantle rather than their erasure by mantle convection. The refractory domains would not be sampled by mid-ocean-ridge basalts because they contribute little to the genesis of magmas. We thus suggest that the upwelling mantle beneath mid-ocean ridges is highly heterogeneous, which makes it difficult to constrain its composition by mid-ocean-ridge basalts alone. Furthermore, the existence of ancient domains in oceanic mantle suggests that using osmium model ages to constrain the evolution of continental lithosphere should be approached with caution.