The in situ or authigenic formation of calcium phosphate minerals in marine sediments is a major sink for the vital nutrient phosphorus. However, because typical sediment chemistry is not kinetically conducive to the precipitation of these minerals, the mechanism behind their formation has remained a fundamental mystery. Here, we present evidence from high-sensitivity x-ray and electrodialysis techniques to describe a mechanism by which abundant diatom-derived polyphosphates play a critical role in the formation of calcium phosphate minerals in marine sediments. This mechanism can explain the puzzlingly dispersed distribution of calcium phosphate minerals observed in marine sediments worldwide.