Olive stoning during the virgin olive oil (VOO) mechanical extraction process was studied to show the effect on the phenolic and volatile composition of the oil. To study the impact of the constitutive parts of the fruit in the composition of olive pastes during processing, the phenolic compounds and several enzymatic activities such as polyphenoloxidase (PPO), peroxidase (POD), and lipoxygenase (LPO) of the olive pulp, stone, and seed were also studied. The olive pulp showed large amounts of oleuropein, demethyloleuropein, and lignans, while the contribution of the stone and the seed in the overall phenolic composition of the fruit was very low. The occurrence of crushed stone in the pastes, during malaxation, increased the peroxidase activity in the pastes, reducing the phenolic concentration in VOO and, at the same time, modifying the composition of volatile compounds produced by the lipoxygenase pathway. The oil obtained from stoned olive pastes contained higher amounts of secoiridoid derivatives such as the dialdehydic forms of elenolic acid linked to (3,4-dihydroxyphenyl)ethanol and (p-hydroxyphenyl)ethanol (3,4-DHPEA-EDA and p-HPEA-EDA, respectively) and the isomer of the oleuropein aglycon (3,4-DHPEA-EA) and, at the same time, did not show significant variations of lignans. The stoning process modified the volatile profile of VOO by increasing the C6 unsaturated aldehydes that are strictly related to the cut-grass sensory notes of the oil.