Advanced glycation end products (AGEs), which are readily formed and accumulated with sustained hyperglycemia, contribute to the development of diabetic complications. As a consequence, inhibition of AGE formation constitutes an attractive therapeutic/preventive target. In the current study, we explored the phytochemical composition and the in vitro effect of two different olive leaf extracts (an aqueous and a methanolic) on AGE formation. The methanolic olive leaf extract inhibited fluorescent AGE formation in a bovine serum albumin (BSA)-ribose system, whereas the aqueous extract had no effect in both BSA-fructose and BSA-ribose systems. The phytochemical profile was investigated with liquid chromatography-ultraviolet-visible (UV-Vis) diode array coupled to electrospray ionization multistage mass spectrometry (LC/DAD/ESI-MS(n)). Quantification of the major phenolic compounds was performed with high performance liquid chromatography with UV-Vis diode array detection and nuclear magnetic resonance spectroscopy. Among the major phenolic components (luteolin, hydroxytyrosol, luteolin-4'-O-β-D-glucopyranoside, luteolin-7-O-β-D-glucopyranoside, and oleuropein), luteolin and luteolin-4'-O-β-D-glucopyranoside were assigned as potent inhibitors of AGE formation. The extraction procedure greatly affects the composition and therefore the anti-glycation potential of olive leaves.