Fumitremorgin B endoperoxidase (FtmOx1) from Aspergillus fumigatus is the first reported α-ketoglutarate-dependent mononuclear non-haem iron enzyme that catalyzes the endoperoxide formation reaction, converting Fumitremorgin B to verruculogen. Experiments reveal that the molecular oxygen (O2) is incorporated into verruculogen without O-O bond scission, which differs from the currently known non-haem iron enzymes, but the mechanistic details are still unclear. In this paper, on the basis of the crystal structures of FtmOx1 in complex with either the co-substrate (α-ketoglutarate) or the substrate (fumitremorgin B), a ternary complex model of the enzyme-α-ketoglutarate-substrate has been constructed, and combined quantum mechanics and molecular mechanics (QM/MM) calculations have been performed to unravel the novel mechanism of FtmOx1. Our calculations indicate the quintet of the FeIV[double bond, length as m-dash]O complex as the ground state. The FeIV[double bond, length as m-dash]O complex firstly abstracts a hydrogen from the hydroxyl of Tyr228 to initiate the reaction, which corresponds to a lower energy barrier (9.1 kcal mol-1). If the FeIV[double bond, length as m-dash]O complex directly abstracts a hydrogen from C21 of the substrate, the energy barrier would increase to 33.9 kcal mol-1. When Tyr228 was mutated to Ala228, this energy barrier decreases to 24.0 kcal mol-1. In the subsequent reaction, the generated Tyr228 radical abstracts the hydrogen (H2) from C21 of the substrate with an energy barrier of 23.8 kcal mol-1. The second molecular oxygen binds to the C21 radical of the substrate in the active pocket and further completes the epoxidation with an energy barrier of 4.8 kcal mol-1. These results may provide useful information for understanding the reaction mechanism of FtmOx1 and provide guidance for further experimental investigations.