The olive fruit fly (OFF), Bactrocera oleae is the most devastating pest affecting olive fruit worldwide. Previous investigations have addressed the fungal microbiome associated with olive drupes or B. oleae, but the impact of the insect on fungal communities of olive fruit remains undescribed. In the present work, the fungal microbiome of olive drupes, infested and non-infested by the OFF, was investigated in four different localities and cultivars. Olive fruit fly infestations caused a general reduction of the fungal diversity, a higher quantity of the total DNA and an increase in taxa that remained unidentified or had unknown roles. The infestations led to imbalanced fungal communities with the growth of taxa that are usually outcompeted. While it was difficult to establish a cause-effect link between fly infestation and specific fungi, it is clear that the fly alters the natural microbial balance, especially the low abundant taxa. On the other hand, the most abundant ones, were not significantly influenced by the insect. In fact, despite the slight variation between the sampling locations, Aureobasidium, Cladosporium, and Alternaria, were the dominant genera, suggesting the existence of a typical olive fungal microbiome.