Prostate cancer (PCa) is one of the most common types of cancer in the urinary system in men. Zeylenone (Zey), a naturally occurring cyclohexene oxide, has an anticancer effect. In the present study, the role and potential mechanism of Zey in PCa were examined. The proliferative, invasive and migratory capacities of DU145 cells were analyzed using Cell Counting Kit‑8, transwell and wound healing assays, respectively. The expression levels of matrix metalloproteinase (MMP)‑2 and MMP‑9 were determined with an ELISA. Reverse transcription‑quantitative polymerase chain reaction and western blotting assays were performed to evaluate the expression levels of extracellular matrix, epithelial‑mesenchymal transition and Wnt/β‑catenin pathway‑associated factors. In the present study, it was observed that Zey not only suppressed the viability of DU145 cells; however, it additionally attenuated the invasive and migratory capacities of cells in a concentration‑dependent manner. Treatment of Zey decreased the expression levels of MMP‑2, MMP‑9 and fibronectin‑1; whereas, it increased tissue inhibitor of metalloproteinases‑1 and collagen‑1 expression levels. Additionally, the vimentin expression level was downregulated, however, the epithelial‑cadherin expression level was upregulated in cells treated with Zey. Furthermore, Zey decreased the expression levels of wnt5a, β‑catenin and cyclin D1. In conclusion, the present results demonstrated that Zey decreased the viability and metastasis of human PCa cells (DU145), via the Wnt/β‑catenin signaling pathway. Therefore, Zey may be applied as a novel drug for treating PCa in the future.