Identification of novel botanicals that can selectively induce apoptosis and arrest growth of cancer cells without producing cytotoxic effects is highly appreciable for cancer therapy. The present study aimed to investigate the possibility of acetylbritannilactone (ABL) derivative 5-(5-(ethylperoxy)pentan-2-yl)-6-methyl-3-methylene-2-oxo-2,3,3a,4,7,7a‑hexahydroben-zofuran-4-yl2-(6-methoxynaphthalen-2-yl) propanoate (ABL-N) as a therapeutic agent in human prostate cancer and potential mechanisms. Human prostate cancer cells were treated with ABL-N of different concentrations (0, 5, 10, 20, 30 and 40 µmol/l). Cell viability, migration and apoptosis were determined. Activities of caspases were assayed, as well as protein expression of cancer‑related proteins KLF5, Stat5b and ICAM-1 in PC3 cells. The therapeutic effect of ABL-N was further evaluated in our tumor xenografts. ABL-N inhibited growth of prostate cancer cells in a dose-dependent manner, without obvious effect on normal human prostate epithelial PrEC cells. ABL-N administration induced apoptosis of PC3 cells in a dose-dependent manner, along with the enhanced activity of caspases and increased Bax/Bcl-2 ratio. Expression of KLF5, Stat5b and ICAM-1 was significantly downregulated in PC3 cells. Our in vivo study further confirmed that ABL-N significantly inhibited the tumor growth of PC3 cells in the xenograft mouse model. ABL-N induces apoptosis of prostate cancer cells through activation of caspases, increasing the ratio of Bax/Bcl-2, as well as suppression of KLF5, Stat5b and ICAM-1 expressions. The present study indicated that ABL-N may be a potential therapeutic drug for human prostate cancer, and our data supported further studies to explore the therapeutic potential of ABL-N in other types of human cancer.