To investigate the inhibitory effect of acetoacetate extract from Celastrus orbiculatus Thumb (COT) on the growth of red fluorescent protein (RFP)-xenografted human hepatocellular carcinoma (HCC) in a nude mouse model. Human HCC HepG2 cells were transduced with RFP and inoculated into the liver of BALB/c nude mice. The tumor-bearing mice were randomly divided into five groups: control group (G1), oxaliplatin positive control group (G2; 25 mg/kg), COT low-dose group (G3; 20 mg/kg), COT high-dose group (G4; 40 mg/kg), and COT early treatment group (G5; 20 mg/kg). The early treatment group received oral COT from day 2 post-tumor implantation. All other mice were treated from day 20 post-tumor implantation. Growth of xenografted tumors was monitored weekly by in vivo real-time fluorescence imaging technology. At the end of the four-week treatment period, all mice were sacrificed and tumor tissues were collected and weighed. The two-sided t-test was used to evaluate intergroup differences in tumor volumes, final tumor weights, and final body weights. Mice treated with COT had significantly smaller xenografted tumors. On day 45 post-implantation, the mean tumor volumes (mm3) in the different groups were: G1, 803.1+/-512.3 ; G2, 83.8+/-23.5; G3, 852.7+/-502.6; G4, 410.0+/-231.6; and G5, 120.5+/-60.1. The mean tumor weights (g) were: G1, 0.95+/-0.49; G2, 0.36+/-0.09; G3, 0.67+/-0.29; G4, 0.48+/-0.15; and G5, 0.38+/-0.11. The differences in tumor weights from G2, G4 and G5 were significantly less than the weight in G1 (P less than 0.05); however, there was no significant differences between the tumor weights in G2, G4 and G5 (P more than 0.05). The tumor weight from the G2 group was significantly less than that of the G3 group (P less than 0.05). COT significantly inhibited the proliferation of human HCC in a nude mouse model. Early treatment with COT produced a more robust inhibitory effect, which was very similar to that achieved with oxaliplatin treatment.