Perineural invasion (PNI) as a grossly underreported independent risk predictor in rectal cancer is hard to identify preoperatively. We aim to predict PNI status in rectal cancer using multi-modality radiomics. In total, 396 radiomics features were extracted from T2-weighted images (T2WIs), diffusion-weighted images (DWIs), and portal venous phase of contrast-enhanced CT (CE-CT) respectively of 94 consecutive patients with histologically confirmed rectal cancer. T2WI score, DWI score, and CT score were calculated via the radiomics features selection and optimization. Discrimination, calibration, and clinical benefit ability were used to evaluate the performance of the radiomics scores in both training and testing datasets. CT score and T2WI score were independent risk predictors [CT score, OR (95% CI) = 4.218 (1.070-16.620); T2WI score, OR (95% CI) = 105.721 (3.091-3615.790)]. The concise score which combined CT score and T2WI score, showed the best performance [training dataset, AUC (95% CI) = 0.906 (0.833-0.979); testing dataset, AUC (95% CI) = 0.884 (0.761-1.000)] and good calibration (P > 0.05 in the Hosmer-Lemeshow test for the training and testing datasets). Decision curve analysis showed that the multi-modality radiomics nomogram had a higher clinical net benefit. The multi-modality radiomics score could be used to preoperatively assess PNI status in rectal cancer.