Distant metastasis (DM) is the main cause of treatment failure in locally advanced rectal cancer. Adjuvant chemotherapy is usually used for distant control. However, not all patients can benefit from adjuvant chemotherapy, and particularly, some patients may even get worse outcomes after the treatment. We develop and validate an MRI-based radiomic signature (RS) for prediction of DM within a multicenter dataset. The RS is proved to be an independent prognostic factor as it not only demonstrates good accuracy for discriminating patients into high and low risk of DM in all the four cohorts, but also outperforms clinical models. Within the stratified analysis, good chemotherapy efficacy is observed for patients with pN2 disease and low RS, whereas poor chemotherapy efficacy is detected in patients with pT1-2 or pN0 disease and high RS. The RS may help individualized treatment planning to select patients who may benefit from adjuvant chemotherapy for distant control.