To reduce the residual distortion in high resolution diffusion MRI (dMRI) data preprocessed by the HCP-Pipeline, we propose an unsupervised deep learning based method to correct the residual susceptibility induced distortion. Instead of using B0 images from two phase encoding (PE), fiber orientation distribution (FOD) images computed from dMRI data, which provide more reliable contrast information, are used in our method. Our deep learning framework named DistoRtion Correction Net (DrC-Net) uses an U-Net to capture the latent features from FOD images and estimates a deformation field along the phase encoding direction. With the help of a transformer network, we can propagate the deformation feature to the FOD images and back propagate the losses between the deformed images and true undistorted images. The proposed DrC-Net is trained on 60 subjects randomly selected from 100 subjects in the Human Connectome Project (HCP) dataset. We evaluated the DrC-Net on the rest 40 subjects and the results show a similar performance compared to the training dataset. Our evaluation method used mean squared difference (MSD) of fractional anisotropy (FA) and minimum angular difference between two PEs. We compared the DrC-Net to topup method used in the HCP-Pipeline, and the results show a significant improvement to correct the susceptibility induced distortions in both evaluation methods.