Sleep restriction (SR) is defined as the condition of not having enough sleep, and it can cause brain injury. In this study, we examined the impact of SR on the structural brain network. We obtained diffusion MRI (dMRI) data for the SR group of fourteen participants who got less than or equal to 5.5 h of sleep for the last 1 month and normal group of the same number of participants who got 7 h of sleep. We constructed the structural brain networks from the dMRI data and analyzed them using graph theoretical approaches. In comparison with the normal group, the SR group showed higher vulnerability to the targeted node attack and alterations of regional efficiency in the brain regions such as the bilateral orbital part of the frontal gyri, superior occipital gyri, left insula, fusiform, right supplementary motor area, and cingulate gyrus. These findings indicate that SR may cause the reduction of the potential alternative neuronal pathways in the brain and rewiring of neuronal fibers in the structural brain networks, which may result in potential functional impairments, as well as alterations of the structural brain connectivity. Therefore, investigating the structural brain network offers new insight into how SR influences the human brain.