Identifying the binding residues of protein-peptide complexes is essential for understanding protein function mechanisms and exploring drug discovery. Recently, many computational methods have been developed to predict the interaction sites of either protein or peptide. However, to our knowledge, no prediction method can simultaneously identify the interaction sites on both the protein and peptide sides. Here, we propose a deep graph convolutional network (GCN)-based method called GraphPPepIS to predict the interaction sites of protein-peptide complexes using protein and peptide structural information. We also propose a companion method, SeqPPepIS, for assisting with the lack of structural information and the flexibility of peptides. SepPPepIS replaces the peptide structural features in GraphPPepIS by learning features from peptide sequences. We performed a comprehensive evaluation of the benchmark data sets, and the results show that our two methods outperform state-of-the-art methods on the accurate interaction sites of both protein and peptide sides. We show that our methods can help improve protein-peptide docking. For docking data sets, our methods maintain robust performance in identifying binding sites, thereby enhancing the prediction of peptide binding poses. Finally, we visualized the analysis of protein and peptide graph embedding to demonstrate the learning ability of graph convolution in predicting interaction sites, which was mainly obtained through the shared parameters of a protein graph and peptide graph.