Epilepsy is one of the most common neurological diseases. Clinically, epileptic seizure detection is usually performed by analyzing electroencephalography (EEG) signals. At present, deep learning models have been widely used for single-channel EEG signal epilepsy detection, but this method is difficult to explain the classification results. Researchers have attempted to solve interpretive problems by combining graph representation of EEG signals with graph neural network models. Recently, the combination of graph representations and graph neural network (GNN) models has been increasingly applied to single-channel epilepsy detection. By this methodology, the raw EEG signal is transformed to its graph representation, and a GNN model is used to learn latent features and classify whether the data indicates an epileptic seizure episode. However, existing methods are faced with two major challenges. First, existing graph representations tend to have high time complexity as they generally require each vertex to traverse all other vertices to construct a graph structure. Some of them also have high space complexity for being dense. Second, while separate graph representations can be derived from a single-channel EEG signal in both time and frequency domains, existing GNN models for epilepsy detection can learn from a single graph representation, which makes it hard to let the information from the two domains complement each other. For addressing these challenges, we propose a Weighted Neighbour Graph (WNG) representation for EEG signals. Reducing the redundant edges of the existing graph, WNG can be both time and space-efficient, and as informative as its less efficient counterparts. We then propose a two-stream graph-based framework to simultaneously learn features from WNG in both time and frequency domain. Extensive experiments demonstrate the effectiveness and efficiency of the proposed methods.