Sleep apnea (SA) is a common sleep-related breathing disorder that tends to induce a series of complications, such as pediatric intracranial hypertension, psoriasis, and even sudden death. Therefore, early diagnosis and treatment can effectively prevent malignant complications SA incurs. Portable monitoring (PM) is a widely used tool for people to monitor their sleep conditions outside of hospitals. In this study, we focus on SA detection based on single-lead electrocardiogram (ECG) signals which are easily collected by PM. We propose a bottleneck attention based fusion network named BAFNet, which mainly includes five parts of RRI (R-R intervals) stream network, RPA (R-peak amplitudes) stream network, global query generation, feature fusion, and classifier. To learn the feature representation of RRI/RPA segments, fully convolutional networks (FCN) with cross-learning are proposed. Meanwhile, to control the information flow between RRI and RPA networks, a global query generation with bottleneck attention is proposed. To further improve the SA detection performance, a hard sample scheme with k-means clustering is employed. Experiment results show that BAFNet can achieve competitive results, which are superior to the state-of-the-art SA detection methods. It means that BAFNet has great potential to be applied in the home sleep apnea test (HSAT) for sleep condition monitoring.