Arrhythmia is an abnormal heart rhythm, a common clinical problem in cardiology. Long-term or severe arrhythmia may lead to stroke and sudden cardiac death. The electrocardiogram (ECG) is the most commonly used tool to diagnose arrhythmia. However, the traditional diagnosis relies on experts for manual interpretation, which is time-consuming and laborious. In recent years, many automatic arrhythmia detection methods have emerged due to advancements in deep learning. These methods can reduce manual intervention and improve diagnostic efficiency. However, extracting useful features from raw ECG signals for arrhythmia detection is still challenging due to the low frequency of ECG signals and noise distribution. In this paper, we propose a novel hidden attention residual network (HA-ResNet) for automated arrhythmia classification. In this model, the one-dimensional ECG signals are first converted into two-dimensional images and fed into an embedding layer to obtain the relevant shallow features in ECG. Then, a hidden attention layer combining Squeeze-and-Excitation (SE) block and Bidirectional Convolutional LSTM (BConvLSTM) is used to further capture the deep Spatio-temporal features. We evaluate our HA-ResNet on two public datasets and achieve F1 scores of 96.0%, 96.7%, and 87.6% on 2s segments, 5s segments, and 10s segments, respectively, which significantly outperform the existing state-of-the-art approaches. The experimental results demonstrate the effectiveness and generalization of our method.