In the field of optical fiber vibration signal recognition, one-dimensional signals have few features. People often used the shallow layer of a one-dimensional convolutional neural network (1D-CNN), which results in fewer features being learned by the network, leading to a poor recognition rate. There are also many complex algorithms and data processing methods, which make the whole signal recognition process more complicated. Therefore, an optical vibration signal recognition method based on an efficient multidimensional feature extraction network was proposed. Based on ResNet-50, efficient channel attention (ECA) was used to improve image features extraction ability, and a long short-term memory (LSTM) network was used to enhance the extraction of temporal features. Three different vibration signals were collected using a phase-sensitive optical time-domain reflectometry (Φ-OTDR) optical fiber sensing system. Vibration signals were converted into 128×128 grayscale images, which have more effective vibration information. The experimental results show that the three types of signals can be recognized and classified effectively by the network, and the average recognition rate is 98.67%.