Feature extraction is the key to the fault detection of rotating machinery based on vibration signals, and the quality of the features influences the reliability of the detection. This paper develops a fault feature extraction method of rotating machinery based on optimized resonance-based sparse signal decomposition and refined composite multiscale fluctuation dispersion entropy. First, resonance-based sparse signal decomposition is used to decompose the vibration signals adaptively. In order to obtain the resonance-based sparse signal decomposition algorithm with optimal performance, the marine predator algorithm is used for the parameters optimization with correlation kurtosis as the fitness function. Subsequently, based on the refined composite coarse-grained process and fluctuation dispersion entropy, a refined composite multiscale fluctuation dispersion entropy is developed, enabling a more accurate and comprehensive measure of the complexity of time series. Then, all feature matrices are input to the support matrix machine for fault identification. Experiments are conducted using two typical rotating machinery datasets for the validity of the proposed method, and comparisons are made with other methods. The results show that the proposed scheme outperforms other comparative methods regarding classification accuracy and stability. In addition, the proposed scheme can obtain relatively reliable classification results even when the data volume is small and the background noise is significant, demonstrating the scheme's potential for application in practical engineering.