This paper proposes a segmentation-free radiomics method to classify malignant and benign breast tumors with shear-wave elastography (SWE) data. The method is targeted to integrate the advantage of both SWE in providing important elastic with morphology information and convolutional neural network (CNN) in automatic feature extraction and accurate classification. Compared to traditional methods, the proposed method is designed to directly extract features from the dataset without the prerequisite of segmentation and manual operation. This can keep the peri-tumor information, which is lost by segmentation-based methods. With the proposed model trained on 540 images (318 of malignant breast tumors and 222 of benign breast tumors, respectively), an accuracy of 95.8%, a sensitivity of 96.2%, and a specificity of 95.7% was obtained for the final test. The superior performances compared to the existing state-of-the-art methods and its automatic nature both demonstrate that the proposed method has a great potential to be applied to clinical computer-aided diagnosis of breast cancer.