We present the Mueller matrix imaging system to classify morphologically similar algae based on convolutional neural networks (CNNs). The algae and cyanobacteria data set contains 10,463 Mueller matrices from eight species of algae and one species of cyanobacteria, belonging to four phyla, the shapes of which are mostly randomly oriented spheres, ovals, wheels, or rods. The CNN serves as an automatic machine with learning ability to help in extracting features from the Mueller matrix, and trains a classifier to achieve a 97% classification accuracy. We compare the performance in two ways. One way is to compare the performance of five CNNs that differ in the number of convolution layers as well as the classical principle component analysis (PCA) plus the support vector machine (SVM) method; the other way is to quantify the differences of scores between full Mueller matrix and the first matrix element m11, which does not contain polarization information under the same conditions. As the results show, deeper CNNs perform better, the best of which outperforms the conventional PCA plus SVM method by 19.66% in accuracy, and using the full Mueller matrix earns 6.56% increase of accuracy than using m11. It demonstrates that the coupling of Mueller matrix imaging and CNN may be a promising and efficient solution for the automatic classification of morphologically similar algae.