This study aimed to develop a machine learning-based algorithm for glaucoma diagnosis in patients with open-angle glaucoma, based on three-dimensional optical coherence tomography (OCT) data and color fundus images. In this study, 208 glaucomatous and 149 healthy eyes were enrolled, and color fundus images and volumetric OCT data from the optic disc and macular area of these eyes were captured with a spectral-domain OCT (3D OCT-2000, Topcon). Thickness and deviation maps were created with a segmentation algorithm. Transfer learning of convolutional neural network (CNN) was used with the following types of input images: (1) fundus image of optic disc in grayscale format, (2) disc retinal nerve fiber layer (RNFL) thickness map, (3) macular ganglion cell complex (GCC) thickness map, (4) disc RNFL deviation map, and (5) macular GCC deviation map. Data augmentation and dropout were performed to train the CNN. For combining the results from each CNN model, a random forest (RF) was trained to classify the disc fundus images of healthy and glaucomatous eyes using feature vector representation of each input image, removing the second fully connected layer. The area under receiver operating characteristic curve (AUC) of a 10-fold cross validation (CV) was used to evaluate the models. The 10-fold CV AUCs of the CNNs were 0.940 for color fundus images, 0.942 for RNFL thickness maps, 0.944 for macular GCC thickness maps, 0.949 for disc RNFL deviation maps, and 0.952 for macular GCC deviation maps. The RF combining the five separate CNN models improved the 10-fold CV AUC to 0.963. Therefore, the machine learning system described here can accurately differentiate between healthy and glaucomatous subjects based on their extracted images from OCT data and color fundus images. This system should help to improve the diagnostic accuracy in glaucoma.