High-throughput screening of catalysts can be performed using density functional theory calculations to predict catalytic properties, often correlated with adsorbate binding energies. However, more complete investigations would require an order of 2 more calculations compared to the current approach, making the computational cost a bottleneck. Recently developed machine-learning methods have been demonstrated to predict these properties from hand-crafted features but have struggled to scale to large composition spaces or complex active sites. Here, we present an application of a deep-learning convolutional neural network of atomic surface structures using atomic and Voronoi polyhedra-based neighbor information. The model effectively learns the most important surface features to predict binding energies. Our method predicts CO and H binding energies after training with 12 000 data for each adsorbate with a mean absolute error of 0.15 eV for a diverse chemical space. Our method is also capable of creating saliency maps that determine atomic contributions to binding energies.