Backscatter has significant influence on detection efficiency for underwater lidar, especially for coaxial photon-counting lidar using a Geiger-mode avalanche photodiode. In this paper, based on our underwater coaxial photon-counting lidar structure and volume scatter function, a detection model with consideration of backscatter and refraction indices is proposed. Using this detection model, analysis of the detection efficiency is conducted. It reveals that in an underwater environment, higher pulse energy or a closer target range is not necessarily helpful for a higher target detection probability, which is vastly different from our traditional concepts. For example, the detection probability for a 5 m target would be 0.76 using a 200 pJ pulse and 0.55 using a 1000 pJ pulse for our coaxial photon-counting lidar. Monte Carlo simulation is conducted to verify our model and analysis, and some practical methods for improving the target detection probability in an underwater environment are proposed.