Plasmon photochemistry can potentially play a significant role in photocatalysis. To realize this potential, it is critical to enhance the plasmon excited hot carrier transfer and collection. However, the lack of atomistic understanding of the carrier transfer across the interface, especially when the carrier is still "hot", makes it challenging to design a more efficient system. In this work, we apply the nonadiabatic molecular dynamics simulation to study hot carrier dynamics in the system of a Au nanocluster on top of a GaN surface. By setting up the initial excited hole in Au, the carrier transfer from Au to GaN is found to be on a subpicosecond time scale. The hot hole first cools to the band edge of Au d-states while it transfers to GaN. After the hole has cooled down to the band edge of GaN, we find that some of the charges can return back to Au. By applying different external potentials to mimic the Schottky barrier band bending, the returning charge can be reduced, demonstrating the importance of the internal electric field. Finally, with the understanding of the carrier transfer's pathway, we suggest that a ZnO layer between GaN and Au can effectively block the "cold" carrier from returning back to Au but still allow the hot carrier to transfer from Au to GaN.