Historic Japanese culture evolved from at least two distinct migrations that originated on the Asian continent. Hunter-gatherers arrived before land bridges were submerged after the last glacial maximum (>12,000 years ago) and gave rise to the Jomon culture, and the Yayoi migration brought wet rice agriculture from Korea beginning approximately 2,300 years ago. A set of 81 Y chromosome single nucleotide polymorphisms (SNPs) was used to trace the origins of Paleolithic and Neolithic components of the Japanese paternal gene pool, and to determine the relative contribution of Jomon and Yayoi Y chromosome lineages to modern Japanese. Our global sample consisted of >2,500 males from 39 Asian populations, including six populations sampled from across the Japanese archipelago. Japanese populations were characterized by the presence of two major (D and O) and two minor (C and N) clades of Y chromosomes, each with several sub-lineages. Haplogroup D chromosomes were present at 34.7% and were distributed in a U-shaped pattern with the highest frequency in the northern Ainu and southern Ryukyuans. In contrast, haplogroup O lineages (51.8%) were distributed in an inverted U-shaped pattern with a maximum frequency on Kyushu. Coalescent analyses of Y chromosome short tandem repeat diversity indicated that haplogroups D and C began their expansions in Japan approximately 20,000 and approximately 12,000 years ago, respectively, while haplogroup O-47z began its expansion only approximately 4,000 years ago. We infer that these patterns result from separate and distinct genetic contributions from both the Jomon and the Yayoi cultures to modern Japanese, with varying levels of admixture between these two populations across the archipelago. The results also support the hypothesis of a Central Asian origin of Jomonese ancestors, and a Southeast Asian origin of the ancestors of the Yayoi, contra previous models based on morphological and genetic evidence.