The arterial and venous poles of the mammalian heart are hotspots of congenital heart defects (CHD) such as those observed in 22q11.2 deletion (or DiGeorge) and Holt-Oram syndromes. These regions of the heart are derived from late differentiating cardiac progenitor cells of the Second Heart Field (SHF) located in pharyngeal mesoderm contiguous with the elongating heart tube. The T-box transcription factor Tbx1, encoded by the major 22q11.2 deletion syndrome gene, regulates SHF addition to both cardiac poles from a common progenitor population. Despite the significance of this cellular addition the mechanisms regulating the deployment of common progenitor cells to alternate cardiac poles remain poorly understood. Here we demonstrate that Tbx5, mutated in Holt-Oram syndrome and essential for venous pole development, is activated in Tbx1 expressing cells in the posterior region of the SHF at early stages of heart tube elongation. A subset of the SHF transcriptional program, including Tbx1 expression, is subsequently downregulated in Tbx5 expressing cells, generating a transcriptional boundary between Tbx1-positive arterial pole and Tbx5-positive venous pole progenitor cell populations. We show that normal downregulation of the definitive arterial pole progenitor cell program in the posterior SHF is dependent on both Tbx1 and Tbx5. Furthermore, retinoic acid (RA) signaling is required for Tbx5 activation in Tbx1-positive cells and blocking RA signaling at the time of Tbx5 activation results in atrioventricular septal defects at fetal stages. Our results reveal sequential steps of cardiac progenitor cell patterning and provide mechanistic insights into the origin of common forms of CHD.