Elongation factors NusG and RfaH evolved from a common ancestor and utilize the same binding site on RNA polymerase (RNAP) to modulate transcription. However, although NusG associates with RNAP transcribing most Escherichia coli genes, RfaH regulates just a few operons containing ops, a DNA sequence that mediates RfaH recruitment. Here, we describe the mechanism by which this specificity is maintained. We observe that RfaH action is indeed restricted to those several operons that are devoid of NusG in vivo. We also show that RfaH and NusG compete for their effects on transcript elongation and termination in vitro. Our data argue that RfaH recognizes its DNA target even in the presence of NusG. Once recruited, RfaH remains stably associated with RNAP, thereby precluding NusG binding. We envision a pathway by which a specialized regulator has evolved in the background of its ubiquitous paralogue. We propose that RfaH and NusG may have opposite regulatory functions: although NusG appears to function in concert with Rho, RfaH inhibits Rho action and activates the expression of poorly translated, frequently foreign genes.