Translation-targeting toxic small alarmone synthetases (toxSAS) are effectors of bacterial toxin-antitoxin systems that pyrophosphorylate the 3'-CCA end of transfer RNA (tRNA) to prevent aminoacylation. toxSAS are implicated in antiphage immunity: Phage detection triggers the toxSAS activity to shut down viral production. We show that the toxSAS FaRel2 inspects the tRNA acceptor stem to specifically select tRNAGly and tRNAThr. The first, second, fourth, and fifth base pairs of the stem act as the specificity determinants. We show that the toxSASs PhRel2 and CapRelSJ46 differ in tRNA specificity from FaRel2 and rationalize this through structural modeling: While the universal 3'-CCA end slots into a highly conserved CCA recognition groove, the acceptor stem recognition region is variable across toxSAS diversity. As phages use tRNA isoacceptors to overcome tRNA-targeting defenses, we hypothesize that highly evolvable modular tRNA recognition allows for the escape of viral countermeasures through tRNA substrate specificity switching.