Pyrrolysine (Pyl, O) exists in nature as the 22nd proteinogenic amino acid. Despite being a fundamental building block of proteins, studies of Pyl have been hindered by the difficulty and inefficiency of both its chemical and biological syntheses. Here, we improve Pyl biosynthesis via rational engineering and directed evolution of the entire biosynthetic pathway. To accommodate toxicity of Pyl biosynthetic genes in Escherichia coli, we also develop Alternating Phage Assisted Non-Continuous Evolution (Alt-PANCE) that alternates mutagenic and selective phage growths. The evolved pathway provides 32-fold improved yield of Pyl-containing reporter protein compared to the rationally engineered ancestor. Evolved PylB mutants are present at up to 4.5-fold elevated levels inside cells, and show up to 2.2-fold increased protease resistance. This study demonstrates that Alt-PANCE provides a general approach for evolving proteins exhibiting toxic side effects, and further provides an improved pathway capable of producing substantially greater quantities of Pyl-proteins in E. coli.