Ribulose-1,5-bisphosphate carboxylase/oxygenase (Rubisco) catalyses the conversion of atmospheric carbon dioxide into organic compounds in photosynthetic organisms. Alongside carboxylating the five-carbon sugar ribulose-1,5-bisphosphate (RuBP)(1-3), Rubisco produces a small amount of xylulose-1,5-bisphosphate (XuBP), a potent inhibitor of Rubisco(4). The AAA+ protein Rubisco activase removes XuBP from the active site of Rubisco in an ATP-dependent process(5,6). However, free XuBP rapidly rebinds to Rubisco, perpetuating its inhibitory effect. Here, we combine biochemical and structural analyses to show that the CbbY protein of the photosynthetic bacterium Rhodobacter sphaeroides and Arabidopsis thaliana is a highly selective XuBP phosphatase. We also show that CbbY converts XuBP to the non-inhibitory compound xylulose-5-phosphate, which is recycled back to RuBP. We solve the crystal structures of CbbY from R. sphaeroides and A. thaliana, and through mutational analysis show that the cap domain of the protein confers the selectivity for XuBP over RuBP. Finally, in vitro experiments with CbbY from R. sphaeroides reveal that CbbY cooperates with Rubisco activase to prevent a detrimental build-up of XuBP at the Rubisco active site. We suggest that CbbY, which is conserved in algae and plants, is an important component of the cellular machinery that has evolved to deal with the shortcomings of the ancient enzyme Rubisco.