Anti-PD-1/PD-L1 therapy, either by anti-PD-1 antibody or anti-PD-L1 antibody, has efficacy by reinvigorating tumor-infiltrating CD8+ T cells in a subset of patients with cancer, but it has unequal effects on heterogeneous CD8+ T cell populations. Hence, the subset crucial to efficacious PD-1 blockade therapy remains elusive. Here, we found an increase in tumor-infiltrating CD200+ cytotoxic T lymphocytes (CTLs) upon PD-1/PD-L1 blockade, with higher proportions of CD200+ T cells positively related to a favorable clinical outcome to anti-PD-1/PD-L1 therapy in three independent cohorts of patients with cancer. Using multiple mouse tumor models, we demonstrated that CD200+ CTLs are essential for efficacious anti-PD-L1 therapy. Mechanistically, we observed a unique chromatin landscape in CD200+ CTLs and found that these cells are enriched for tumor antigen-specific CTLs and have antitumor effector functions. Coinoculation of CD200+ CTLs with tumor cells led to robust tumor regression in two transplanted mouse models. Clinically, we found that infiltration of CD200+ CTLs into tumors could predict immunotherapy efficacy in six patient cohorts. Together, our findings reveal that CD200+ CTLs in the tumor microenvironment are crucial for efficacious anti-PD-1/PD-L1 therapy and could serve as a predictor of successful immunotherapy in the clinic.