During the last two decades, considerable efforts have been made to improve the prevention, early diagnosis and therapy of colorectal cancer by gaining enhanced insights into disease-specific pathogenesis. Along these lines, tumor-infiltrating immune cells turned out to be critical indicators for an efficient antitumor immune response and the number and type of tumor-infiltrating immune cells determined the resulting tumor prognosis. This review aims to describe the prognostic relevance of the different subsets of tumor-infiltrating immune cells and highlights their specific function in the complex process of immune system-mediated rejection of colorectal cancer cells. Considering the clinical impact of this increased insight into tumor-associated immunology, resulting innovative approaches in the treatment of colorectal cancer are summarized and discussed immunotherapeutic critically.