Characterizing the tumor immune microenvironment enables the identification of new prognostic and predictive biomarkers, the development of novel therapeutic targets and strategies, and the possibility to guide first-line treatment algorithms. Although the driving elements within the tumor microenvironment of individual primary organ sites differ, many of the salient features remain the same. The presence of a robust antitumor milieu characterized by an abundance of CD8+ cytotoxic T-cells, Th1 helper cells, and associated cytokines often indicates a degree of tumor containment by the immune system and can even lead to tumor elimination. Some of these features have been combined into an 'Immunoscore', which has been shown to complement the prognostic ability of the current TNM staging for early stage colorectal carcinomas. Features of the immune microenvironment are also potential therapeutic targets, and immune checkpoint inhibitors targeting the PD-1/PD-L1 axis are especially promising. FDA-approved indications for anti-PD-1/PD-L1 are rapidly expanding across numerous tumor types and, in certain cases, are accompanied by companion or complimentary PD-L1 immunohistochemical diagnostics. Pathologists have direct visual access to tumor tissue and in-depth knowledge of the histological variations between and within tumor types and thus are poised to drive forward our understanding of the tumor microenvironment. This review summarizes the key components of the tumor microenvironment, presents an overview of and the challenges with PD-L1 antibodies and assays, and addresses newer candidate biomarkers, such as CD8+ cell density and mutational load. Characteristics of the local immune contexture and current pathology-related practices for specific tumor types are also addressed. In the future, characterization of the host antitumor immune response using multiplexed and multimodality biomarkers may help predict which patients will respond to immune-based therapies.