Leukocyte infiltrates into or around tumor cell nests are found in the context of protumorigenic inflammation and anticancer immunosurveillance. Hence, the detailed composition, density, architecture, and function of leukocyte infiltrates must be analyzed to understand their prognostic impact. The ectopic presence within tumors of high endothelial venule cells, which are normally characteristic for secondary lymphoid organs, correlates with a more pronounced infiltration by T lymphocytes and has a positive predictive impact on local advanced breast cancer treated with neoadjuvant chemotherapy. Recent progress in the field indicates that immune infiltrates of the primary tumors, as well as of metastases, are not only independent prognostic biomarkers but can also constitute predictive factors, suggesting that the pretherapeutic immune response can determine the efficacy of conventional chemotherapies. Moreover, accumulating evidence indicates that chemotherapy can stimulate anticancer immune responses coupled with an increased intratumoral lymphoid infiltration, which correlates with tumor mass reduction and patient survival. Improved methods for the automation of immunohistochemistry and digitalized image analyses will pave the way to an improved understanding of the complex interplay between cancer parenchyma, stroma, and immune effectors, as well as to the routine evaluation of immune-related parameters to the clinical management of cancer patients.