The role of the immune system in the course of colorectal cancer has been elucidated in the last decade. While quantification of immune cell infiltrates within the resected specimen at diagnosis has a clear power to estimate the prognosis of the patient, the role of infiltrating immune cells within the metastatic situation and especially within the metastatic lesion itself requires further detailed analyses. Recent analyses of infiltrates in colorectal cancer liver metastases revealed a role for the infiltrate density not only for prognosis but also in the prediction of treatment response. This not only broadens the view on these infiltrates and indicates a systematic role of the local immunological microenvironment, but also raises the question how these infiltrates change during repeated courses of treatment (i.e., resection, chemotherapy, etc.). To address this question, sequential lung or sequential liver metastases of colorectal cancer patients were analyzed using whole slide image quantification after immunohistochemical staining against CD3, CD8, FOXP3, CD68 and Granzyme B. The clinical data and interventions were associated with each individual patient and the metastatic lesions. The resulting cell densities reveal a heterogeneous profile: after successful treatment of a metastatic lesion, the recurrent lesion can still have the same immunophenotype with similar cell distributions. In a situation of a favorable immune cell profile, this profile can return and apparently convey a similar favorable course throughout the disease. But also the opposite was found: the recurrent metastatic lesion could have a different profile with alterations in specific immune cell subsets over time. Further analyses are required to elucidate the different patterns and their associations to the treatment, the tumor cell phenotype and other dynamic factors. However, it is clear from this data however, that there is an immune cell plasticity that needs to be analyzed for individual patients.