The three-way interaction between oncolytic viruses, the tumor microenvironment, and the immune system is critical to the outcome of antitumor therapy. Classically, the immune system is thought to limit the efficacy of therapy, leading to viral clearance. However, preclinical and clinical data suggest that in some cases virotherapy may in fact act as cancer immunotherapy. In this review we discuss the ability of oncolytic viruses to alter the immunogenic milieu of the tumor microenvironment, and the role of innate and adaptive immunity in both restricting and augmenting therapy. Strategies to improve virotherapy by immunomodulation, including suppression or enhancement of the innate and adaptive responses, are discussed.