The review recounts the history of how the study of the DNA tumor viruses including polyoma, SV40 and Adenovirus brought key insights into the structure and function of the Retinoblastoma protein (Rb). Knudsen's model of the two-hit hypothesis to explain patterns of hereditary and sporadic retinoblastoma provided the foundation for the tumor suppressor hypothesis that ultimately led to the cloning of the Rb gene. The discovery that SV40 and Adenovirus could cause tumors when inoculated into animals was startling not only because SV40 had contaminated the poliovirus vaccine and Adenovirus was a common cause of viral induced pneumonia but also because they provided an opportunity to study the genetics and biochemistry of cancer. Studies of mutant forms of these viruses led to the identification of the E1A and Large T antigen (LT) oncogenes and their small transforming elements including the Adenovirus Conserved Regions (CR), the SV40 J domain and the LxCxE motif. The immunoprecipitation studies that initially revealed the size and ultimately the identity of cellular proteins that could bind to these transforming elements were enabled by the widespread development of highly specific monoclonal antibodies against E1A and LT. The identification of Rb as an E1A and LT interacting protein quickly led to the cloning of p107, p130, p300, CBP, p400 and TRRAP and the concept that viral transformation was due, at least in part, to the perturbation of the function of normal cellular proteins. In addition, studies on the ability of E1A to transactivate the Adenovirus E2 promoter led to the cloning of the heterodimeric E2F and DP transcription factor and recognition that Rb repressed transcription of cellular genes required for cell cycle entry and progression. More recent studies have revealed how E1A and LT combine the activity of Rb and the other cellular associated proteins to perturb expression of many genes during viral infection and tumor formation.