One of the major challenges in the development of virtual environments for medical simulations is photorealistic rendering, permitting high fidelity visual effects and user interaction. Digitized videos recorded from the laparoscopic camera are a rich source of information about surgical scenarios. How to fully utilize the information is important for improving the realism of the simulated scenarios. In reality, the camera viewpoint changes frequently and even for the same viewpoint, the scene is dynamic due to rhythmic heartbeat. Hence, the results of classical texture mapping are usually visually unappealing as they fail to capture the pulsatile effect, as well as other global illumination properties of the scene. In this paper we present a hybrid technique to improve the photorealistic rendering of the virtual surgery scenarios by spatio-temporally utilizing videos recorded during actual surgical procedures.