In situ visualization of laparoscopic ultrasound in both conventional and robot-assisted laparoscopic surgery requires robust and efficient computation of the pose of the laparoscopic ultrasound probe with respect to the laparoscopic camera. Image-based intrinsic methods of computing this relative pose need to overcome challenges due to irregular illumination, partial feature occlusion, and clutter that are unavoidable in practical laparoscopic surgery. In this paper, we propose an accurate image-based method that is robust to partial occlusion of the fiducials and outliers. The method is extended to multi-view imaging model with applications in stereoscopic laparoscopy and robot-assisted surgery. Rather than treating the model-to-image correspondence and pose computation as separate problems, we solve them jointly using the Kalman Filter-based framework that demonstrates video rate running time (~24fps). By keeping the optical tracking measurements as a reference, we demonstrate that the proposed methods result in clinically acceptable tracking accuracy, reaching target registration errors well below 1.5mm on average. In addition, our multi-view tracking method is compared to a conventional stereo triangulation-based pose estimation scheme that commercial optical tracking systems are based on, to experimentally demonstrate its superiority in terms of accuracy. Finally, we qualitatively demonstrate the suitability of our methods for practical laparoscopic applications by conducting a phantom-based experiment.