Methods that allow online lung tumor tracking during radiotherapy are desirable for a variety of applications that have the potential to vastly improve treatment accuracy, dose conformity and sparing of healthy tissue. Several publications have proposed the use of an on-board kV x-ray imager to assess the tumor location during treatment. However, there is some concern that this strategy may expose the patient to a significant amount of additional dose over the course of a typical radiotherapy treatment. In this paper we present an algorithm that utilizes the on-board portal imager of the treatment machine to track lung tumors. This does not expose the patient to additional dose, but is somewhat more challenging as the quality of portal images is inferior when compared to kV x-ray images. To quantify the performance of the proposed algorithm we retrospectively applied it to portal image sequences retrieved from a dynamic chest phantom study and an SBRT treatment performed at our institution. The results were compared to manual tracking by an expert. For the phantom data the tracking error was found to be smaller than 1 mm and for the patient data smaller than 2 mm, which was in the same range as the uncertainty of the gold standard.