We present a novel head-mounted display setup that uses the pinhole imaging principle coupled with a low-latency dynamic pupil follower. A transmissive LCD is illuminated by a single LED backlight. LED illumination is focused onto the viewer's pupil to form an eyebox smaller than the average human pupil, thereby creating a pinhole display effect where objects at all distances appear in focus. Since nearly all the light is directed to the viewer's pupil, a single low-power LED for each primary color with 0.42 lumens total output is sufficient to create a bright and full-color display of 360 cd/m2 luminance. In order to follow the viewer's pupil, the eyebox needs to be steerable. We achieved a dynamic eyebox using an array of LEDs that is coupled with a real-time pupil tracker. The entire system is operated at 11 msec motion-to-photon latency, which meets the demanding requirements of the real-time pupil follower system. Experimental results effectively demonstrated our head-mounted pinhole display with 37° FOV and very high light efficiency, equipped with a pupil follower with low motion-to-photon latency.