Wave optics theory and 3-D deconvolution for the light field microscope

Abstract

Light field microscopy is a new technique for high-speed volumetric imaging of weakly scattering or fluorescent specimens. It employs an array of microlenses to trade off spatial resolution against angular resolution, thereby allowing a 4-D light field to be captured using a single photographic exposure without the need for scanning. The recorded light field can then be used to computationally reconstruct a full volume. In this paper, we present an optical model for light field microscopy based on wave optics, instead of previously reported ray optics models. We also present a 3-D deconvolution method for light field microscopy that is able to reconstruct volumes at higher spatial resolution, and with better optical sectioning, than previously reported. To accomplish this, we take advantage of the dense spatio-angular sampling provided by a microlens array at axial positions away from the native object plane. This dense sampling permits us to decode aliasing present in the light field to reconstruct high-frequency information. We formulate our method as an inverse problem for reconstructing the 3-D volume, which we solve using a GPU-accelerated iterative algorithm. Theoretical limits on the depth-dependent lateral resolution of the reconstructed volumes are derived. We show that these limits are in good agreement with experimental results on a standard USAF 1951 resolution target. Finally, we present 3-D reconstructions of pollen grains that demonstrate the improvements in fidelity made possible by our method.

Fig. 1

USAF 1951 resolution test target translated to depths below the native object plane (z = 0 μm) and imaged using a light field microscope with a 20× 0.5NA water-dipping objective. (a) Photographs taken with a conventional microscope as the target is translated to the z-heights denoted below each image. (b) Computational re-focusing using our 2009 method [2] while the microscope was defocused to the same heights as (a). While some computational refocusing is possible, there has been a significant loss of lateral resolution. (c) The reconstruction algorithm presented in this paper brings the target back into focus, achieving up to an 8-fold improvement in lateral resolution compared to (b) except at the native object plane (left image).

Fig. 2

Optical model of the light field microscope. (a) A fluorescence microscope can be converted into a light field microscope by placing a microlens array at the native image plane. (b) A ray-optics schematic indicating the pattern of illumination generated by one point source. The gray grid delineates pixel locations, and the white circles depict the back-aperture of the objective imaged onto the sensor by each lenslet. Red level indicates the intensity of illumination arriving at the sensor. For a point source on the native object plane (red dot), all rays pass through a single lenslet. (c) A point source below the native object plane generates a more complicated intensity pattern involving many lenslets. (d) A real light field recorded on our microscope using a 60× 1.4NA oil objective and a 125 μm pitch f/20 microlens array of a 0.5 μm fluorescent bead placed 4 μm below the native object plane. Diffraction effects are present in the images formed behind each lenslet. (e) A schematic of our wave optics model based on the LFM optical path (not drawn to scale). In this model, the microlens array at the native image plane is modeled as a tiled phase mask operating on this wavefront, which is then propagated to the camera sensor. The xy cross section on the far right shows the simulated light field generated at the sensor plane. The simulation is in good agreement with the experimentally measured light field in (d).

Fig. 3

Sampling of the volume recorded in a microlens-based light field. (a) A bundle of lenslet chief rays captured by the same pixel position relative to each lenslet (blue pixels) form a parallel projection through the volume, providing one of many angular views necessary to perform 3-D deconvolution. (b) When lenslet chief rays passing through every pixel in the light field are simultaneously projected back into the object volume, these rays cross at a diversity of x-positions (readers are encouraged to zoom into this figure in a PDF file to see how this pattern evolves with depth). This dense sampling pattern permits 3-D deconvolution with resolution finer than the lenslet spacing. The only place where this diversity does not occur is close to the native object plane; here resolution enhancement is not possible. (c) The distribution of the lenslet chief rays in xy cross-sections of the object volume changes at different distances from the native object plane. The outline of the lenslets are shown in light gray for scale. At z = 0 μm (rightmost image), the lack of spatial diversity in sample locations is evident.

Fig. 4

The discrete light field imaging model, prior to accounting for sensor noise. The dimensionality of the light field f is fixed by the image sensor, but the dimensionality (or equivalently, the sampling rate) of the volume g is a user-selectable parameter in the reconstruction algorithm. (a) Column i of the measurement matrix H (purple) contains the discretized light field point spread function for voxel i, which corresponds to the forward projection of that point in the volume. (b) Row j of the measurement matrix (green) contains a pixel back projection: a visualization that shows how much each voxel in the volume contributes to a single pixel j in the light field. The cross sections in this figure were computed using our wave optics model for a 20× 0.5NA water dipping objective and a 125μm pitch f/20 microlens array.

Fig. 5

Experimentally characterized resolution limits for two optical configurations of the light field microscope. (a) In a widefield microscope with no lenslet array, the target quickly goes out of focus when it is translated in z. (b) In a 3-D deconvolution from a light field, we lose resolution if the test target is placed at the native object plane (z = 0 μm), but we can reconstruct the target at much higher resolution than the spacing between lenslets when it is moved z = −15 μm (see also Fig. 1). Resolution falls off gradually beyond this depth (z = ±50 μm and ±100 μm). (c) Experimental MTF measured by analyzing the contrast of different line pair groupings in the USAF reconstruction. The colormap shows normalized contrast as measured using Eq. (10). The region of fluctuating resolution from z = −30 μm to 30 μm show that not all spatial frequencies are equally well reconstructed at all depths. (d) A slightly higher peak resolution (z = ±10 μm) can be achieved in the light field recorded with a 40× 0.8NA objective. However, the z = ±25 μm and ±50 μm planes in (d) have the same apparent resolution as the z = ±50 μm and ±100 μm planes in (b). (e) The experimental MTF for the 40× configuration shows that the region of fluctuating resolution (from −7.5 μm to 7.5 μm) is one quarter the size compared to (c). The solid green line in (c) and orange line in (e) are a 10% contrast cut-off representing the band limit of the reconstruction as a function of depth. Note that these plots are clipped to 645 cycles/mm, which is the highest resolution group on the USAF target.

Fig. 6

Experimentally measured band-limits from Fig. 5 re-plotted along with the lenslet sampling rate (dotted black line) and the Nyquist sampling rate at the diffraction limit of a widefield fluorescence microscope (dashed blue line). For comparison, we have plotted the depth of field of a widefield microscope (thin blue lines). The theoretical band limit we propose in Eq. 11 (dotted purple line) is in good agreement with the experimental band limit. However, this criterion only predicts the resolution fall-off at moderate to large z-positions, and not near the native object plane where diffraction and sampling effects cause the band limit to fluctuate.

Fig. 7

Lateral resolution fall-off as a function of depth for various optical design choices. (a) For a fixed 125μm pitch lenslet array, a larger objective magnification results in better peak resolution, but a more rapid fall-off and hence a diminished axial range over which good resolution can be achieved. (b) For a fixed magnification factor, decreasing the lenslet pitch achieves the same trade-off as in (a). In fact, Eq. (11) shows that multiplying the lenslet pitch by some constant has the same effect on the resolution criterion as dividing the objective magnification by that same amount.

Fig. 8

Pollen grains imaged with a 60× 1.4NA oil dipping objective and a 125 μm f/20 microlens array. (a) Max-intensity projections of a volume reconstructed using the computational refocusing algorithm presented in our 2009 paper [2] shows the irregular shape of the pollen grains, but low contrast and little high frequency detail. (b) Maximum intensity projections of a volume reconstructed using the 3-D deconvolution algorithm introduced in this paper shows the structures of the pollen grain more clearly. A small region of reconstruction artifacts appears around the native object plane. (c) Individual xy slices from the computational refocusing algorithm. (d) Slices from the 3-D deconvolved volume. A gamma correction of 0.6 was applied to panels (a), (b), (c), and (d) to help visualize their full dynamic range. After gamma correction, each panel was separately normalized so that the 99% of the intensity range was represented by the colormap shown by the scale bar.

Fig. 9

Geometric construction of the light field resolution criterion introduced in Eq. (11). In these figures conjugate images of the microlens array and image sensor are depicted on the object side of the microscope. Taking the magnification factor into account, the effective object-side pitch of the conjugated lenslet is d/M and its effective focal length is f_μlens/M². (a) The resolution criterion holds when point sources p₁ and p₂ are at sufficient depth |z| that they form diffraction limited spots on the sensor. Under this condition, the discriminability of the spots can be determined by simple geometric construction using similar triangles. Here, c is a constant that allows us to specify a particular resolution criterion (e.g., c = 1.22 selects the Rayleigh 2-point criterion, and c = 0.94 selects the Sparrow 2-point criterion [, p. 340]). In this paper, we use the Sparrow 2-point criterion since it is best suited for measuring where contrast drops to zero between two point projections measured in a digital image [, p. 340]. (b) To see where this resolution criterion holds, we measure the diameter b of the blur disk predicted using geometric optics. Our approximation is valid when this diameter is less than the diameter of a diffraction-limited spot (i.e. when b < cλ/NA).