ArticlePDF Available

Plenoptic Camera Resolution

Authors:
Todor Georgiev at Adobe
Todor Georgiev
  • Adobe
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

To resolve the low resolution problem of Plenoptic cameras we analyze optical signal sampling at frequencies above Nyquist. The resultant aliased signal is superresolved interleaving the array of microimages, thus cancelling the aliasing components. The rendered image can reach full sensor resolution.
Figures - uploaded by Todor Georgiev
Author content
All figure content in this area was uploaded by Todor Georgiev
Content may be subject to copyright.
Content uploaded by Todor Georgiev
Author content
All content in this area was uploaded by Todor Georgiev on Feb 25, 2021
Content may be subject to copyright.
JTh4A.2.pdf Imaging and Applied Optics © 2015 OSA
Plenoptic Camera Resolution
Todor Georgiev
Qualcomm Technologies Inc, SCL.A-112L, 3195 Kifer Road, Santa Clara, CA 95051
todorg@qti.qualcomm.com
Abstract: To resolve the low resolution problem of Plenoptic cameras we analyze optical signal
sampling at frequencies above Nyquist. The resultant aliased signal is superresolved interleaving
the array of microimages, thus cancelling the aliasing components. The rendered image can reach
full sensor resolution.
OCIS codes: (100.6640) Superresolution; (110.1758) Computational imaging; (110.5200) Photography; (110.4190) Multiple
imaging;
1. Introduction
The Plenoptic camera (see Fig 1) is a universal phase space sampling device. It measures the radiance, i.e. captures
ray intensities in 4D optical phase space. In comparison traditional cameras simply measure the irradiance, i.e.
record 2D image pixels. Recent lightfield research [1-4] demonstrates a number of results impossible with
traditional cameras, including image refocusing after capture, stereo and multi-view stereo with single camera, HDR
and multimodal imaging. Commercialization efforts made by companies like Raytrix and Lytro bring the plenoptic
camera closer to the goal of replacing the traditional camera in photography and possibly in in imaging, in general.
There is, however, a significant obstacle to making the plenoptic camera competitive in the market. That’s the
extremely low resolution of the final rendered image. Typically the final image has 40X lower resolution than the
resolution of the image sensor, measured in megapixels [5]. This makes the plenoptic camera look inferior compared
to the traditional digital camera. It appears that either we have to use current image sensors and accept the reduced
resolution, or we have to use extremely high resolution sensor at high cost. Neither of those is acceptable.
In this paper we are looking at another option: Superresolution [6, 7]. We analyze the plenoptic data capture
process in frequency domain and demonstrate that not only resolution can be improved without introducing artifacts,
but that it can actually reach the original sensor resolution.
2. Basic setting and formulas
To simplify our formulas, and without loss of generality, we will consider only 1D images. Based on microlenses,
the plenoptic camera captures an array of microimages, one for each microlens. We assume relay imaging model
(i.e. plenoptic 2.0 camera), where each microlens remaps part of the main lens image to the sensor, acting as a little
camera re-projecting the in-camera radiance from its own perspective.
Fig. 1 The main lens creates the main lens image, parts of which are mapped to the sensor by individual microlenses.
First we describe the formation of one separate microimage. Assuming pixel pitch p and pixel size ε, we represent
the image convolved with the pixel response function, and periodically sampled, as
g = ( f * )· where f (x) is the optical image, (x) is the pixel response function, (x) = Σ δ(x-np) is the Dirac
comb with peaks at the centers of pixels, and is the convolution. We will denote the Fourier transforms of f and g
by F and G respectively. Also u will be the independent frequency variable, and the Fourier transform of (x) will
be assumed to be sinc(πεu) = sin(πεu)/(πεu). Using these notations, after Fourier transform the captured image is
represented as
JTh4A.2.pdf Imaging and Applied Optics © 2015 OSA
G(u) = (F(u)·sinc(πεu)) 
 (1)
Usually pixel size is assumed to be the same as the pixel pitch, i.e. p = ε, and also F(u)·sinc(πεu) is assumed
essentially zero for frequencies above the Nyquist, i.e. above 1/(2p). This avoids aliasing. See Fig 2.
(a) (b)
Fig 2 (a) G(u) is periodic with period 1/p due to the convolution with Dirac comb. The central piece fits within the Nyquist bandwidth so there is
no aliasing. G(u) is generated from F(u) by modulation with the sinc function, thus reducing the bandwidth. However (see dotted curve) for
smaller and smaller pixel size the bandwidth can increase unlimitedly as ε decreases. (b) If pixel size is small and the optical signal has high
bandwidth, we observe aliasing in the microimages.
For this paper we will assume that the highest frequency to which pixels can respond reliably beyond the noise
level is determined by the location of first zero of the sinc function, which is at 1/ε. When ε = p the signal is limited
to no more than two times the Nyquist frequency. The related superresolution cannot achieve more than 2X increase
in resolution. However if we use a mask to limit the optically active pixel area to ε < p, the width of the sinc function
increases and we can sample unlimitedly high frequencies as ε gets smaller. Unfortunately, the signal becomes
severely aliased and not usable directly for imaging.
3. Superresolution removes aliasing
To introduce our approach first consider computing the sum of two aliased microimages containing frequencies up
to two times higher than the Nyquist, and shifted by half pixel relative to each-other. In the frequency domain such
shift corresponds to a phase multiplier  . As a result adding two such images cancels the component centered at
u = 1/p, see Fig 2(b). This effect may be viewed as “interference in the frequency domain” that removes aliasing.
Actually this process cancels all copies of the signal shifted by 1/p, 3/p, 5/p, …
Similar effect takes place with three images shifted by p/3 relative to each-other. Cancellation comes because of
the identity         The same can be shown with n images, each shifted by p/n relative to the
previous one. In this way all aliasing is removed by superresolution.
This process opens up a very interesting possibility to increase the resolution of the rendered image from a
plenoptic camera. Pixel size ε can be made as small as needed by using appropriate mask. The MTF of the lenses
can be made much better than the typical Nyquist frequency of today’s SLR cameras. The only limit is the sensor
resolution itself. This makes it possible to build a plenoptic camera that has no loss in resolution, i.e. the final
rendered image has the same resolution as the sensor.
4. Results
A wide range of experiments have been performed to confirm the above results. If optical bandwidth is sufficient to
support given frequencies, those frequencies are represented in the final image, with no aliasing. See Fig 3.
JTh4A.2.pdf Imaging and Applied Optics © 2015 OSA
Fig. 3. Raw captured image (left), and the image rendered from it (right). Text is not readable in the raw image.
The F-number of the lenses is critical to provide high frequencies because of the related cutoff frequency in the
MTF. This has been observed multiple times, especially when using cameras with small pixels at large F-numbers.
Cutoff frequency is the only real limitation of the method described.
In conclusion, up to full sensor resolution of the final rendered image can be achieved with a plenoptic camera.
One needs to take care of pixel size, mask pinholes and F/number of the lenses in order to achieve capture of
frequencies as high as 6 times the pixel pitch – which would ensure  times increase in resolution to cover
the full resolution of the sensor. This can be done for example with 6 μm pixels and lenses working at F/1.
5. References
[1] R Ng, M Levoy, M Bredif, G Duval, M Horowitz, et al., “Light field photography with a hand-held plenoptic camera,” Computer Science
Technical Report CSTR(Jan 2005).
[2] R. Ng, “Fourier slice photography,” ACM Trans. Graph., 735–744(2005).
[3] T. Georgiev, A. Lumsdaine, “Focused Plenoptic Camera and Rendering”, Journal of Electronic Imaging, Vol 19, Issue 2, 2010.
[4] T. Georgiev, A Lumsdaine, G. Chunev, “Using Focused Plenoptic Cameras for Rich Image Capture”, IEEE Computer Graphics and
Applications (Jan 2011).
[5] T. Georgiev, Z. Yu, A. Lumsdaine, S. Goma, “Lytro Camera Technology: Theory, Algorithms, Performance Analysis”, Proc. SPIE 8667,
Multimedia Content and Mobile Devices, 86671J (March 7, 2013).
[6] T. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution”, Proceedings ICCP 2009.
[7] T. Georgiev, G. Chunev, A. Lumsdaine, “Superresolution with the Focused Plenoptic Camera”, SPIE Electronic Imaging, Jan 2011.
... Knowledge of an optical system's light field (LF), i.e., of the locations and directions at which light rays cross transverse planes within a system, allows for numerical refocusing along the axial direction and perspective shifts after image acquisition [1][2][3][4][5][6][7][8][9][10]. The simplest conceptual implementation of a light field system is perhaps the original light field type 1 (i.e., LF1) configuration [1][2][3], which substitutes a lenslet array (LA) for a detector array in a system's nominal image plane, while shifting the detector array slightly downstream to the location of the array of pupil images produced by the LA (Fig. 1). ...
... For large N ppl , this can be a significant resolution loss. As a result, a number of approaches, with varying degrees of overlap and resolution improvement, have been investigated in order to attempt to improve spatial resolution beyond the lenslet scale, e.g., super-resolution techniques [8][9][10], removing the restriction that the final output pupil plane be coincident with the detector array (yielding the light-field type 2 (LF2) configuration [11][12][13]), moving the lenslet array to a pupil plane [14][15][16], the use of deconvolution [17], and combining LF images with a standard image [18]. ...
Improving image resolution on point-like sources in a type 1 light-field camera
Article
Full-text available
A ray-trace simulation of a type 1 light-field imager is used to show that resolutions significantly better than the lenslet scale can be deterministically reached in reconstructed images of isolated point-like sources. This is enabled by computationally projecting the system pupil onto the lenslet-array plane to better estimate the lenslet-plane-crossing locations through which the rays from a point source have passed on their way to the detector array. Improving light-field type 1 image resolution from the lenslet scale to the pixel scale can significantly enhance signal-to-noise ratios on faint point-like sources such as fluorescent microbes, making the technique of interest in, e.g., in situ microbial life searches in extreme environments.
View
... Even if a straightforward way to spatially super-resolve a light field is to apply known super-resolution techniques on each view individually, this means underusing the information that a light field exhibit, especially in terms of inter-view similarities. The work in [37] takes advantage of the non-integral shifts between corresponding pixels in two views, to propagate pixel intensities to a target view from its neighboring views. Using the same principle, various methods estimate the depth to derive these non-integral shifts [23,38,39]. ...
Light Field Image Compression and Compressive Acquisition
Thesis
Full-text available
  • May 2019
By capturing a scene from several points of view, a light field provides a rich representation of the scene geometry that brings a variety of novel post-capture applications and enables immersive experiences. The objective of this thesis is to study the compressibility of light field contents in order to propose novel solutions for higher-resolution light field imaging. Two main aspects were studied through this work. The compression performance on light fields of the actual coding schemes still being limited, there is need to introduce more adapted approaches to better describe the light field structures. We propose a scalable coding scheme that encodes only a subset of light field views and reconstruct the remaining views via a sparsity-based method. A residual coding provides an enhancement to the final quality of the decoded light field.Acquiring very large-scale light fields is still not feasible with the actual capture and storage facilities, a possible alternative is to reconstruct the densely sampled light field from a subset of acquired samples. We propose an automatic reconstruction method to recover a compressively sampled light field, that exploits its sparsity in the Fourier domain. No geometry estimation is needed, and an accurate reconstruction is achieved even with very low number of captured samples. A further study is conducted for the full scheme including a compressive sensing of a light field and its transmission via the proposed coding approach. The distortion introduced by the different processing is measured. The results show comparable performances to depth-based view synthesis methods.
View
... However, in simple light field cameras, the increased depth of field can come at the price of decreased lateral resolution [5]. Although this trade-off can be mitigated by making use of, e.g., super-resolution techniques [8][9][10][11], a better understanding of the processes that limit the basic light-field camera's resolution and reconstructed depth of field would be valuable as well. ...
Pupil segmentation in the light-field camera and its relation to 3D object positions and the reconstructed depth of field
Article
Full-text available
  • Feb 2019
A ray-trace simulation of the light-field camera is used to calculate point source responses as a function of 3D source positions. Each point source location yields a unique and well-determined segmented-pupil pattern in the lenslet array’s focal plane, with lateral object offsets changing the pattern’s location and symmetry, and defocus distances altering the pattern’s diameter. Segmented-pupil images can thus be used to infer point sources’ 3D locations. Numerical simulations show that the centroids and widths of segmented pupil images can be used to deduce lateral image positions to the size of a detector pixel, and image defocus to the accuracy of the lenslet focal length. In sparse-source cases, such as, e.g., fluorescence microscopy or particle tracking, 3D point-source locations can thus be accurately determined from the observed point source response patterns. The degree of pupil segmentation also directly constrains the ability to refocus light-field images—for image defocus distances large enough that the number of pupil segments exceeds the number of pixels within a “whole” pupil behind a single lenslet, the image can no longer be brought to focus numerically, thus defining the light-field camera’s depth of field. This constraint implies a depth of field larger than the usual imaging depth of focus by a factor of the number of detector pixels per lenslet, consistent with the general expectation.
View
... The quality of results in computational photography applications such as stereo depth, multi-image superresolution, and deconvolution, where a goal is to capture subpixel information, specifically requires a high MTF of both sensor and lens well above the Nyquist frequency. 1,2,5 Examination of measured sensor MTF at a wide range of frequencies allows us to understand pixel active area and shape in detail 3 and to assess sensor utility for an intended application. ...
Contrast computation methods for interferometric measurement of sensor modulation transfer function
Article
Full-text available
  • Feb 2018
  • J ELECTRON IMAGING
Accurate measurement of image-sensor frequency response over a wide range of spatial frequencies is very important for analyzing pixel array characteristics, such as modulation transfer function (MTF), crosstalk, and active pixel shape. Such analysis is especially significant in computational photography for the purposes of deconvolution, multi-image superresolution, and improved light-field capture. We use a lensless interferometric setup that produces high-quality fringes for measuring MTF over a wide range of frequencies (here, 37 to 434 line pairs per mm). We discuss the theoretical framework, involving Michelson and Fourier contrast measurement of the MTF, addressing phase alignment problems using a moiré pattern. We solidify the definition of Fourier contrast mathematically and compare it to Michelson contrast. Our interferometric measurement method shows high detail in the MTF, especially at high frequencies (above Nyquist frequency). We are able to estimate active pixel size and pixel pitch from measurements. We compare both simulation and experimental MTF results to a lens-free slanted-edge implementation using commercial software.
View
... Instead, we can use larger microlenses to produce higher-quality images, with fewer boundary pixels resulting in a more efficient use of the sensor [90]. This flexibility in spatioangular sampling also results in higher-resolution reconstructed images [92,93]. ...
Computational photography with plenoptic camera and light field capture: tutorial
Article
Full-text available
Photography is a cornerstone of imaging. Ever since cameras became consumer products more than a century ago, we have witnessed great technological progress in optics and recording mediums, with digital sensors replacing photographic films in most instances. The latest revolution is computational photography, which seeks to make image reconstruction computation an integral part of the image formation process; in this way, there can be new capabilities or better performance in the overall imaging system. A leading effort in this area is called the plenoptic camera, which aims at capturing the light field of an object; proper reconstruction algorithms can then adjust the focus after the image capture. In this tutorial paper, we first illustrate the concept of plenoptic function and light field from the perspective of geometric optics. This is followed by a discussion on early attempts and recent advances in the construction of the plenoptic camera. We will then describe the imaging model and computational algorithms that can reconstruct images at different focus points, using mathematical tools from ray optics and Fourier optics. Last, but not least, we will consider the trade-off in spatial resolution and highlight some research work to increase the spatial resolution of the resulting images.
View
Conception d'un capteur CMOS intégré pour la capture d'image plénoptique et traitement d'images associé
Thesis
Full-text available
  • Nov 2021
Un capteur plénoptique permet de capturer un champ de lumière c'est-à-dire d'échantillonner spatialement et angulairement les rayons lumineux provenant d'une scène, contrairement à un capteur d'image classique qui permet uniquement un échantillonage spatial. Cela offre des possibilités plus poussées en termes de traitement d'images. On peut par exemple refaire la mise au point après la prise de vue, corriger les aberrations de l'objectif, ou calculer une carte de profondeur de manière passive. L'échantillonnage angulaire est produit en plaçant plusieurs sous-pixels sous une même microlentille du capteur. Cette thèse concerne un capteur quad-pixel où une microlentille recouvre un patch de 2x2 sous-pixels. Il s'agit de l'évolution des capteurs "dual-pixels" développés par Canon en 2012, principalement utilisés pour contrôler l'autofocus de leurs appareils photographiques.Elle a pour objectif de tisser des liens entre deux des trois aspects du système d'imagerie, les pixels d'une part, et le traitement de l'images d'autre part. L'optique utilisée pour produire l'image sur le capteur est laissée de coté. Je présente d'abord des simulations au niveau des pixels, principalement en utilisant des sous-pixels de 1.75 microns. A partir du critère de performance défini, je présente des pistes d'amélioration des pixels. En me basant sur ces simulations, j'ai modifié un outil de rendu d'images en lancé de rayons qui permet de prendre en compte la diffraction dans les microlentilles. Je montre que la diffraction due à la microlentille dégrade les performances des algorithmes, je prends pour exemple le cas de la correction des aberrations de l'objectif principal. Cet outil a de plus permis de générer des images puis d'étudier les algorithmes de traitement d'images. Je présente plusieurs pistes d'utilisation d'un quad-pixel à l'aide des techniques d'apprentissage profond et de réseaux de neurones, comme par exemple le dématriçage couleur ou la correction de la diffraction. Enfin, je présente et caractérise un prototype quad-pixel réalisé dans le cadre de cette thèse.
View
Light-field image acquisition from a conventional camera: Design of a four minilens ring device
Article
Full-text available
  • Jan 2019
In the past few years, a new type of camera has been emerging on the market: a digital camera capable of capturing both the intensity of the light emanating from a scene and the direction of the light rays. This camera technology called a light-field camera uses an array of lenses placed in front of a single image sensor, or simply, an array of cameras attached together. An optical device is proposed: a four minilens ring that is inserted between the lens and the image sensor of a digital camera. This device prototype is able to convert a regular digital camera into a light-field camera as it makes it possible to record four subaperture images of the scene. It is a compact and cost-effective solution to perform both postcapture refocusing and depth estimation. The minilens ring makes also the plenoptic camera versatile; it is possible to adjust the parameters of the ring so as to reduce or increase the size of the projected image. Together with the proof of concept of this device, we propose a method to estimate the positions of each optical component depending on the observed scene (object size and distance) and the optics parameters. Real-world results are presented to validate our device prototype. © 2019 Society of Photo-Optical Instrumentation Engineers (SPIE).
View
Lytro camera technology: Theory, algorithms, performance analysis
Article
Full-text available
  • Mar 2013
  • Proceedings of SPIE
The Lytro camera is the first implementation of a plenoptic camera for the consumer market. We consider it a successful example of the miniaturization aided by the increase in computational power characterizing mobile computational photography. The plenoptic camera approach to radiance capture uses a microlens array as an imaging system focused on the focal plane of the main camera lens. This paper analyzes the performance of Lytro camera from a system level perspective, considering the Lytro camera as a black box, and uses our interpretation of Lytro image data saved by the camera. We present our findings based on our interpretation of Lytro camera file structure, image calibration and image rendering; in this context, artifacts and final image resolution are discussed.
View
Light Field Photography with a Hand-Held Plenopic Camera
Article
Full-text available
  • Jan 2005
This paper presents a camera that samples the 4D light field on its sensor in a single photographic exposure. This is achieved by in- serting a microlens array between the sensor and main lens, creat- ing a plenoptic camera. Each microlens measures not just the total amount of light deposited at that location, but how much light ar- rives along each ray. By re-sorting the measured rays of light to where they would have terminated in slightly different, synthetic cameras, we can compute sharp photographs focused at different depths. We show that a linear increase in the resolution of images under each microlens results in a linear increase in the sharpness of the refocused photographs. This property allows us to extend the depth of field of the camera without reducing the aperture, en- abling shorter exposures and lower image noise. Especially in the macrophotography regime, we demonstrate that we can also com- pute synthetic photographs from a range of different viewpoints. These capabilities argue for a different strategy in designing photo- graphic imaging systems. To the photographer, the plenoptic camera operates exactly like an ordinary hand-held camera. We have used our prototype to take hundreds of light field photographs, and we present examples of portraits, high-speed action and macro close-ups.
View
Superresolution with the Focused Plenoptic Camera
Article
Full-text available
  • Feb 2011
  • Proceedings of SPIE
Digital images from a CCD or CMOS sensor with a color filter array must undergo a demosaicing process to combine the separate color samples into a single color image. This interpolation process can interfere with the subsequent superresolution process. Plenoptic superresolution, which relies on precise sub-pixel sampling across captured microimages, is particularly sensitive to such resampling of the raw data. In this paper we present an approach for superresolving plenoptic images that takes place at the time of demosaicing the raw color image data. Our approach exploits the interleaving provided by typical color filter arrays (e.g., Bayer filter) to further refine plenoptic sub-pixel sampling. Our rendering algorithm treats the color channels in a plenoptic image separately, which improves final superresolution by a factor of two. With appropriate plenoptic capture we show the theoretical possibility for rendering final images at full sensor resolution.
View
Using Focused Plenoptic Cameras for Rich Image Capture
Article
Full-text available
  • Mar 2011
Edward Adelson and James Bergen defined the plenoptic function as a record of the 3D distribution of light rays and their dependence on parameters such as wavelength and polarization. Most research on these ideas has emphasized the 3D aspect of plenoptic-function capture and manipulation, paying less attention to other parameters. A new approach leverages the focused plenoptic camera's high-resolution and flexible-sampling trade-offs to perform high-resolution capture of the plenoptic function's rich "non 3D" structure. This approach employs two techniques. The first simultaneously captures multiple exposures with a microlens array having an interleaved set of different filters. The second places multiple filters at the main lens aperture. Experimental results validate the approach, producing 1.3-megapixel high-dynamic-range images with one capture.
View
Light field superresolution
Conference Paper
Full-text available
  • May 2009
Light field cameras have been recently shown to be very effective in applications such as digital refocusing and 3D reconstruction. In a single snapshot these cameras provide a sample of the light field of a scene by trading off spatial resolution with angular resolution. Current methods produce images at a resolution that is much lower than that of traditional imaging devices. However, by explicitly modeling the image formation process and incorporating priors such as Lambertianity and texture statistics, these types of images can be reconstructed at a higher resolution. We formulate this method in a variational Bayesian framework and perform the reconstruction of both the surface of the scene and the (superresolved) light field. The method is demonstrated on both synthetic and real images captured with our light-field camera prototype.
View
Focused plenoptic camera and rendering
Article
Full-text available
  • Apr 2010
  • J ELECTRON IMAGING
Plenoptic cameras, constructed with internal microlens arrays, capture both spatial and angular information, i.e., the full 4-D radiance, of a scene. The design of traditional plenoptic cameras assumes that each microlens image is completely defocused with respect to the image created by the main camera lens. As a result, only a single pixel in the final image is rendered from each microlens image, resulting in disappointingly low resolution. A recently developed alternative approach based on the focused plenoptic camera uses the microlens array as an imaging system focused on the image plane of the main camera lens. The flexible spatioangular trade-off that becomes available with this design enables rendering of final images with significantly higher resolution than those from traditional plenoptic cameras. We analyze the focused plenoptic camera in optical phase space and present basic, blended, and depth-based rendering algorithms for producing high-quality, high-resolution images. We also present our graphics-processing-unit-based implementations of these algorithms, which are able to render full screen refocused images in real time.
View
Fourier slice photography
Article
  • Jul 2005
This paper contributes to the theory of photograph formation from light fields. The main result is a theorem that, in the Fourier domain, a photograph formed by a full lens aperture is a 2D slice in the 4D light field. Photographs focused at different depths correspond to slices at different trajectories in the 4D space. The paper demonstrates the utility of this theorem in two different ways. First, the theorem is used to analyze the performance of digital refocusing, where one computes photographs focused at different depths from a single light field. The analysis shows in closed form that the sharpness of refocused photographs increases linearly with directional resolution. Second, the theorem yields a Fourier-domain algorithm for digital refocusing, where we extract the appropriate 2D slice of the light field's Fourier transform, and perform an inverse 2D Fourier transform. This method is faster than previous approaches.
View