Figure - available from: Optics Express
This content is subject to copyright. Terms and conditions apply.
Diagram of four multishot compressive multispectral cameras: (a) DD-CASSI, (b) SD-CASSI, (c) 3D-CASSI and (d) proposed architecture.
Source publication
![Spectral image reconstruction with K = 2 and SNR = 50[dB]. Here, for...](publication/333738277/figure/fig3/AS:11431281253112069@1718825621269/Spectral-image-reconstruction-with-K-2-and-SNR-50dB-Here-for-the-first-snapshot_Q320.jpg)
Compressive Spectral Imaging (CSI) is an emerging technology that aims at reconstructing a spectral image from a limited set of two-dimensional projections. To capture these projections, CSI architectures often combine light dispersive elements with coded apertures or programmable spatial light modulators. This work introduces a novel and compact C...
Similar publications
Deep networks have achieved great success in many areas in recent years. However, with the increasing sophistication of deep neural networks (DNNs), the memory consumption and computational cost expand exponentially, greatly hindering their application in mobile devices and other limited resources. Therefore, there is impending necessity to conside...
Citations
... Compressive Spectral Imaging via Deformable Mirror and Colored-Mosaic Detector (CSI-DMCMD) [61] introduces a compact CSI architecture composed of a confocal 4f system with a deformable mirror (DM) located in the middle and a colored-filter detector array located in the focal output. Here, the DM introduces a controlled phase modulation and the colored-filter spatialspectral modulation. ...
Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow the identification of objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared with conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarizes the advances in CSI, starting with SI and its relevance and continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, as well as the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks.
... Height map [24], [25] response, known as the transmission coefficient. Based on a Riemann summation of the integral (1), the propagation of U 0 up to the DOE as well as the DOE's modulation can be mathematically expressed as ...
Computational optical imaging (COI) systems leverage optical coding elements (CEs) in their setups to encode a high-dimensional scene in a single or in multiple snapshots and decode it by using computational algorithms. The performance of COI systems highly depends on the design of its main components: the CE pattern and the computational method used to perform a given task. Conventional approaches rely on random patterns or analytical designs to set distribution of the CE. However, the available data and algorithm capabilities of deep neural networks (DNNs) have opened a new horizon in CE data-driven designs that jointly consider the optical and computational decoders. Specifically, by modeling the COI measurements through a fully differentiable image-formation model that considers the physics-based propagation of light and its interaction with the CEs, the parameters that define the CE and the computational decoder can be optimized in an end-to-end (E2E) manner. Moreover, by optimizing just CEs in the same framework, inference tasks can be performed from pure optics. This work surveys the recent advances in CE data-driven design and provides guidelines on how to parameterize different optical elements to include them in the E2E framework. As the E2E framework can handle different inference applications by changing the loss function and the DNN, we present low-level tasks such as spectral imaging reconstruction or high-level tasks such as pose estimation with privacy preservation enhanced by using optimal task-based optical architectures. Finally, we illustrate classification and 3D object-recognition applications performed at the speed of the light using all-optics DNNs.
... Concluding the section, we note the possibility of combining different regularizing functionals as it was done, for example, in [34], where the linear combination of norm L 2 , norm L 1 , and the total variation was used. In this case, more complicated methods are required (because it is necessary to minimize or determine the root of a function of several parameters and not of one parameter) for choosing the values of regularizing parameters, which generalize the discrepancy principle [35,36], the L-curve method [37], the generalized cross-validation method [38], or the methods based on application of test measurements [39]. ...
... Height map [24], [25] deep optical design. The first step defines the more suitable light propagation model, which is selected based on the light source conditions, the propagation distances, the optical elements, and the target physical constraints. ...
Computational optical imaging (COI) systems leverage optical coding elements (CE) in their setups to encode a high-dimensional scene in a single or multiple snapshots and decode it by using computational algorithms. The performance of COI systems highly depends on the design of its main components: the CE pattern and the computational method used to perform a given task. Conventional approaches rely on random patterns or analytical designs to set the distribution of the CE. However, the available data and algorithm capabilities of deep neural networks (DNNs) have opened a new horizon in CE data-driven designs that jointly consider the optical encoder and computational decoder. Specifically, by modeling the COI measurements through a fully differentiable image formation model that considers the physics-based propagation of light and its interaction with the CEs, the parameters that define the CE and the computational decoder can be optimized in an end-to-end (E2E) manner. Moreover, by optimizing just CEs in the same framework, inference tasks can be performed from pure optics. This work surveys the recent advances on CE data-driven design and provides guidelines on how to parametrize different optical elements to include them in the E2E framework. Since the E2E framework can handle different inference applications by changing the loss function and the DNN, we present low-level tasks such as spectral imaging reconstruction or high-level tasks such as pose estimation with privacy preserving enhanced by using optimal task-based optical architectures. Finally, we illustrate classification and 3D object recognition applications performed at the speed of the light using all-optics DNN.
... active depth sensing methods-and dependent on the use of multiple acquisitions. On the other hand, recent passive CDI and CSI systems have separately exploited the use of DM as a programmable phase coding device to perform wavefront modulation at the pupil plane [24][25][26]. By combining the DM and a DMD (such as in multishot CASSI), in this work we propose a CSDI optical system that jointly modulates phase and amplitude to acquire compressive measurements of the SDI 4D hypercube in a single snapshot. ...
Compressive spectral depth imaging (CSDI) is an emerging technology aiming to reconstruct spectral and depth information of a scene from a limited set of two-dimensional projections. CSDI architectures have conventionally relied on stereo setups that require the acquisition of multiple shots attained via dynamically programmable spatial light modulators (SLM). This work proposes a snapshot CSDI architecture that exploits both phase and amplitude modulation and uses a single image sensor. Specifically, we modulate the spectral-depth information in two steps. Firstly, a deformable mirror (DM) is used as a phase modulator to induce a focal length sweeping while simultaneously introducing a controlled aberration. The phase-modulated wavefront is then spatially modulated and spectrally dispersed by a digital micromirror device (DMD) and a prism, respectively. Therefore, each depth plane is modulated by a variable phase and binary code. Complimentary, we also propose a computational methodology to recover the underlying spectral depth hypercube efficiently. Through simulations and our experimental proof-of-concept implementation, we demonstrate that the proposed computational imaging system is a viable approach to capture spectral-depth hypercubes from a single image.
... Moreover, in the optical system, there is a suggestion for a technique to insert a coded aperture filter [28][29][30][31]; however, its calculation is complex and a long time is required to acquire an image. There have been attempts to solve the problem of massive data quantity of the multispectral image with a compression sensing (coded aperture) [32][33][34][35][36][37][38][39]. However, the optics is complicated, and it is unsuitable for handy photography. ...
This study proposes a new imaging technique for snapshot multispectral imaging in which a multispectral image was captured using an imaging lens that combines a set of multiple spectral filters and polarization filters, as well as a pixel-wise color polarization image sensor. The author produced a prototype nine-band multispectral camera system that covered from visible to near-infrared regions and was very compact. The camera’s spectral performance was evaluated using experiments; moreover, the camera was used to detect the freshness of food and the activity of wild plants and was mounted on a vehicle to obtain a multispectral video while driving.
... An RGB sensor functions similarly to the colored CA, which provides an additional modulation in the spectral dimension with the spectral response (spectral transmission for colored CA). And as the Foveon structure absorbs all light arriving at the sensor, the light efficiency of DD-RGB CASSI system is identical to the common SD CASSI or DD CASSI system, which is generally twice higher than the colored CA systems [35]. Furthermore, the coherence of A can be further decreased with the DD-RGB CASSI system, which is also discussed later in the results part. ...
Coded aperture snapshot spectral imaging (CASSI) reconstructs a hyperspectral image from several two-dimensional (2D) projections via compressive sensing. The reconstruction quality and the sampling efficiency of CASSI can be effectively improved by decreasing the coherence of the underlying sensing matrix. Efforts have been made to minimize the coherence with individual optimization on coded aperture or sparse basis. In this paper, a simultaneous optimization on the system projection and the over-complete dictionary is introduced to minimize the Frobenius norm coherence. The dual-disperser structure and the RGB image sensor are adopted for the lowest coherence in terms of system configuration. The coded aperture and the dictionary are optimized with genetic algorithm and gradient descent respectively, and simultaneous optimization is conducted iteratively. Low coherence of sensing matrix is acquired after the simultaneous optimization, with both reconstruction quality and sampling efficiency significantly improved. Compared to the non-optimized system and state-of-the-art systems with individually optimized coded aperture or dictionary, the simultaneous optimization promotes the peak signal-to-noise ratio by more than 5dB. The coherence minimization via simultaneous optimization on the system matrix and the sparse representation basis may open opportunities for further development of other compressive-sensing-based computational imaging systems.
... Hence, their cardinal drawback is the inherent time-consuming acquisition, which prevents the usage of a spectral imager in fast-changing scenes and exposes the system to large volumes of data. On the other hand, in snapshot spectral imaging (SSI) systems [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23], the spectral cube is acquired within a single shot (snapshot) and potentially eliminates the use of large data volumes and moving mechanisms. However, they are generally complex, expensive, show insufficient accuracy of the spectral cube, and suffer from low light throughput. ...
... Other compressive SI systems have also been suggested. In Ref. [14], an architecture based on a deformable mirror and a colored-filter detector is presented. Combined with a set of imaging lenses and a beam splitter, the system produces a compressive spatiospectral projection without the need of a grating or prism like in common CASSI-based architectures. ...
We propose a snapshot spectral imaging method for the visible spectral range using two digital cameras placed side-by-side: a regular red–green–blue (RGB) camera and a monochromatic camera equipped with a dispersive diffractive diffuser placed at the pupil of the imaging lens. While spectral imaging was shown to be feasible using a single monochromatic camera with a pupil diffuser [Appl. Opt. 55, 432 (2016)APOPAI0003-693510.1364/AO.55.000432], adding an RGB camera provides more spatial and spectral information for stable reconstruction of the spectral cube of a scene. Results of optical experiments confirm that the combined data from the two cameras relax the complexity of the underdetermined reconstruction problem and improve the reconstructed image quality obtained using compressed sensing-based algorithms.
This chapter provides a comprehensive and up-to-date review of the rolling shutter (RS) mechanism found in complementary metal-oxide semiconductor (CMOS) imaging sensors as a coding opportunity for high-speed videos. Based on this, we present a cutting-edge readout architecture called shuffled rolling shutter (SRS) that optimally designs the scanlines entries of the RS for improved sampling of the space-time datacube. The RS restriction implies that only one entry per column is sampled at each exposure time. This restriction can be modeled using a three-dimensional N2 queens problem (3DN2QP), such that the optimal design of entries is computed straightforwardly, solving a sphere packing (SP) problem. The main advantage of packing equal-size spheres in a cubic container is promoting uniform spatiotemporal sampling. Simulation and experimental results prove that with the captured measurements using the proposed SRS, we can recover the underlying video from a snapshot. This opens the avenue for the development of new CMOS imaging detectors with shuffled positions of the RS, now able to turn a typical nuisance into an opportunity to capture and recover videos from snapshots.
Significance
Compressed ultrafast photography (CUP) is currently the world’s fastest single-shot imaging technique. Through the integration of compressed sensing and streak imaging, CUP can capture a transient event in a single camera exposure with imaging speeds from thousands to trillions of frames per second, at micrometer-level spatial resolutions, and in broad sensing spectral ranges.
Aim
This tutorial aims to provide a comprehensive review of CUP in its fundamental methods, system implementations, biomedical applications, and prospect.
Approach
A step-by-step guideline to CUP’s forward model and representative image reconstruction algorithms is presented with sample codes and illustrations in Matlab and Python. Then, CUP’s hardware implementation is described with a focus on the representative techniques, advantages, and limitations of the three key components—the spatial encoder, the temporal shearing unit, and the two-dimensional sensor. Furthermore, four representative biomedical applications enabled by CUP are discussed, followed by the prospect of CUP’s technical advancement.
Conclusions
CUP has emerged as a state-of-the-art ultrafast imaging technology. Its advanced imaging ability and versatility contribute to unprecedented observations and new applications in biomedicine. CUP holds great promise in improving technical specifications and facilitating the investigation of biomedical processes.
