Figure - available from: Journal of the Optical Society of America A
This content is subject to copyright. Terms and conditions apply.
Example of a discrete color filter array. Each spatial location ( m , n ) contains the spectral response of the correspondent optical filter. Black features in C m , n , k block the correspondent wavelengths; white features correspond to the passbands of the filter.
Source publication
Traditional spectral imaging approaches require sensing all the voxels of a scene. Colored mosaic FPA detector-based architectures can acquire sets of the scene’s spectral components, but the number of spectral planes depends directly on the number of available filters used on the FPA, which leads to reduced spatiospectral resolutions. Instead of s...
Citations
... Initially, both HSI and depth imaging need scanning to acquire accurate measurements, but the scanning acquisition is limited by a time-consuming process and the use of cumbersome equipment. Due to the significant development of image sensors and reconstruction algorithms, snapshot HSI [7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23] and non-scanning depth imaging [4,[24][25][26][27] have been developed rapidly and provide the possibility of acquiring both depth and spectrum information through a single exposure. ...
We propose, to the best of our knowledge, a novel deep learning–enabled four-dimensional spectral imaging system composed of a reflective coded aperture snapshot spectral imaging system and a panchromatic camera. The system simultaneously captures a compressively coded hyperspectral measurement and a panchromatic measurement. The hyperspectral data cube is recovered by the U-net-3D network. The depth information of the scene is then acquired by estimating a disparity map between the hyperspectral data cube and the panchromatic measurement through stereo matching. This disparity map is used to align the hyperspectral data cube and the panchromatic measurement. A designed fusion network is used to improve the spatial reconstruction of the hyperspectral data cube by fusing aligned panchromatic measurements. The hardware prototype of the proposed system demonstrates high-speed four-dimensional spectral imaging that allows for simultaneously acquiring depth and spectral images with an 8 nm spectral resolution between 450 and 700 nm, 2.5 mm depth accuracy, and a 1.83 s reconstruction time.
... They argue that the increased modulation depth better satisfies the restricted isometry property (RIP) during the measurement process [109]. They mainly introduced two configurations: The first replaces the binary coded aperture in SD-CASSI with a spectral filter array [109], [110], and the second configuration employs an image sensor based on an MSFA, designed to capture light dispersed by a dispersive element in the scene [111], [112]. The spectral response function at each detector pixel is position-dependent; by rotating the dispersive element in the system, multiple snapshots can be obtained for improved imaging [113]. ...
Snapshot spectral imaging technology enables the capture of complete spectral information of objects in an extremely short period of time, offering wide-ranging applications in fields requiring dynamic observations such as environmental monitoring, medical diagnostics, and industrial inspection. In the past decades, snapshot spectral imaging has made remarkable breakthroughs with the emergence of new computational theories and optical components. From the early days of using various spatial-spectral data mapping methods, they have evolved to later attempts to encode various dimensions of light, such as amplitude, phase, and wavelength, and then computationally reconstruct them. This review focuses on a systematic presentation of the system architecture and mathematical modeling of these snapshot spectral imaging techniques. In addition, the introduction of metasurfaces expands the modulation of spatial-spectral data and brings advantages such as system size reduction, which has become a research hotspot in recent years and is regarded as the key to the next-generation snapshot spectral imaging techniques. This paper provides a systematic overview of the applications of metasurfaces in snapshot spectral imaging and provides an outlook on future directions and research priorities.
... Thus, these systems face challenges in terms of complexity, slow speed, and inconsistencies between the spectral and image resolutions 27,28 . Building upon the compressive sensing theory 29,30 , a number of scan-less [31][32][33][34][35] HI systems has been successively proposed that can reconstruct a complete spectral image snapshot without the need for scanning, but with a coded aperture. Owing to the imaging solution and bandwidth limitations of the aperture as well as recent advancements in diffractive optical elements (DOEs) [36][37][38][39][40][41] , there has been a growing interest in incorporating coding capabilities into a single DOE. ...
Thin and flat diffractive optical elements (DOEs) are significant in the field of integrated optics and provide a novel and optimal solution for hyperspectral imaging (HI) which is expected to be compact, snapshot, with large depth of field (DoF) and resolution. The tradeoff between spectral and spatial resolutions caused by the restricted DoF, limits the application scenarios for HI. To address this, based on the prior of spatial and spectral sparse, we propose a spatial–spectral achromatic (SSA) neural network to end-to-end optimize a broad-bandwidth system with a DOE to provide the support for snapshotly achromatic extreme-DoF HI. We experimentally show that our system can snapshotly capture achromatic, high-fidelity hyperspectral images with 25 spectral channels ranging from 420 nm to 660 nm, covering distances from 0.5 m to 5 m. The proposed system enables precise and dynamic reconstruction of spectra within an extreme DoF, a capability previously unattainable with compact computational spectral cameras. The precise reconstruction of spectra demonstrates the potential of the developed system in various applications, such as precision agriculture, food quality inspection, and object detection.
... Building upon this, here we propose a TMCA codification strategy for SD imaging based on the synchronization of a time-varying phase-coded aperture (phase modulator) with a shutter function, in conjunction with a CCA. Phase-coded apertures provide depth-variant PSFs, useful to encode depth in a single snapshot [18,22,23], while the CCA boosts the flexibility of encoding spectral information [24][25][26]. Here, we exploit these benefits using TMCA and demonstrate that this combination induces a novel codification strategy for snapshot SD imaging. ...
Depth and spectral imaging are essential technologies for a myriad of applications but have been conventionally studied as individual problems. Recent efforts have been made to optically encode spectral-depth (SD) information jointly in a single image sensor measurement, subsequently decoded by a computational algorithm. The performance of single snapshot SD imaging systems mainly depends on the optical modulation function, referred to as codification, and the computational methods used to recover the SD information from the coded measurement. The optical modulation has been conventionally realized using coded apertures (CAs), phase masks, prisms or gratings, active illumination, and many others. In this work, we propose an optical modulation (codification) strategy that employs a color-coded aperture (CCA) in conjunction with a time-varying phase-coded aperture and a spatially-varying pixel shutter, thus yielding an effective time-multiplexed coded aperture (TMCA). We show that the proposed TMCA entails a spatially-variant point spread function (PSF) for a constant depth in a scene, which, in turn, facilitates the distinguishability, and therefore, better recovery of the depth information. Further, the selective filtering of specific spectral bands by the CCA encodes relevant spectral information that is disentangled using a reconstruction algorithm. We leverage the advances of deep learning techniques to jointly learn the optical modulation and the computational decoding algorithm in an end-to-end (E2E) framework. We demonstrate via simulations and with a real testbed prototype that the proposed TMCA strategy outperforms state-of-the-art snapshot SD imaging alternatives in both spectral and depth reconstruction quality.
... Researchers have designed various encoded snapshot spectral imaging setups, in which disperser [24][25][26] or diffuser 27,28 play important roles and emerging metasurface further promotes the miniaturization 29,30 . Among these encoding schemes, the compressive sensing method enables high spatio-spectral acquisition and inspires some new setups, including the precedent Coded Aperture Snapshot Spectral Imager (CASSI) 31 and its variants for performance improvement or system compactness 20,[32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47][48][49][50] . Besides, further introducing temporal continuity shares the same mathematical model as CASSI 51 and can achieve multi-spectral videography with increased frame rate 52 . ...
Multi-spectral imaging is a fundamental tool characterizing the constituent energy of scene radiation. However, current multi-spectral video cameras cannot scale up beyond megapixel resolution due to optical constraints and the complexity of the reconstruction algorithms. To circumvent the above issues, we propose a tens-of-megapixel handheld multi-spectral videography approach (THETA), with a proof-of-concept camera achieving 65-megapixel videography of 12 wavebands within visible light range. The high performance is brought by multiple designs: We propose an imaging scheme to fabricate a thin mask for encoding spatio-spectral data using a conventional film camera. Afterwards, a fiber optic plate is introduced for building a compact prototype supporting pixel-wise encoding with a large space-bandwidth product. Finally, a deep-network-based algorithm is adopted for large-scale multi-spectral data decoding, with the coding pattern specially designed to facilitate efficient coarse-to-fine model training. Experimentally, we demonstrate THETA’s advantageous and wide applications in outdoor imaging of large macroscopic scenes.
... For instance, Rueda et al. successfully replaced the binary CA with an array of spectral filters entailing better encoding strategies in the colored coded aperture snapshot compressive imaging (C-CASSI) [19]. In 2015, Correa et al. introduced the snapshot colored compressive spectral imager (SCCSI) [20], which shears the spatio-spectral information using a prism. Then, an array of filters encodes the light before the FPA. ...
Multispectral imaging (MSI) collects a datacube of spatio-spectral information of a scene. Many acquisition methods for spectral imaging use scanning, preventing its widespread usage for dynamic scenes. On the other hand, the conventional color filter array (CFA) method often used to sample color images has also been extended to snapshot MSI using a Multispectral Filter Array (MSFA), which is a mosaic of selective spectral filters placed over the Focal Plane Array (FPA). However, even state-of-the-art MSFAs coding patterns produce artifacts and distortions in the reconstructed spectral images, which might be due to the nonoptimal distribution of the spectral filters. To reduce the appearance of artifacts and provide tools for the optimal design of MSFAs, this paper proposes a novel mathematical framework to design MSFAs using a Sphere Packing (SP) approach. By assuming that each sampled filter can be represented by a sphere within the discrete datacube, SP organizes the position of the equal-size and disjoint spheres's centers in a cubic container. Our method is denoted Multispectral Filter Array by Optimal Sphere Packing (MSFA-OSP), which seeks filter positions that maximize the minimum distance between the spheres's centers. Simulation results show an image quality improvement of up to 2 dB and a remarkable boost in spectral similarity when using our proposed MSFA design approach for a variety of reconstruction algorithms. Moreover, MSFA-OSP notably reduces the appearance of false colors and zipper effect artifacts, often seen when using state-of-the-art demosaicking algorithms. Experiments using synthetic and real data prove that the proposed MSFA-OSP outperforms state-of-the-art MSFAs in terms of spatial and spectral fidelity. The code that reproduces the figures of this article is available at https://github.com/nelson10/DemosaickingMultispectral3DSpherePacking.git.
... In addition, there are other solutions for a compact hyperspectral imaging system. The compact system based on wavelength coding [16,24,25] achieves hyperspectral imaging by plating filters of different wavelengths on the pixels. However, the imaging channels of this system are limited, since more imaging channels mean lower spatial resolution. ...
Hyperspectral imaging attempts to determine distinctive information in spatial and spectral domain of a target. Over the past few years, hyperspectral imaging systems have developed towards lighter and faster. In phase-coded hyperspectral imaging systems, a better coding aperture design can improve the spectral accuracy relatively. Using wave optics, we post an equalization designed phase-coded aperture to achieve desired equalization point spread functions (PSFs) which provides richer features for subsequent image reconstruction. During the reconstruction of images, our raised hyperspectral reconstruction network, CAFormer, achieves better results than the state-of-the-art networks with less computation by substituting self-attention with channel-attention. Our work revolves around the equalization design of the phase-coded aperture and optimizes the imaging process from three aspects: hardware design, reconstruction algorithm, and PSF calibration. Our work is putting snapshot compact hyperspectral technology closer to a practical application.
... Nonetheless, CSI systems impose an ill-conditioned sensing problem where a recovery stage that includes prior information of the SI is needed to estimate the scene [8,12,13,14]. For example, SD-CASSI [10], DD-CASSI [11], SCCSI [15] are proposed CSI systems for the visible (VIS) range. However, scanning and CSI systems require high-cost specialized optical sensors with a substantial limitation in acquiring high-resolution infrared SI from the NIR spectrum [16]. ...
Hadamard single-pixel imaging (HSI) is a promising sensing approach for acquiring spectral images in the near-infrared spectrum with high spatial resolution and fast recovery times due to the efficient invertible properties of the Hadamard matrix. The potential of the HSI system is diminished because of the large number of required measurements which implies long acquisition times. Recent advances proposed optimizing the HSI sensing matrix structure based on a superpixels map estimated from a side-information acquisition of the scene, reducing the number of required measurements. However, these matrix designs are detached from the recovery task, which falls on a sub-optimal strategy. In this work, we proposed an adaptive end-to-end sensing methodology for the HSI sensing matrix design based on deep superpixels estimation by coupling the sensing and recovery of the near-infrared spectral images. Experimental results show the superiority of the proposed sensing methodology compared with state-of-art sensing design schemes.
... One of the earliest forms of coded aperture snapshot spectral imaging used a binary spatial aperture function imaged onto a dispersive optical element through relay optics, encoding both the spatial and spectral features contained within the input FOV into an intensity pattern collected by a monochrome focal-plane array 26 . Since this initial proof-of-concept demonstration, various improvements have been reported on coded aperture-based snapshot spectral imaging systems based on, e.g., the use of color-coded apertures 27 , compressive sensing techniques [28][29][30][31] and others 32 . On the other hand, these systems still require the use of optical relay systems and dispersive optical elements such as prisms, and diffractive elements, resulting in relatively bulky form factors; furthermore, their frame rate is often limited by the computationally intense iterative recovery algorithms that are used to digitally retrieve the multispectral image cube from the raw data. ...
Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 λ m , where λ m is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.
... Thus, multiple SI priors have been used. To name, [30,31,55] assume that a SI is spatially smooth, such that it provides a low total variation (TV), i.e, the SI is sparse in the spatial difference domain [56,59] explore spatial sparsity using 1 -norm, on a given orthonormal basis such as Wavelet, and spectral sparsity DCT domain [16,81] or [26,82] assumes that the SI has a low-rank structure which is evidenced by considering the linear mixture model. The most powerful DNNs have been adapted with DL techniques to recover the SI from the measurements (Unets, recurrent neural networks, generative adversarial networks, transformers) [44,[83][84][85][86], [37,40,58,87]. ...
Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow the identification of objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared with conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarizes the advances in CSI, starting with SI and its relevance and continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, as well as the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks.
