Fig 4 - available via license: CC BY
Content may be subject to copyright.
Comparison of PhysenNet, conventional end-to-end DL and RED. a, f The diffraction patterns at a distance of 10 mm corresponding to the real phase distributions shown in b and g. c, h The phase images reconstructed using the conventional end-to-end strategy, d, i the phase images reconstructed by PhysenNet, and e, j the phase images reconstructed by RED
Source publication
Most of the neural networks proposed so far for computational imaging (CI) in optics employ a supervised training strategy, and thus need a large training set to optimize their weights and biases. Setting aside the requirements of environmental and system stability during many hours of data acquisition, in many practical applications, it is unlikel...
Contexts in source publication
Context 1
... approaches for phase imaging. We employed the same neural network structure (without the physical model) to fit the training set (10,000 human face images from Faces-LFW 32 ) to obtain a trained model for mapping intensity patterns to phase images (see Table S1 in the Supplementary Information for more details). The results are illustrated in Fig. 4. Again, we used the MSE to measure the quality of the reconstructed phase image in comparison to the ground truth shown in Fig. 4b, which is one of the test images. The MSE value between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is ...
Context 2
... (10,000 human face images from Faces-LFW 32 ) to obtain a trained model for mapping intensity patterns to phase images (see Table S1 in the Supplementary Information for more details). The results are illustrated in Fig. 4. Again, we used the MSE to measure the quality of the reconstructed phase image in comparison to the ground truth shown in Fig. 4b, which is one of the test images. The MSE value between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase ...
Context 3
... (see Table S1 in the Supplementary Information for more details). The results are illustrated in Fig. 4. Again, we used the MSE to measure the quality of the reconstructed phase image in comparison to the ground truth shown in Fig. 4b, which is one of the test images. The MSE value between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the ...
Context 4
... details). The results are illustrated in Fig. 4. Again, we used the MSE to measure the quality of the reconstructed phase image in comparison to the ground truth shown in Fig. 4b, which is one of the test images. The MSE value between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the conventional end-to-end approach (Fig. 4h) and the ground truth ...
Context 5
... phase image in comparison to the ground truth shown in Fig. 4b, which is one of the test images. The MSE value between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the conventional end-to-end approach (Fig. 4h) and the ground truth is 0.127 rad, whereas the corresponding error associated with PhysenNet ( Fig. 4i) is 0.025 rad, which ...
Context 6
... between the phase reconstructed from the diffraction pattern (Fig. 4a) using the pure end-to-end deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the conventional end-to-end approach (Fig. 4h) and the ground truth is 0.127 rad, whereas the corresponding error associated with PhysenNet ( Fig. 4i) is 0.025 rad, which is tenfold better. As expected, for the conventional end-to-end deep learning approach, the recovery quality decreases ...
Context 7
... deep learning approach (Fig. 4c) and the ground truth is 0.038 rad, whereas the corresponding value associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the conventional end-to-end approach (Fig. 4h) and the ground truth is 0.127 rad, whereas the corresponding error associated with PhysenNet ( Fig. 4i) is 0.025 rad, which is tenfold better. As expected, for the conventional end-to-end deep learning approach, the recovery quality decreases as the similarity between the test object and the training objects decreases. However, the ...
Context 8
... associated with PhysenNet ( Fig. 4d) is 0.033 rad. However, we observe that when the phase image is from another set, such as the cat face shown in Fig. 4g, the MSE between the phase reconstructed using the conventional end-to-end approach (Fig. 4h) and the ground truth is 0.127 rad, whereas the corresponding error associated with PhysenNet ( Fig. 4i) is 0.025 rad, which is tenfold better. As expected, for the conventional end-to-end deep learning approach, the recovery quality decreases as the similarity between the test object and the training objects decreases. However, the performance of PhysenNet is not similarly affected. We also performed simulations to compare PhysenNet with ...
Context 9
... R 1 θ is the deep neural network we used to generate the phase ϕ from the diffraction pattern I, λ is the RED regularization strength, and R 2 θ Ã is the pre-trained denoising model. The results are illustrated in Fig. 4e, j. The MSE values between these results and the ground-truth images are 0.039 and 0.068 rad, respectively. Now, we will present the experimental demonstration. The experimental apparatus is schematically shown in Fig. 5a. One can see that this is actually a single-beam lens-less imaging geometry. A laser beam emitted from a He-Ne laser at ...
Citations
... Here we describe a self-supervised machine learning algorithm called CoCoA, which stands for coordinate-based neural representations for computational adaptive optics, for joint estimation of wavefront aberration and three-dimensional structural recovery. Although self-supervised learning approaches have been previously used for denoising [19][20][21][22][23] , blind deconvolution 24 , two-dimensional (2D) phase imaging [25][26][27] and tomography [28][29][30][31] , here a self-supervised scheme is described for computational AO in fluorescence microscopy. CoCoA takes a three-dimensional (3D) aberrated image stack as input and returns the estimated aberration and underlying structures. ...
Widefield microscopy is widely used for non-invasive imaging of biological structures at subcellular resolution. When applied to a complex specimen, its image quality is degraded by sample-induced optical aberration. Adaptive optics can correct wavefront distortion and restore diffraction-limited resolution but require wavefront sensing and corrective devices, increasing system complexity and cost. Here we describe a self-supervised machine learning algorithm, CoCoA, that performs joint wavefront estimation and three-dimensional structural information extraction from a single-input three-dimensional image stack without the need for external training datasets. We implemented CoCoA for widefield imaging of mouse brain tissues and validated its performance with direct-wavefront-sensing-based adaptive optics. Importantly, we systematically explored and quantitatively characterized the limiting factors of CoCoA’s performance. Using CoCoA, we demonstrated in vivo widefield mouse brain imaging using machine learning-based adaptive optics. Incorporating coordinate-based neural representations and a forward physics model, the self-supervised scheme of CoCoA should be applicable to microscopy modalities in general.
... Once a deep learning architecture is defined, it can be represented by a mathematical function Z δ which is dependent on network parameters -weights and biases δ ä Δ [33]. The purpose of the training process is to solve the following equation (5) and arrive at an optimized set of weights and biases δ * ä Δ, resulting in function Z δ* that should accurately unwrap training dataset as well as unknown dataset. ...
A deep learning Hybrid architecture for phase unwrapping has been proposed. The hybrid architecture is based on integration of Convolutional Neural Networks (CNN) with Vision Transformer. The performance of Hybrid architecture/network in phase unwrapping is compared against CNN based standard UNET network. Structural Similarity Index (SSIM) and Root Mean Square Error (RMSE) have been used as performance metrics to assess the performance of these deep learning networks for phase unwrapping. To train and test the networks, dataset with high mean Entropy has been generated using Gaussian filtering of random noise in Fourier plane. The Hybrid architecture is tested on test dataset and is found to have superior performance metrics against the UNET network. Their performance is also tested in noisy environment with various noise levels and Hybrid architecture demonstrated better anti-noise capability than UNET network. Hybrid architecture was successfully validated in real world scenario using experimental data from custom built Digital Holographic Microscope. With the advent of newer architectures and hardware, Deep learning networks can further improve the performance in solving inverse problems.
... 44 But the generalization of these deep learning methods across different sample types and imaging system parameter settings remains limited. Self-supervised network methods 45,46 are also explored to bypass the poor generalizability of deep learning, achieving the better convergence performance such as less artifacts, higher contrast, and cleaner background, than traditional non-deep-learning methods. 31,34 However, the data acquisition manner of the self-supervised methods remains the same as traditional methods, which needs diverse measurements, and the reconstruction speed of large-scale and high-resolution imaging are even slower. ...
Lensless microscopy provides a wide field of view (FOV) determined by the image sensor size, allowing visualization of large sample areas. Coupled with advanced and even pixel super‐resolution phase retrieval algorithms, it can achieve resolutions up to the sub‐micron level, enabling both large‐FOV and high‐resolution imaging. However, high‐throughput lensless imaging encounters challenges in rapid data acquisition and large‐scale phase retrieval. Furthermore, when examining biological samples over a large FOV, focal plane inconsistencies often emerge among distinct regions. This study introduces a fast acquisition and efficient reconstruction method for coherent lensless imaging. Multiple measurements are manually modulated using an axial translation stage and sequentially captured by an image sensor, requiring no hardware synchronization. Optical parameter calibration, region‐wise auto‐focusing, and region‐wise phase retrieval algorithms are integrated to establish a general parallel computing framework for rapid, efficient, and high‐throughput lensless imaging. Experimental results demonstrate a 7.4 mm × 5.5 mm FOV and 1.38 µm half‐pitch resolution imaging of human skin and lung tumor sections with region‐wise focusing, requiring ≈0.5‐s acquisition time and 17‐s reconstruction time. By incorporating pixel super‐resolution, a 0.98 µm half‐pitch resolution is achieved in full‐FOV peripheral blood smears without additional data required, advantageous for discerning hollow shapes and segmenting blood cells.
... Deep learning has been redefining the leading edge in many areas of optics, including the inverse design of optics [14,15], optical microscopy [16][17][18][19], and holography [20][21][22][23]. Inspired by this, recent researches have also begun to employ deep neural networks, ranging from relatively simple network architectures to more complex structures such as convolutional neural networks (CNN) and residual neural networks (RNN), to recover images through diffusive mediums [24][25][26][27][28][29]. ...
The optical diffractive neural network (ODNN) offers the benefits of high-speed parallelism and low energy consumption. This kind of method holds great potential in the task of reconstructing diffusive images. In this work, we capture a double-scattering dataset by designing optical experiments and use it to evaluate the image reconstruction capability of the constructed ODNNs under more complex scattering scenarios. The Pearson Correlation Coefficient, which is used as a quantitative index of the reconstruction performance, shows that the constructed diffractive networks enable to achieve high performance in the direct recovery of double-scattering data, as well as in the recovery task of stitching images based on two different kinds of double-scattering data. Meanwhile, due to the high redundancy of valid information in the speckle patterns of scattering images, even if parts of the information in the speckle patterns are blocked, the constructed diffractive networks can also show high reconstruction performance without retraining. The capability of the proposed ODNN to reconstruct double-scattering images indicates that the optical diffractive network has the potential to bring transformative applications in more complex scattering scenarios.
... Recently, fueled by the advances made in deep learning, the application of deep neural networks has been adopted for accurate and rapid reconstruction of phase information in complex fields through a single feed-forward operation [22][23][24][25][26][27][28][29] . While these deep learning-based approaches offer considerable benefits, they typically demand intensive computational resources for network inference, requiring the use of graphics processing units (GPUs). ...
Complex field imaging, which captures both the amplitude and phase information of input optical fields or objects, can offer rich structural insights into samples, such as their absorption and refractive index distributions. However, conventional image sensors are intensity-based and inherently lack the capability to directly measure the phase distribution of a field. This limitation can be overcome using interferometric or holographic methods, often supplemented by iterative phase retrieval algorithms, leading to a considerable increase in hardware complexity and computational demand. Here, we present a complex field imager design that enables snapshot imaging of both the amplitude and quantitative phase information of input fields using an intensity-based sensor array without any digital processing. Our design utilizes successive deep learning-optimized diffractive surfaces that are structured to collectively modulate the input complex field, forming two independent imaging channels that perform amplitude-to-amplitude and phase-to-intensity transformations between the input and output planes within a compact optical design, axially spanning ~100 wavelengths. The intensity distributions of the output fields at these two channels on the sensor plane directly correspond to the amplitude and quantitative phase profiles of the input complex field, eliminating the need for any digital image reconstruction algorithms. We experimentally validated the efficacy of our complex field diffractive imager designs through 3D-printed prototypes operating at the terahertz spectrum, with the output amplitude and phase channel images closely aligning with our numerical simulations. We envision that this complex field imager will have various applications in security, biomedical imaging, sensing and material science, among others.
... Recently there has been a notable upsurge in the utilization of deep learning methods for image-based wavefront sensing. Of significant interest are the approaches that incorporate supervised [3,12,13] or self-supervised approaches [14][15][16] for effective phase retrieval. Supervised learning employs labeled datasets to train algorithms for prediction. ...
... [18]. In contrast, self-supervised net [14,16] eliminates the need for labelled or experimental training data. Considering pixel-wise output is not suitable for wavefront retrieval for it lack of inter-pixel connections, we can integrate Zernike representations into self-supervised net. ...
Wavefront aberration describes the deviation of a wavefront in an imaging system from a desired perfect shape, such as a plane or a sphere, which may be caused by a variety of factors, such as imperfections in optical equipment, atmospheric turbulence, and the physical properties of imaging subjects and medium. Measuring the wavefront aberration of an imaging system is a crucial part of modern optics and optical engineering, with a variety of applications such as adaptive optics, optical testing, microscopy, laser system design, and ophthalmology. While there are dedicated wavefront sensors that aim to measure the phase of light, they often exhibit some drawbacks, such as higher cost and limited spatial resolution compared to regular intensity measurement. In this paper, we introduce a lightweight and practical learning-based method, named LWNet, to recover the wavefront aberration for an imaging system from a single intensity measurement. Specifically, LWNet takes a measured point spread function (PSF) as input and recovers the wavefront aberration with a two-stage network. The first stage network estimates an initial wavefront aberration via supervised learning, and the second stage network further optimizes the wavefront aberration via self-supervised learning by enforcing the statistical priors and physical constraints of wavefront aberrations via Zernike decomposition. For supervised learning, we created a synthetic PSF-wavefront aberration dataset via ray tracing of 88 lenses. Experimental results show that even trained with simulated data, LWNet works well for wavefront aberration estimation of real imaging systems and consistently outperforms prior learning-based methods.
... Yao et al. [40] proposed a single-pixel classification method with deep learning for fast-moving objects and obtained feature information for classification with S R = 3.8%. However, these data-driven networks suffer from problems such as generalizability and interpretability, which may prohibit their practical applications [41][42][43]. ...
We propose and demonstrate a single-pixel imaging method based on deep learning network enhanced singular value decomposition. The theoretical framework and the experimental implementation are elaborated and compared with the conventional methods based on Hadamard patterns or deep convolutional autoencoder network. Simulation and experimental results show that the proposed approach is capable of reconstructing images with better quality especially under a low sampling ratio down to 3.12%, or with fewer measurements or shorter acquisition time if the image quality is given. We further demonstrate that it has better anti-noise performance by introducing noises in the SPI systems, and we show that it has better generalizability by applying the systems to targets outside the training dataset. We expect that the developed method will find potential applications based on single-pixel imaging beyond the visible regime.
... The setup emits red light at 617 nm, and according to the simulation, MPs can be detected based on their interaction with this red-light source. Zhu et al. (2021) put forward an image-descattering method based on polarization imaging combined with an untrained neural network, like the approach used in the study by Wang et al. (2020). The use of an untrained neural network, which relies on predefined principles, was chosen to address the unsatisfactory imaging effect in the image descattering process. ...
... Recent studies have revealed that even without datasets, the convolutional neural network (CNN) structure itself has some regularization ability to capture a large number of statistical priors of low-level images, which is called deep image prior (DIP) [25]. DIP employs random noise as the input and learns appropriate network parameters from degraded images, which has been proven to be an effective tool to solve the reconstruction problems in spectral imaging [26], SIM [27], low light imaging [28], coherent phase imaging [29] and other computational imaging technologies. Different from those data-driven DL algorithms, inspired by non-data-driven DL approaches [30] such as DIP [25] and MMES [31], and in order to solve the above image mismatch problem, a new approach is proposed. ...
Compressed ultrafast photography (CUP) is a computational imaging technology capable of capturing transient scenes in picosecond scale with a sequence depth of hundreds of frames. Since the inverse problem of CUP is an ill-posed problem, it is challenging to further improve the reconstruction quality under the condition of high noise level and compression ratio. In addition, there are many articles adding an external charge-coupled device (CCD) camera to the CUP system to form the time-unsheared view because the added constraint can improve the reconstruction quality of images. However, since the images are collected by different cameras, slight affine transformation may have great impacts on the reconstruction quality. Here, we propose an algorithm that combines the time-unsheared image constraint CUP system with unsupervised neural networks. Image registration network is also introduced into the network framework to learn the affine transformation parameters of input images. The proposed algorithm effectively utilizes the implicit image prior in the neural network as well as the extra hardware prior information brought by the time-unsheared view. Combined with image registration network, this joint learning model enables our proposed algorithm to further improve the quality of reconstructed images without training datasets. The simulation and experiment results demonstrate the application prospect of our algorithm in ultrafast event capture.
... Deep learning has emerged as a promising approach for solving inverse problems encountered in computational imaging [2]. Groundbreaking studies have successfully demonstrated the effectiveness of deep learning for applications including optical tomography [3], 3D image reconstruction [4,5], phase retrieval [6,7], computational ghost imaging [8], digital holography [9][10][11], imaging through scattering media [12], fluorescence lifetime imaging under low-light conditions [13], unwrapping [14,15], and fringe analysis [16]. The deep learning-based artificial neural networks employed in computational imaging typically rely on a substantial collection of labeled data to optimize their weight and bias parameters through a training process [17]. ...
... The network's weights are then repeatedly updated using a loss function that compares the generated image with the input data, such as a noisy image. This approach has demonstrated remarkable effectiveness in simulated image denoising [23], deblurring [24], phase retrieval [6,19], and super-resolution tasks [25]. ...
... (4), using the estimated phaseθ (x , y ), and comparing against the measured far-field intensity I z (x , y ). The neural network is trained by minimizing the mean square error (MSE) between the measured and estimated intensities, ||I z (x , y ) −Ĩ z (x , y )||, following the approach introduced in Ref. [6]. This minimization of the cost function via gradient descent allows for the gradual refinement of the estimated phase until a desired accuracy, as measured by the difference between subsequent iterations, is obtained. ...
Conventional deep learning-based image reconstruction methods require a large amount of training data, which can be hard to obtain in practice. Untrained deep learning methods overcome this limitation by training a network to invert a physical model of the image formation process. Here we present a novel, to our knowledge, untrained Res-U2Net model for phase retrieval. We use the extracted phase information to determine changes in an object’s surface and generate a mesh representation of its 3D structure. We compare the performance of Res-U2Net phase retrieval against UNet and U2Net using images from the GDXRAY dataset.
