Figure 1 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Source publication
Flat-field correction (FFC) is commonly used in image signal processing (ISP) to improve the uniformity of image sensor pixels. Image sensor nonuniformity and lens system characteristics have been known to be temperature-dependent. Some machine vision applications, such as visual odometry and single-pixel airborne object tracking, are extremely sen...
Contexts in source publication
Context 1
... FPN Noise Reduction in CMOS Sensors Figure 1 shows the typical structure of a CMOS image sensor. A matrix of pixels can collect electrons generated by the photoelectric effect. ...
Context 2
... a first approximation a single prior, ˆ d x,y , was used without adjusting the magnitude (G d ) to cancel the DSNU across the entire temperature and analog gain range. Figure 10 presents the SD of the residual DSNU when corrected with a static reference image captured at 45 • C and 18.0 dB. At these parameter values, the SD of the DSNU was 28.64 (Table 2), about half of the worst-case SD. ...
Context 3
... correlation between a scaled reference frame and an actual dark input frame remained unaffected by scaling with (G d ). Equation (16) establishes the theoretical background for the exponential relation with temperature, while the definition, α = 20log 10 A g (dB) of analog gain consequently results in α, used by the programmable gain amplifiers in the sensor to scale exponentially with the A g factor as shown in Figure 11a. By fitting exponentials along the axes (Figure 11) and modeling DSNU as a product of an FPN template, with the magnitude approximated with a separable surface yielded: ...
Context 4
... (16) establishes the theoretical background for the exponential relation with temperature, while the definition, α = 20log 10 A g (dB) of analog gain consequently results in α, used by the programmable gain amplifiers in the sensor to scale exponentially with the A g factor as shown in Figure 11a. By fitting exponentials along the axes (Figure 11) and modeling DSNU as a product of an FPN template, with the magnitude approximated with a separable surface yielded: ...
Context 5
... the high correlation of FPN patterns for different parameters (T, α), FPN suppression could be significantly improved by scaling the reference DSNU image captured using the parametric approximation model of Equation (20). Figure 12 presents the standard deviation of the residual DSNU after correction with a single reference that was scaled by ...
Context 6
... intuitive selection ensured that for any parameter combination p in parameter space ¯ p, a set of three references could be selected such that p was inside the triangle defined by the references. Figure 13 shows that interpolating the reference image produced very good results, significantly reducing the worst-case DSNU. ...
Context 7
... suppression quality can be also improved by optimizing the reference parameters for a given number of reference frames. Figure 14 Table 3 lists the Pearson correlation values between the actual frames and the interpolated references. Besides improved DSNU suppression measurements, notice the improvement in the correlation with respect to the correlation values in Table 2. Every time the analog gain or measured die temperature deltas exceed a predefined threshold, the embedded processor controlling the ISP needs to recompute the interpolated reference imagê d. ...
Context 8
... when comparing FF images captured with different analog gain settings resulting from similar output white levels, the SD is expected to scale with the square root of the gain applied. Figure 15, plotting the measured standard deviations E[σ 1 (T, α)] on a lin-log scale, confirms this expectation. The 24.0 dB (maximum gain for the IMX265 and IMX273) applied a factor of 16 amplification, for which a 4× increase in shot noise was observed. ...
Context 9
... is also worth noting that the distribution of per-pixel standard deviation σ 1 (α, T) is not necessarily Gaussian (Figure 16). Per the central limit theorem, the effects of multiple, uncorrelated noise sources with different means and SDs (such as shot noise, thermal noise, and electronic noise) superimposed on pixel outputs would present as a single Gaussian even if the distributions of the individual noise sources were not Gaussian. ...
Context 10
... sensitivities translate to different photon counts and in turn, different SD distributions of shot-noise. For a monochrome sensor, with no lens assembly attached, Figure 16 is proof of a continuum of different sensitivities, which effectively is the definition of the PRNU present. To analyze noise variance on the pixel output, all temporal noise sources dependent on the illumination and gain, such as shot noise, were incorporated into n s (L, α) and all other temporal noise sources, such as reset, electronic, thermal noise into n T (α, T). ...
Context 11
... measurements of σ s (α, T) and σ d (α, T), the SD of the fixed-pattern component (PRNU) could be deduced: Figure 18 illustrates the dependence of the PRNU on the analog gain and temperature. It is also worth noting that the nonuniformity was only visible, though subtle, at very low gains where shot noise was at minimum. ...
Context 12
... a single correction image (Figure 19) is the simplest way to calibrate nonuniformity and has demonstrated good results (Yao, [27]). For suppression of the visible PRNU artifacts, selecting a calibration frame in the middle of the temperature range and at analog gain α = 0.0 dB removed all visible artifacts. ...
Context 13
... results suggested that using multiple reference frames along the gain axis in the middle of the temperature range could minimize the SD over the entire parameter range ( Figure 21). Even though a multipoint correction can reduce the worst-case PRNU by a factor of four, a practical implementation of this method requires capturing multiple PRNU reference images in a temperature-controlled environment. ...
Context 14
... the introduction, we reviewed sources of nonuniformity in imaging systems: DSNU, PRNU and lens shading. In the subsequent Results section, we analyzed the dependency of the DSNU and PRNU of two modern, global shutter machine vision sensors on exposure time, die temperature, and analog gain (Figures 9 and 18). In Section 5.1, we also provided an FFC architecture for ISPs, optimized for FPGA or ASIC implementation supporting different FPN suppression performance and resource use trade-offs. ...
Citations
... Gaussian white noise is often treated as additive noise in denoising processes. Traditional denoising techniques such as BM3D [5], bilateral filtering, and Gabor utilize linear interpolation between multiple reference images for correction [6]. ...
In the imaging process of Complementary Metal Oxide Semiconductor (CMOS) image sensor, noise is inevitably introduced at various stages. This effect is particularly serious when detecting weak signals, such as ultraviolet light. Through theoretical analysis of CMOS noise, we deduce that the main noise under illumination is non-uniformity noise after averaging the images to eliminate granular noise. Existing image denoising methods mainly rely on simulated noise, which perform poorly when applied to real detectors. To address this issue, we establish a novel CMOS image dataset. Initially, we obtain the non-uniformity noise blocks from the CMOS imaging system designed in this paper. Then, we utilize a GAN network to augment the noise data. Next, we randomly combine the noise blocks with high-quality images from the DF2K dataset to form paired image datasets. In recent years, the development of various deep learning algorithms has significantly improved the effectiveness of image noise reduction compared to traditional algorithms. This paper combines convolutional neural network and proposes the MSR-Net denoising algorithm, which is based on the U-Net network and incorporates the Res2Net module as its main network structure. It provides features with different scale receptive fields and enhances image details. Additionally, to more accurately reflect the visual perceptual effects of the images, we propose a novel image evaluation metric, Uniform pixel outliers (UPO), making the image evaluation more adequate. Experiments were conducted on our proposed image dataset, results indicate that compared with similar noise reduction algorithms, this method performs better in both qualitative and quantitative aspects, effectively suppressing noise dominated by non-uniform noise.
... Researchers have enhanced the visual reasoning ability by introducing semantic features [56], and improving the understanding of images can likewise generate more accurate questions. The question-and-answer-based interactive system can also be deployed on robots [57,58] to respond to complex and changing environments by increasing the flexibility of decision-making. ...
Auxiliary clinical diagnosis has been researched to solve unevenly and insufficiently distributed clinical resources. However, auxiliary diagnosis is still dominated by human physicians, and how to make intelligent systems more involved in the diagnosis process is gradually becoming a concern. An interactive automated clinical diagnosis with a question-answering system and a question generation system can capture a patient’s conditions from multiple perspectives with less physician involvement by asking different questions to drive and guide the diagnosis. This clinical diagnosis process requires diverse information to evaluate a patient from different perspectives to obtain an accurate diagnosis. Recently proposed medical question generation systems have not considered diversity. Thus, we propose a diversity learning-based visual question generation model using a multi-latent space to generate informative question sets from medical images. The proposed method generates various questions by embedding visual and language information in different latent spaces, whose diversity is trained by our newly proposed loss. We have also added control over the categories of generated questions, making the generated questions directional. Furthermore, we use a new metric named similarity to accurately evaluate the proposed model’s performance. The experimental results on the Slake and VQA-RAD datasets demonstrate that the proposed method can generate questions with diverse information. Our model works with an answering model for interactive automated clinical diagnosis and generates datasets to replace the process of annotation that incurs huge labor costs.
White balance (WB) is a critical back-end processing function in image sensor to keep color constancy under various lighting conditions by adjusting RGB color channel gain values. In this work, we proposed a novel gate-controlled gain tuning method for Fully-Depleted Silicon-on-Insulator (FDSOI) one-transistor (1T) pixel to achieve WB inside sensor. Based on the pixel structure of p-well under BOX for sensing and nMOSFET on the top for readout, the 1T pixel output gain is modulated by MOSFET gate according to transistor transfer characteristic. About 5x gain modulation range in RGB spectrums photoresponse (nonlinearity<3%) is experimentally demonstrated in the devices fabricated by 22nm FDSOI-based technology. The scheme for in-sensor WB demonstration is provided with a novel 1T pixel array design, and the evaluation result shows in-sensor WB achieving almost equivalent performance (Delta-E deviation<1) compared with using conventional back-end WB.
