Fig 5 - uploaded by Daniel Keren
Content may be subject to copyright.
Vending machine (same image as in Fig. 1) restored using local MAP algorithm ( left ) and processing in the angle domain ( right )
Source publication
In some capturing devices, such as digital cam- eras, there is only one color sensor at each pixel. Usually, 50% of the pixels have only a green sensor, 25% only a red sensor, and 25% only a blue sensor. The problem is then to restore the two missing colors at each pixel - this is called "demosaicing", because the original samples are usu- ally arr...
Similar publications
![Examples where Liu et al. [17] de-fencing approach fails in fence...](profile/Muhammad-Shahid-Farid/publication/296626086/figure/fig3/AS:962660745101317@1606527615958/Examples-where-Liu-et-al17-de-fencing-approach-fails-in-fence-detection-a-test-image_Q320.jpg)
![Examples where Liu et al. [17] de-fencing approach fails in fence...](profile/Muhammad-Shahid-Farid/publication/296626086/figure/fig4/AS:962660749283331@1606527616120/Examples-where-Liu-et-al17-de-fencing-approach-fails-in-fence-detection-or-in-removal_Q320.jpg)
Detection and removal of fences from digital images become essential when an important part of the scene turns to be occluded by such unwanted structures. Image de-fencing is challenging because manually marking fence boundaries is tedious and time-consuming. In this paper, a novel image de-fencing algorithm that effectively detects and removes fen...
Citations
... Demosaicing of Bayer Color Filter Array (CFA) has been extensively studied for several decades [1], [2], [6]. Various demosaicing approaches are exploited, such as color difference based interpolation [24], [25], edge directional interpolation [26], frequency domain filtering [3], [4], [5], and reconstruction methods [27], [28]. However, when it comes to new patterns, such as Quad Bayer depicted in Fig. 3, there are only a few works that can be applied to it. ...
Can a mobile camera see better through display? Under Display Camera (UDC) is the most awaited feature in mobile market in 2020 enabling more preferable user experience, however, there are technological obstacles to obtain acceptable UDC image quality. Mobile OLED panels are struggling to reach beyond 20% of light transmittance, leading to challenging capture conditions. To improve light sensitivity, some solutions use binned output losing spatial resolution. Optical diffraction of light in a panel induces contrast degradation and various visual artifacts including image ghosts, yellowish tint etc. Standard approach to address image quality issues is to improve blocks in the imaging pipeline including Image Signal Processor (ISP) and deblur block. In this work, we propose a novel approach to improve UDC image quality - we replace all blocks in UDC pipeline with all-in-one network – UDC d^Net. Proposed solution can deblur and reconstruct full resolution image directly from non-Bayer raw image, e.g. Quad Bayer, without requiring remosaic algorithm that rearranges non-Bayer to Bayer. Proposed network has a very large receptive field and can easily deal with large-scale visual artifacts including color moiré and ghosts. Experiments show significant improvement in image quality vs conventional pipeline – over 4dB in PSNR on popular benchmark - Kodak dataset.
... This makes them suitable for a moving UAV. However these sensors suffer from a detrimental effect called crosstalk (Hirakawa, 2008), which means that distinct spectral channels also receive some response of other spectral bands. These sensors also sacrifice spatial resolution to gain spectral resolution (Keren and Osadchy, 1999). These effects become increasingly detrimental as the mosaic sizes increase and physical-pixel sizes decrease. ...
... The usage of a 16-band light-weight hyperspectral camera system with a Multispectral Color Filter Array (MCFA) , mounted on a UAV, has been proposed in this dissertation. The hyperspectral images produced by this system suffer from crosstalk and their spatial resolution is reduced sixteen fold to accommodate for the increase in spectral resolution (Keren and Osadchy, 1999). Several methods for demosaicking images have been proposed in literature: using edge information, neural networks, inpainting and linear models (Monno et al., 2012;Wang, 2014;Wang et al., 2017;Aggarwal and Majumdar, 2014). ...
Recently there has been much interest in hyperspectral imaging research and applications. Hyperspectral cameras collect image data from across the electromagnetic spectrum. These cameras aim to capture a spectrogram for each pixel to form a hyperspectral cube. The application area for these types of cameras is broad and varies from vegetation inspection to chemical finger printing.
Small battery-powered multi-copters called Unmanned Aerial Vehicles (UAVs) have recently shown great potential for a large number of applications. A camera mounted on a UAV is used for the inspection of large objects or large areas. For example, in the field of precision agriculture they are used for measuring Chlorophyll to determine crop health. Furthermore, UAVs are used for the inspection of wind turbines, to take images af cracks, pinholes and other defects.
UAV applications could benefit more from hyperspectral imaging technology, but these devices have intrinsic limitations that makes using them in conjunction with each other challenging. This is mainly caused by the sensitivity to movement or low spatial resolution of the hyperspectral devices combined with the limited payload capabilities of UAVs.
Deep learning or, more specifically, Convolutional Neural Networks (CNNs) have shown to obtain state-of-the art performance in a multitude of research fields. Many applications already benefit from deep learning. Usually these deep learning models are trained directly from data in an end-to-end fashion. This dissertation revolves around the question if algorithms from the field of deep learning can mitigate the difficulties that are caused by the limitations encountered in combining hyperspectralimaging and UAVs.
This trinity of technologies: deep learning, hyperspectral imaging and UAVs, serves as a framework within which this research is defined.
... In the literature, various regularization approaches to demosaicking have been explored [4,14,22,27,33]. They use spatial smoothness and color correlation terms in a cost function that is minimized iteratively. ...
In this article, we introduce a variational demosaicking model that restores full-color images from sampled data acquired with a RGBW color filter array (CFA). The proposed model employs the concept of the low-dimensional patch manifold model (LDMM) in Shi et al. (J Sci Comput 75(2):638–656, 2018) and inter-channel correlation terms. The LDMM enables regions without color data to be filled out smoothly from given sparse data, while conserving textures. Moreover, the inter-correlation terms defined in the gradient domain help diminish color artifacts in demosaicked images. We also present an efficient iterative algorithm for solving the proposed model. Numerical experiments validate the effectiveness of our model for demosaicking images acquired with RGBW CFAs as compared to the state-of-the-art models.
... This makes them suitable for a moving UAV. However these sensors suffer from a detrimental effect called crosstalk [14], which means that distinct spectral channels also receive some response of other spectral bands. These sensors also sacrifice spatial resolution to gain spectral resolution [15]. These effects become increasingly detrimental as the mosaic sizes increase and physical-pixel sizes decrease. ...
... MC hc (M) = C (15) where M is the input mosaic image with a subscript indicating the 2-d pixel coordinate, the 3-d coordinate in the hyperspectral cube C is denoted by x, y and z. The size of the mosaic is denoted by s which is 4, for a 4 × 4 mosaic. ...
Precision agriculture using unmanned aerial vehicles (UAVs) is gaining popularity. These UAVs provide a unique aerial perspective suitable for inspecting agricultural fields. With the use of hyperspectral cameras, complex inspection tasks are being automated. Payload constraints of UAVs require low weight and small hyperspectral cameras; however, such cameras with a multispectral color filter array suffer from crosstalk and a low spatial resolution. The research described in this paper aims to reduce crosstalk and to increase spatial resolution using convolutional neural networks. We propose a similarity maximization framework which is trained to perform end-to-end demosaicking and crosstalk-correction of a \(4 \times 4\) raw mosaic image. The proposed method produces a hyperspectral image cube with 16 times the spatial resolution of the original cube while retaining a median structural similarity (SSIM) index of 0.85 (compared to an SSIM of 0.55 when using bilinear interpolation). Furthermore, this paper provides insight into the beneficial effects of crosstalk for hyperspectral demosaicking and gives best practices for several architectural and hyperparameter variations as well as a theoretical reasoning behind certain observations.
... Such ideas have been exploited in traditional image fusion type super-resolution techniques [118], yet sparsity-based single image super resolution lacks a concrete color super resolution framework. Edge similarities across RGB color channels may be enforced in the following manner [118,133,134]. ...
... Experimental results show the merits of our proposed method both visually and quantitatively. While our work offers one possible way to capture cross-channel color constraints, chrominance geometry can be captured via alternative quantitative formulations as in [52,118,124,133]. Incorporating these as constraints or regularizers in a sparsity based color SR framework forms a viable direction for future work. ...
... For image restoration, we used a color prior for super-resolution task. While our multi-channel super-resolution work incorporated signal priors to capture cross-channel color constraints, chrominance geometry can be captured via many different ways as is suggested in [52,118,124,133]. Incorporating these as constraints or regularizers in a sparsity-based color SR framework forms a viable direction for future work. ...
Parsimony in signal representation is a topic of active research. Sparse signal processing and representation is the outcome of this line of research which has many applications in information processing and has shown significant improvement in real-world applications such as recovery, classification, clustering, super resolution, etc. This vast influence of sparse signal processing in real-world problems raises a significant need in developing novel sparse signal representation algorithms to obtain more robust systems. In such algorithms, a few open challenges remain in (a) efficiently posing sparsity on signals that can capture the structure of underlying signal and (b) the design of tractable algorithms that can recover signals under aforementioned sparse models.
... Such ideas have been exploited in traditional image fusion type super-resolution techniques [30], yet sparsity-based single image super resolution lacks a concrete color super resolution framework. Edge similarities across RGB color channels may be enforced in the following manner [30], [45], [46]. ...
... Experimental results show the merits of our proposed method both visually and quantitatively. While our work offers one possible way to capture cross-channel color constraints, chrominance geometry can be captured via alternative quantitative formulations as in [30], [36], [44], [45]. Incorporating these as constraints or regularizers in a sparsity based color SR framework forms a viable direction for future work. ...
Sparsity constrained single image super-resolution (SR) has been of much recent interest. A typical approach involves sparsely representing patches in a low-resolution (LR) input image via a dictionary of example LR patches, and then using the coefficients of this representation to generate the high-resolution (HR) output via an analogous HR dictionary. However, most existing sparse representation methods for super resolution focus on the luminance channel information and do not capture interactions between color channels. In this work, we extend sparsity based super-resolution to multiple color channels by taking color information into account. Edge similarities amongst RGB color bands are exploited as cross channel correlation constraints. These additional constraints lead to a new optimization problem which is not easily solvable; however, a tractable solution is proposed to solve it efficiently. Moreover, to fully exploit the complementary information among color channels, a dictionary learning method is also proposed specifically to learn color dictionaries that encourage edge similarities. Merits of the proposed method over state of the art are demonstrated both visually and quantitatively using image quality metrics.
... Regularization is also widely used. Keren and Osadchy [55] proposed a regularizer including a spatial smoothness and a color correlation term. Saito and Komatsu [56] used the total variation to impose smoothness on intra-channel and color differences. ...
The Penrose pixel layout, an aperiodic pixel layout in rhombus Penrose tiling, has been shown to substantially outperform the existing square pixel layout in super-resolution. However, it was only tested on grayscale images. To study its performance on color images, we have to reconstruct regular color images from Penrose raw images, i.e., images with only one color component at each Penrose pixel, resulting in the problem of demosaicking from Penrose pixels. Penrose demosaicking is more difficult than regular demosaicking because none of the color components of the reconstructed regular color images are available. So most of the traditional demosaicking methods do not apply. We develop a sparse representation based method for Penrose demosaicking. Extensive experiments show that Penrose pixel layout outperforms regular pixel layouts in terms of both perceptual evaluation and S-CIELAB. The Penrose pixel layout is unique among all irregular layouts because it is uniformly three-colorable and it has only two pixel shapes, thick and thin rhombi, making its manufacturing relatively easy.
... In search of general MSFA demosaicing techniques, Wang et al. first extended vector median filtering demosaicing [81] to the multispectral domain [82]. The nature of the vector median filter ensures that the results of filtering are derived from input vectors, namely the filtering does not introduce new values to the vectors, but only interpolates a missing value at one band with another value in the vicinity at the same or another band. ...
Thanks to technical progress in interferential filter design, we can finally implement in practice the concept of Multispectral Filter Array based sensors. This article presents the characteristics of the elements of our sensor as a case study. The spectral characteristics are based on two different spatial arrangements that distribute eight different bandpass filters in the visible and near-infrared area of the spectrum. We demonstrate that the system is viable and evaluate its performance through sensor spectral simulation and characterization.
... In search of general MSFA demosaicing techniques, Wang et al. first extended vector median filtering demosaicing [81] to the multispectral domain [82]. The nature of the vector median filter ensures that the results of filtering are derived from input vectors, namely the filtering does not introduce new values to the vectors, but only interpolates a missing value at one band with another value in the vicinity at the same or another band. ...
Thanks to some technical progress in interferencefilter design based on different technologies, we can finally successfully implement the concept of multispectral filter array-based sensors. This article provides the relevant state-of-the-art for multispectral imaging systems and presents the characteristics of the elements of our multispectral sensor as a case study. The spectral characteristics are based on two different spatial arrangements that distribute eight different bandpass filters in the visible and near-infrared area of the spectrum. We demonstrate that the system is viable and evaluate its performance through sensor spectral simulation.
... (red) and (blue) color channels, which share the other half pixels in the Bayer CFA structure, are considered as the chrominance signals. In order to recover the full color image from the input mosaic image, the demosaicing process is used to estimate the other two color channels for each pixel [15], [20]. Bilinear interpolation (BI) [38] is the simplest demosaicing algorithm in which the unknown two color channels of each pixel are obtained by averaging its proper adjacent pixels. ...
Without demosaicing processing, this paper first proposes a new approach to extract more accurate gradient/edge information on mosaic images directly. Next, based on spectral-spatial correlation, a novel adaptive heterogeneity-projection with proper mask size for each pixel is presented. Combining the extracted gradient/edge information and the adaptive heterogeneity-projection values, a new edge-sensing demosaicing algorithm is presented. Based on 24 popular testing images, experimental results demonstrated that our proposed high-quality demosaicing algorithm has the best image quality performance when compared with several recently published algorithms.
