Fig 1 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
A camera and its components: the Lens, and the Camera Body composed of Bayer Filter, Image Sensor, and Image Signal Processor.
Source publication
RGB cameras are one of the most relevant sensors for autonomous driving applications. It is undeniable that failures of vehicle cameras may compromise the autonomous driving task, possibly leading to unsafe behaviors when images that are subsequently processed by the driving system are altered. To support the definition of safe and robust vehicle a...
Context in source publication
Similar publications
Despite recent advances in autonomous driving systems, accidents such as the fatal Uber crash in 2018 show these systems are still susceptible to edge cases. Such systems must be thoroughly tested and validated before being deployed in the real world to avoid such events. Testing in open-world scenarios can be difficult, time-consuming, and expensi...
As shown by recent studies, machine intelligence-enabled systems are vulnerable to test cases resulting from either adversarial manipulation or natural distribution shifts. This has raised great concerns about deploying machine learning algorithms for real-world applications, especially in the safety-critical domains such as autonomous driving (AD)...
The preview of the road surface states is essential for improving the safety and the ride comfort of autonomous vehicles. The created dataset in this data article consists of 370151 road surface images captured under a wide range of road and weather conditions in China. The original pictures are acquired with a vehicle-mounted camera and then the p...
Citations
... The failures occurring in an automated vehicle (AV) for example, can lead to safety failures resulting in crashes and in some cases, fatalities. Currently, to the best of authors' knowledge, there are no rigorous methods for generating camera based sensor failures (Ceccarelli and Secci, 2022). ...
... In this paper, we will be focusing on the sensor failure occurring due to broken lens, although the process detailed in this paper can be used for any of the camera failures listed in (Ceccarelli and Secci, 2022). Broken or cracked lens in a camera can be caused due to an external object hitting the lens or a result of heat and/or pressure developing suddenly within the camera system. ...
While there has been extensive work on generating physics-based adversarial samples recently, an overlooked class of such samples come from physical failures in the camera. Camera failures can occur as a result of an external physical process, i.e. breakdown of a component due to stress, or an internal component failure. In this work, we develop a simulated physical process for generating broken lens as a class of physics-based adversarial samples. We create a stress-based physical simulation by generating particles constrained in a mesh and apply stress at a random point and at a random angle. We perform stress propagation through the mesh and the end result of the mesh is a corresponding image which simulates the broken lens pattern. We also develop a neural emulator which learns the non-linear mapping between the mesh as a graph and the stress propagation using constrained propagation setup. We can then statistically compare the difference between the generated adversarial samples with real, simulated and emulated adversarial examples using the detection failure rate of the different classes and in between the samples using the Frechet Inception distance. Our goal through this work is to provide a robust physics based process for generating adversarial samples.
... While traditional datasets cover normal driving situations, our focus is on "corner-case" scenarios -uncommon and challenging situations that rarely occur but pose significant challenges or anomalies to the perception system of an autonomous vehicle [1]. These scenarios involve various factors, including environmental factors such as weather or light conditions [28], object variability with unusual characteristics like pedestrians in disguises or partially obstructed objects [27,26], unpredictable situations requiring effective perception system responses [22], scenarios exploiting sensor limitations such as poor visibility or noise [25], and failure modes related to sensor failures, system malfunctions, or degraded performance [6]. ...
Autonomous driving development requires rigorous testing in real-world scenarios, including adverse weather, unpredictable events, object variations, and sensor limitations. However, these challenging "corner cases" are elusive in conventional datasets due to their unpredictability, high costs, and inherent risks. Recognizing the critical role of ground truth data in autonomous driving , the demand for synthetic data becomes evident. Contemporary machine learning-based algorithms essential to autonomous vehicles heavily depend on labeled data for training and validation. Simulation of scenarios not only mitigates the scarcity of real-world data but also facilitates controlled experimentation in situations that are challenging to replicate physically. The challenge extends beyond data scarcity, encompassing the impediment posed by the inability to systematically control and manipulate specific scenarios, hindering progress. To overcome these challenges, we present CornerSim, a dynamic virtualization framework simplifying the creation and modification of diverse driving scenarios. Leveraging simulation, CornerSim generates synthetic environments for comprehensive testing, providing essential outputs like raw sensor data (cameras, LiDAR, etc.) and labeled data (object detection bounding boxes, classes, semantic segmentation). The unpredictable nature of real-world corner cases complicates obtaining a sufficiently large and diverse annotated dataset. CornerSim addresses this challenge by not only generating synthetic data but also supplying necessary ground truth for training and evaluating machine learning models. This paper emphasizes the introduction of CornerSim and its ability to challenges related to testing autonomous vehicles in realistic scenarios. It focuses on the framework's capabilities, design principles, and integration, with the goal of enhancing thorough testing and validation of autonomous driving systems in a simulated environment, improving their robustness and safety. Our approach involves running simulations to generate datasets, which are statistically studied and compared with real data. Furthermore, we apply state-of-the-art detection algorithms to assess if data generated by CornerSim is suitable for both training and validation stages.
... For pure data generation, one of the most important points to consider is the degradation of the optical equipment. External problems such as dirt on lenses can be overcome relatively easily, but internal damages, for example in the image stabilization, can lead to poor data quality and is complicated to fix [55]. ...
Smart forestry, an innovative approach leveraging artificial intelligence (AI), aims to enhance forest management while minimizing the environmental impact. The efficacy of AI in this domain is contingent upon the availability of extensive, high-quality data, underscoring the pivotal role of sensor-based data acquisition in the digital transformation of forestry. However, the complexity and challenging conditions of forest environments often impede data collection efforts. Achieving the full potential of smart forestry necessitates a comprehensive integration of sensor technologies throughout the process chain, ensuring the production of standardized, high-quality data essential for AI applications. This paper highlights the symbiotic relationship between human expertise and the digital transformation in forestry, particularly under challenging conditions. We emphasize the human-in-the-loop approach, which allows experts to directly influence data generation, enhancing adaptability and effectiveness in diverse scenarios. A critical aspect of this integration is the deployment of autonomous robotic systems in forests, functioning both as data collectors and processing hubs. These systems are instrumental in facilitating sensor integration and generating substantial volumes of quality data. We present our universal sensor platform, detailing our experiences and the critical importance of the initial phase in digital transformation—the generation of comprehensive, high-quality data. The selection of appropriate sensors is a key factor in this process, and our findings underscore its significance in advancing smart forestry.
... In recent years, depth cameras have been widely utilized in various fields, such as autonomous driving [1][2][3], robot grasping [4][5][6], and simultaneous localization and mapping (SLAM) [7][8][9]. However, due to object material, specular reflection, or occlusion, the captured depth data often exhibits holes at the edges where objects come into contact with the background. ...
RGB-D cameras provide depth and color information and are widely used in 3D reconstruction and computer vision. In the majority of existing RGB-D cameras, a considerable portion of depth values is often lost due to severe occlusion or limited camera coverage, thereby adversely impacting the precise localization and three-dimensional reconstruction of objects. In this paper, to address the issue of poor-quality in-depth images captured by RGB-D cameras, a depth image hole repair algorithm based on non-local means is proposed first, leveraging the structural similarities between grayscale and depth images. Second, while considering the cumbersome parameter tuning associated with the non-local means hole repair method for determining the size of structural blocks for depth image hole repair, an intelligent block factor is introduced, which automatically determines the optimal search and repair block sizes for various hole sizes, resulting in the development of an adaptive block-based non-local means algorithm for repairing depth image holes. Furthermore, the proposed algorithm’s performance are evaluated using both the Middlebury stereo matching dataset and a self-constructed RGB-D dataset, with performance assessment being carried out by comparing the algorithm against other methods using five metrics: RMSE, SSIM, PSNR, DE, and ALME. Finally, experimental results unequivocally demonstrate the innovative resolution of the parameter tuning complexity inherent in-depth image hole repair, effectively filling the holes, suppressing noise within depth images, enhancing image quality, and achieving elevated precision and accuracy, as affirmed by the attained results.
... It is worth noting that the inference time for dPF-M-lrn is higher than any other DF in Table II. Noisy and missing observation: According to [35], failures of vehicle cameras may compromise autonomous driving performance -potentially even leading to injuries and death. Common failures are listed by [35], i.e., brightness, blurred vision, and brackish. ...
... Noisy and missing observation: According to [35], failures of vehicle cameras may compromise autonomous driving performance -potentially even leading to injuries and death. Common failures are listed by [35], i.e., brightness, blurred vision, and brackish. We conducted an evaluation of the performance of DEnKF and other DFs in the presence of noisy observations during inference. ...
... However, there are limitations when detecting objects using a frontalviewing camera. Ceccarelli et al. [8] reported that a flare is one of the causes of the failure of an RGB camera in autonomous driving vehicle applications. Figure 1 shows the generation process of a lens flare. ...
In recent years, active research has been conducted on computer vision and artificial intelligence (AI) for autonomous driving to increase the understanding of the importance of object detection technology using a frontal-viewing camera. However, using an RGB camera as a frontal-viewing camera can generate lens flare artifacts due to strong light sources, components of the camera lens, and foreign substances, which damage the images, making the shape of objects in the images unrecognizable. Furthermore, the object detection performance is significantly reduced owing to a lens flare during semantic segmentation performed for autonomous driving. Flare artifacts pose challenges in their removal, as they are caused by various scattering and reflection effects. The state-of-the-art methods using general scene image retain artifactual noises and fail to eliminate flare entirely when there exist severe levels of flare in the input image. In addition, no study has been conducted to solve these problems in the field of semantic segmentation for autonomous driving. Therefore, this study proposed a novel lens flare removal technique based on a class attention map-based flare removal network (CAM-FRN) and a semantic segmentation method using the images in which the lens flare is removed. CAM-FRN is a generative-based flare removal network that estimates flare regions, generates highlighted images as input, and incorporates the estimated regions into the loss function for successful artifact reconstruction and comprehensive flare removal. We synthesized a lens flare using the Cambridge-driving Labeled Video Database (CamVid) and Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago (KITTI) datasets, which are road scene open datasets. The experimental results showed that semantic segmentation accuracy in images with lens flare was removed based on CAM-FRN, exhibiting 71.26% and 60.27% mean intersection over union (mIoU) in the CamVid and KITTI databases, respectively. This indicates that the proposed method is significantly better than state-of-the-art methods.
... In the past two years, due to the rise of autonomous driving, virtual reality, and drones, the requirements for image acquisition and analysis have increased dramatically, and ISP algorithms have played a cornerstone role in image processing in many of the latest camera applications (such as YOLOv5-tassel [3] in 2022 and HYDRO-3D [4] in 2023). Research on the impact of ISP image quality is also on the rise [5]. ...
To tackle the challenges of edge image processing scenarios, we have developed a novel heterogeneous image signal processor (HISP) pipeline combining the advantages of traditional image signal processors and deep learning ISP (DLISP). Through a multi-dimensional image quality assessment (IQA) system integrating deep learning and traditional methods like RankIQA, BRISQUE, and SSIM, various partitioning schemes were compared to explore the highest-quality imaging heterogeneous processing scheme. The UNet-specific deep-learning processing unit (DPU) based on a field programmable gate array (FPGA) provided a 14.67× acceleration ratio for the total network and for deconvolution and max pool, the calculation latency was as low as 2.46 ms and 97.10 ms, achieving an impressive speedup ratio of 46.30× and 36.49× with only 4.04 W power consumption. The HISP consisting of a DPU and the FPGA-implemented traditional image signal processor (ISP) submodules, which scored highly in the image quality assessment system, with a single processing time of 524.93 ms and power consumption of only 8.56 W, provided a low-cost and fully replicable solution for edge image processing in extremely low illumination and high noise environments.
... It is worth noting that the inference time for dPF-M-lrn is higher than any other DF in Table II. Noisy and missing observation: According to [35], failures of vehicle cameras may compromise autonomous driving performance -potentially even leading to injuries and death. Common failures are listed by [35], i.e., brightness, blurred vision, and brackish. ...
... Noisy and missing observation: According to [35], failures of vehicle cameras may compromise autonomous driving performance -potentially even leading to injuries and death. Common failures are listed by [35], i.e., brightness, blurred vision, and brackish. We conducted an evaluation of the performance of DEnKF and other DFs in the presence of noisy observations during inference. ...
This paper introduces a novel state estimation framework for robots using differentiable ensemble Kalman filters (DEnKF). DEnKF is a reformulation of the traditional ensemble Kalman filter that employs stochastic neural networks to model the process noise implicitly. Our work is an extension of previous research on differentiable filters, which has provided a strong foundation for our modular and end-to-end differentiable framework. This framework enables each component of the system to function independently, leading to improved flexibility and versatility in implementation. Through a series of experiments, we demonstrate the flexibility of this model across a diverse set of real-world tracking tasks, including visual odometry and robot manipulation. Moreover, we show that our model effectively handles noisy observations, is robust in the absence of observations, and outperforms state-of-the-art differentiable filters in terms of error metrics. Specifically, we observe a significant improvement of at least 59% in translational error when using DEnKF with noisy observations. Our results underscore the potential of DEnKF in advancing state estimation for robotics. Code for DEnKF is available at https://github.com/ir-lab/DEnKF
... Therefore, we consider faults in the three cameras used in the autonomous vehicle in our case study (discussed in the experiments section). While many types of faults can be associated with a digital camera, we focus on the common fault of occlusion [5]. We train a monitor using prior data and only use inference on the trained models at decision time. ...
Learning Enabled Components (LEC) have greatly assisted cyber-physical systems in achieving higher levels of autonomy. However, LEC's susceptibility to dynamic and uncertain operating conditions is a critical challenge for the safety of these systems. Redundant controller architectures have been widely adopted for safety assurance in such contexts. These architectures augment LEC "performant" controllers that are difficult to verify with "safety" controllers and the decision logic to switch between them. While these architectures ensure safety, we point out two limitations. First, they are trained offline to learn a conservative policy of always selecting a controller that maintains the system's safety, which limits the system's adaptability to dynamic and non-stationary environments. Second, they do not support reverse switching from the safety controller to the performant controller, even when the threat to safety is no longer present. To address these limitations, we propose a dynamic simplex strategy with an online controller switching logic that allows two-way switching. We consider switching as a sequential decision-making problem and model it as a semi-Markov decision process. We leverage a combination of a myopic selector using surrogate models (for the forward switch) and a non-myopic planner (for the reverse switch) to balance safety and performance. We evaluate this approach using an autonomous vehicle case study in the CARLA simulator using different driving conditions, locations, and component failures. We show that the proposed approach results in fewer collisions and higher performance than state-of-the-art alternatives.
... The images captured by the visual camera may be altered due to a multitude of reasons [32]: this paper accounts for possible failures of the visual camera, which may be due to internal (e.g., failures of the electrical parts), external (e.g., dirt or scratched lenses), or environmental (e.g., rain or icing on lenses) factors. We consider a complete set of visual camera failures resulting from a Failure Mode and Effect Analysis [22] to embrace a complete set of visual camera failures known to date, according to the literature. ...
... We assessed this methodology by considering the deployment of image classifiers in autonomous vehicles for Traffic Sign Recognition (TSR), which may be significantly affected by visual camera failures [22], [28], [33]. We gathered three well-known TSR datasets, namely the German Traffic Sign Recognition Benchmark (GTSRB, [10]), the Belgium Traffic Sign (BelgiumTSC, [11]), and the Dataset of Italian Traffic Signs (DITS, [12]), applied AlexNet [7], MobileNetV2 [9], and Inceptionv3 [8] DNNs, which have wide application in the TSR domain [3], [4], [5], and injected a total of 13 visual camera failures under different configurations. ...
... The most comprehensive work on visual camera failures is [22], where the authors systematically identified the failure modes and effects of a visual camera through the application of a Failure Mode and Effects Analysis (FMEA). The effects of camera failures on the output images are summarized in Fig. 2. Failures are caused by malfunctions of the lens, Image Sensor, Bayer Filter, or ISP, and belong to the following categories. ...
Deep Neural Networks (DNNs) have become an enabling technology for building accurate image classifiers, and are increasingly being applied in many ICT systems such as autonomous vehicles. Unfortunately, classifiers can be deceived by images that are altered due to failures of the visual camera, preventing the proper execution of the classification process. Therefore, it is of utmost importance to build image classifiers that can guarantee accurate classification even in the presence of such camera failures. This study crafts classifiers that are robust to failures of the visual camera by augmenting the training set with artificially altered images that simulate the effects of such failures. Such a data augmentation approach improves classification accuracy with respect to the most common data augmentation approaches, even in the absence of camera failures. To provide experimental evidence for our claims, we exercise three DNN image classifiers on three image datasets, in which we inject the effects of many failures into the visual camera. Finally, we applied eXplainable AI to debate why classifiers trained with the data augmentation approach proposed in this study can tolerate failures of the visual camera.
