Figure 4 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Source publication
SWIR imaging bears considerable advantages over visible-light (color) and thermal images in certain challenging propagation conditions. Thus, the SWIR imaging channel is frequently used in multi-spectral imaging systems (MSIS) for long-range surveillance in combination with color and thermal imaging to improve the probability of correct operation i...
Contexts in source publication
Context 1
... collection station accesses the systems from the console application which has the possibility of controlling the device, to reproduce and record streams from vMSIS3 cameras. Block diagram of the system is shown in Figure 4. ...
Citations
... Even though it increases security, it takes time because it is policy-dependent [17]. Vehicles [18] fully automated systems that can recognize objects on their own frequently involve extremely sophisticated cameras and sensors with incredibly fine resolution. Computer vision models that have been pre-trained to recognize the brands and models of vehicles and other objects shown in the images are used to analyze images and videos. ...
... Then, the saliency maps are fed to a second module that performs the detection using a feedforward ResNet architecture. Other recent techniques include the use of multispectrum SWIR images, which are extended to three channels similar to the format of RGB images [26,27]. ...
... Although most previous studies have relied on publicly available visible image datasets, more recent studies have highlighted the potential of using Short-Wave Infrared (SWIR) sensors for automated target detection and tracking. SWIR sensors, for instance, have been used to identify UAVs [1,2], detect gas leaks [3], and conduct long-range surveillance [4]. Due to their greater sensitivity to longer light wavelengths than visible sensors, SWIR sensors can see through dense fog and thin layers. ...
... Kandylakis et al. [10] developed a multi-sensor camera system consisting of SWIR, thermal, and hyperspectral cameras and detected moving objects using object detection algorithms, including Faster R-CNN [11] and YOLOv2 [12]. Pavlović et al. [4] proposed a method for automatic crossspectral annotation of SWIR sensor images and a deep learning model for detecting two types of objects: cars and people. For SWIR sensor data annotation, Pavlović et al. [4] recorded visible-light sensor and SWIR sensor data in parallel, and performed automatic cross-spectral annotation using the YOLOX model. ...
... Pavlović et al. [4] proposed a method for automatic crossspectral annotation of SWIR sensor images and a deep learning model for detecting two types of objects: cars and people. For SWIR sensor data annotation, Pavlović et al. [4] recorded visible-light sensor and SWIR sensor data in parallel, and performed automatic cross-spectral annotation using the YOLOX model. As an extension of these studies, our work investigates object detection in city-surveillance scenarios using SWIR sensors, as well as object tracking for five classes of objects. ...
Although Short-Wave Infrared (SWIR) sensors have advantages in terms of robustness in bad weather and low-light conditions, the SWIR images have not been well studied for automated object detection and tracking systems. The majority of previous multi-object tracking studies have focused on pedestrian tracking in visible-spectrum images, but tracking different types of vehicles is also important in city-surveillance scenarios. In addition, the previous studies were based on high-computing-power environments such as GPU workstations or servers, but edge computing should be considered to reduce network bandwidth usage and privacy concerns in city-surveillance scenarios. In this paper, we propose a fast and effective multi-object tracking method, called Multi-Class Distance-based Tracking (MCDTrack), on SWIR images of city-surveillance scenarios in a low-power and low-computation edge-computing environment. Eight-bit integer quantized object detection models are used, and simple distance and IoU-based similarity scores are employed to realize effective multi-object tracking in an edge-computing environment. Our MCDTrack is not only superior to previous multi-object tracking methods but also shows high tracking accuracy of 77.5% MOTA and 80.2% IDF1 although the object detection and tracking are performed on the edge-computing device. Our study results indicate that a robust city-surveillance solution can be developed based on the edge-computing environment and low-frame-rate SWIR images.
... In visual object tracking, as well as in thermal object tracking, deep-learning-based methods have gained significant attention in recent period. However, the application of deep learning methods in the SWIR domain is mostly focused on object detection [14], [15]. Deep-learningbased algorithms related to object tracking in SWIR video are applied in [16], [17], where convolutional neural networks are used for multiple targets tracking in a degraded SWIR image with a significant percentage of missing data and bad pixels, called the compressive measurement domain. ...
... If this is not a case, such goal can be achieved by making the observation noise covariance matrix, R, in (13) diagonal, using the Cholesky decomposition [43]. Taking into account that the prediction and the estimation steps in the standard Kalman filter are mutually independent, the roust Kalman filter prediction step remains unchanged, as is defined by (14) - (15), while the estimation step defined by the (16) - (20), has to be redesigned by using the two-step optimization procedure defined by (21) - (30). This can be achieved by utilizing the influence function (26) on the scaled measurement residual. ...
Short-wave infrared (SWIR) imaging has significant advantages in challenging propagation conditions where the effectiveness of visible-light and thermal imaging is limited. Object tracking in SWIR imaging is particularly difficult due to lack of color information, but also because of occlusions and maneuvers of the tracked object. This paper proposes a new algorithm for object tracking in SWIR imaging, using a kernelized correlation filter (KCF) as a basic tracker. To overcome occlusions, the paper proposes the use of the Kalman filter as a predictor and a method to expand the object search area. Expanding the object search area helps in better re-detection of the object after occlusion, but also leads to the occasional appearance of errors in measurement data that can lead to object loss. These errors can be treated as outliers. To cope with outliers, Huber’s M-robust approach is applied, so this paper proposes robustification of the Kalman filter by introducing a nonlinear Huber’s influence function in the Kalman filter estimation step. However, robustness to outliers comes at the cost of reduced estimator efficiency. To make a balance between desired estimator efficiency and resistance to outliers, a new adaptive M-robustified Kalman filter is proposed. This is achieved by adjusting the saturation threshold of the influence function using the detection confidence information from the basic KCF tracker. Experimental results on the created dataset of SWIR video sequences indicate that the proposed algorithm achieves a better performance than state-of-the-art trackers in tracking the maneuvering object in the presence of occlusions.
... In order to obtain accurate predictions for parallel visual and IR frame processing, it is required that they are aligned by reducing the size of the visual frames, as they have a higher resolution than IR frames. In [21], homograph matrix calculations for mapping objects on parallel images was proposed. Both datasets were applied to sets of frames, and the F1 score, precision, and recall were calculated. ...
Thermography is being increasingly used in building inspection due to its capability to determine various defects, as this enables the development of improvement strategies for efficient energy consumption. In this paper, AI algorithms are combined, and new segmentation strategies are proposed to improve the accuracy of building insulation assessments. Paired visual and IR pictures taken from the same angle are used complementarily to feed different sequential neural networks employed to extract the characteristic segments of buildings. The optical images contain the information required to identify and separate objects, such as windows, doors, and walls. The IR pictures contain the information required for the insulation assessment. This enables an automated analysis of a large number of objects within the same assessment with respect to the proper viewing angle and resolution. Variations in measured temperatures for segmented regions are estimated by referring to their representations in the IR frames, which allows for general conclusions concerning insulation state to be drawn, and by using a trained neural network, heat losses are localized in the frames. The output levels of consecutive IR frames are compared to determine the effects on IR object representation due to different recording aspects.
