Figure 1 - uploaded by Francesco Fioranelli
Content may be subject to copyright.
Source publication
In this work, the authors present results for classification of different classes of targets (car, single and multiple people, bicycle) using automotive radar data and different neural networks. A fast implementation of radar algorithms for detection, tracking, and micro-Doppler extraction is proposed in conjunction with the automotive radar transc...
Context in source publication
Context 1
... at 50m / 7.5° Table 1: Radar parameters for the data analysed in this paper 3. Data processing and neural networks architecture As described in section 1, a lot of research has been conducted on target classification using the MD signature of objects. When the target signature is spread across many different range bins, the different target contributions need to be aggregated prior to performing STFT (Short Time Fourier Transform), or an alternative time-frequency distribution, and this is even more important in case of multiple targets crossing their trajectories. To address this issue and easily track multiple moving targets, we have implemented the following processing on the raw data obtained from the NXP radar. The different processing steps have been summarised in Fig. 1 Apply Ordered Statistics CFAR (Constant False Alarm Ratio) algorithm [34] to perform target detection and reduce the undesired contribution from noise and clutter; Detect the position of the targets (i.e. the range bins they occupy) for a given frame and store these coordinates in a detection matrix; Input the detection matrix frame-wise in an algorithm, which combines constant acceleration Kalman filtering and the Hungarian algorithm [35]. The former would produce a better estimation of the target position, as well as continue to output predictions, even if frames are temporarily lost or corrupted. The latter would constantly assign identities to the object detections, based on the estimates from the Kalman filter. The algorithm can also take into consideration new objects entering the radar field of view, or those leaving it, using markers for each track; Concatenate several Range-Time frames and generate segments of micro-Doppler signatures using the object track position estimates, i.e. the range bins where the target signature is located. The duration of the overall micro-Doppler signature can be varied depending on the classification algorithm just by concatenating more or less frames together; Use the generated micro-Doppler spectrograms to train and test classifiers based on neural ...
Similar publications
This paper presents the design and the implementation of a nonlinear integral backstepping control strategy for a single-phase grid-connected voltage source inverter (VSI). The proposed control strategy aims to resolve the problem of the sinusoidal reference current tracking, to have a rapid transient response needed to deal with the transient inci...
Citations
... Furthermore, target classification using the range FFT of a mmWave radar's statistical features is studied in [150]. The utilization of various deep learning techniques and micro-Doppler patterns from radar data for object classification is explored in [151]. Multi-person identification with distinct micro-Doppler signatures is studied in [152]. ...
Human gesture detection, obstacle detection, collision avoidance, parking aids, automotive driving, medical, meteorological, industrial, agriculture, defense, space, and other relevant fields have all benefited from recent advancements in mmWave radar sensor technology. A mmWave radar has several advantages that set it apart from other types of sensors. A mmWave radar can operate in bright, dazzling, or no-light conditions. A mmWave radar has better antenna miniaturization than other traditional radars, and it has better range resolution. However, as more data sets have been made available, there has been a significant increase in the potential for incorporating radar data into different machine learning methods for various applications. This review focuses on key performance metrics in mmWave-radar-based sensing, detailed applications, and machine learning techniques used with mmWave radar for a variety of tasks. This article starts out with a discussion of the various working bands of mmWave radars, then moves on to various types of mmWave radars and their key specifications, mmWave radar data interpretation, vast applications in various domains, and, in the end, a discussion of machine learning algorithms applied with radar data for various applications. Our review serves as a practical reference for beginners developing mmWave-radar-based applications by utilizing machine learning techniques.
... Angelov et al. [71] considered three different types of DNNs, including the CNN, the residual network, and the combination of the CNN and the RNN, for four classes of ground target recognition, and these methods were verified on experimental data. Aiming at the problem of the class-imbalance of pedestrian and vehicle recognition with limited experimental data, Wu et al. [72] introduced a hybrid SVM-CNN method; in the first stage, a modified SVM was utilized to identify vehicle targets and adjust the imbalance between pedestrians and vehicles in the limited data, while the second stage was performed using a CNN to classify the residual unclassified targets. ...
Radar automatic target recognition (RATR) technology is fundamental but complicated system engineering that combines sensor, target, environment, and signal processing technology, etc. It plays a significant role in improving the level and capabilities of military and civilian automation. Although RATR has been successfully applied in some aspects, the complete theoretical system has not been established. At present, deep learning algorithms have received a lot of attention and have emerged as potential and feasible solutions in RATR. This paper mainly reviews related articles published between 2010 and 2022, which corresponds to the period when deep learning methods were introduced into RATR research. In this paper, the current research status of radar target characteristics is summarized, including motion, micro-motion, one-dimensional, and two-dimensional characteristics, etc. This paper reviews the progress of deep learning methods in the feature extraction and recognition of radar target characteristics in recent years, including space, air, ground, sea-surface targets, etc. Due to more and more attention and research results published in the past few years, it is hoped that this review can provide potential guidance for future research and application of deep learning in fields related to RATR.
... In addition, studies applying deep learning in the field of radar signal processing are being actively conducted [24]. In automotive radar signal processing, deep learning is mainly applied for target detection or classification [25][26][27][28][29]. The authors in [26] use range-Doppler maps as input of the deep learning network for target detection. ...
... The authors in [26] use range-Doppler maps as input of the deep learning network for target detection. For target classification, images of the micro-Doppler spectrogram were used as input of the DNN in [27] and authors in [28] use both range-Doppler map and Doppler-time map as input. Moreover, it can be applied in a variety of ways in the radar signal processing, such as mitigating interference [30] or estimating the angle of arrival [31]. ...
The reliability and safety of advanced driver assistance systems and autonomous vehicles are highly dependent on the accuracy of automotive sensors such as radar, lidar, and camera. However, these sensors can be misaligned compared to the initial installation state due to external shocks, and it can cause deterioration of their performance. In the case of the radar sensor, when the mounting angle is distorted and the sensor tilt toward the ground or sky, the sensing performance deteriorates significantly. Therefore, to guarantee stable detection performance of the sensors and driver safety, a method for determining the misalignment of these sensors is required. In this paper, we propose a method for estimating the vertical tilt angle of the radar sensor using a deep neural network (DNN) classifier. Using the proposed method, the mounting state of the radar can be easily estimated without physically removing the bumper. First, to identify the characteristics of the received signal according to the radar misalignment states, radar data are obtained at various tilt angles and distances. Then, we extract range profiles from the received signals and design a DNN-based estimator using the profiles as input. The proposed angle estimator determines the tilt angle of the radar sensor regardless of the measured distance. The average estimation accuracy of the proposed DNN-based classifier is over 99.08%. Therefore, through the proposed method of indirectly determining the radar misalignment, maintenance of the vehicle radar sensor can be easily performed.
... Over the past few years, significant research attention has been paid to the classification and recognition of objects' postures and activities through MDS-based methods, particularly in the areas of object classification [132], [172], human activity and gait recognition [173]- [175], and human gesture recognition [176]. Gao et al. [132] generated STFT maps from the cropped radar RA tensor and fed them to the VGG16 [177] deep learning classifier for MDS extraction. ...
... A decision tree algorithm was then used to evaluate three classes: car, pedestrian and cyclist. Moreover, Angelov et al. [172] converted the raw ADC signal into spectrograms and performed object classification among persons, bicycles and cars. They conducted a comparative evaluation of diverse neural network architectures on MDS, including VGG-like CNN, convolutional residual network, and the combination of convolutional and recurrent Long Short-Term Memory (LSTM) network. ...
Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation. Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse", subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: https://XJTLU-VEC.github.io/Radar-Camera-Fusion.
... Bu sistemler çeşitli elektro optik ve radar sistemlerinden oluşmaktadırlar. Radarlar bu tür sistemlerde olası hedeflerin algılanması ve kullanıcının uyarılması açısından kritik öneme sahiptir [2][3][4][5][6]13]. Kullandıkları teknikler ve amaçlarına göre ayrılan radarlar, cisimlerden yansıyan radyo dalgalarını algılayarak nesne tespiti yapmaktadırlar. ...
... Radarın nesneleri tespit etmesi pratikte tek başına yeterli değildir. Çünkü radar tarafından tespit edilen nesnelerin küçük bir bölümü hedef kümesinde olup, hayatın olağan akışı içerisinde bulunan birçok nesne de (otomobil, insan ve insanlar topluluğu, hayvanlar vb.) radar tarafından algılanmaktadır [2][3][4]. Bu sebeple radar tarafından algılanan nesnelerin sınıflandırılması da bir o kadar önemlidir. Tespit edilen nesneler, mevcut veriler ışığında çeşitli yöntemlerle sınıflandırılmaktadırlar. ...
... Şekil 2'de MSTAR veri setindeki bazı sınıflara ait görseller ve radar görüntüleri verilmektedir. [3][4][5][6][7]. Bu yöntem ile spektrogramlardan frekansa bağlı olarak görüntü kareleri elde edilmektedir. ...
Kritik öneme sahip askeri ve sivil yerleşkelerin korunması geçmişte olduğu gibi günümüzde önemini korumaktadır. Bu amaçla, çeşitli sensörler barındıran sistemler geliştirilmektedir. Sensörlerin sağladığı verilerden bilginin elde edilmesi donanımların en verimli şekilde kullanılması açısından önemlidir. Radar sistemleri keşif, gözetleme ve tespit amacıyla sıklıkla kullanılmaktadır. Radar ile tespit edilen nesnelerin sınıflandırılması amacıyla kural tabanlı ve makine öğrenmesi tabanlı yöntemler mevcuttur. Makine öğrenmesi tabanlı yaklaşımlarda uzman görüşüne gerek kalmadan zaman içerisinde hedef nesnenin karakteristik özellikleri model tarafından öğrenilir. Bu sebeple bu yöntemler kural tabanlı yöntemlere nazaran daha avantajlıdır. Gerçekleştirilen bu çalışmada, dengesiz Doppler Radarı verisi üzerinde İHA’ların diğer nesnelerden ayırt edilmesi amacıyla hedef sınıflandırması yapılmıştır. Deneysel çalışmalarda en yüksek başarım SMOTE kullanılarak dengeli hale getirilen veri setinde elde edilmiş, önerilen CNN modeli ile %99,99 doğruluğa ulaşılmıştır.
... The quantitative results are shown in Table 1. The RODNet results are compared with the following baselines that also use radar-only inputs: (1) a decision tree uses some handcrafted features from radar data [6]; (2) CFAR detection is first implemented, and a radar object classification network with ResNet backbone [47] is appended; and (3) similar to (2), a radar object classification network with VGG-16 backbone based on CFAR detections is mentioned in [6]. Therefore, we make use of exactly the same network and weights of original RODNet to generate ConfMap. ...
In the field of autonomous driving, millimeter-wave (MMW) radar is often used as a supplement sensor of other types of sensors, such as optics, in severe weather conditions to provide target-detection services for autonomous driving. RODNet (A Real-Time Radar Object-Detection Network) is one of the most widely used MMW radar range–azimuth (RA) image sequence target-detection algorithms based on Convolutional Neural Networks (CNNs). However, RODNet adopts an object-location similarity (OLS) detection method that is independent of the number of targets to obtain the final target detections from the predicted confidence map. Therefore, it gives a poor performance on missed detection ratio in dense pedestrian scenes. Based on the analysis of the predicted confidence map distribution characteristics, we propose a new generative model-based target-location detection algorithm to improve the performance of RODNet in dense pedestrian scenes. The confidence value and space distribution predicted by RODNet are analyzed in this paper. It shows that the space distribution is more robust than the value distribution for clustering. This is useful in selecting a clustering method to estimate the clustering centers of multiple targets in close range under the effects of distributed target and radar measurement variance and multipath scattering. Another key idea of this algorithm is the derivation of a Gaussian Mixture Model with target number (GMM-TN) for generating the likelihood probability distributions of different target number assumptions. Furthermore, a minimum Kullback–Leibler (KL) divergence target number estimation scheme is proposed combined with K-means clustering and a GMM-TN model. Through the CRUW dataset, the target-detection experiment on a dense pedestrian scene is carried out, and the confidence distribution under typical hidden variable conditions is analyzed. The effectiveness of the improved algorithm is verified: the Average Precision (AP) is improved by 29% and the Average Recall (AR) is improved by 36%.
... While many different approaches based on traditional radar signal processing exist, they are often based on many imperfect assumptions, require fine-tuning hyper-parameters depending on the specific radar settings, or are infeasible for live applications in terms of computational requirements. Deep learning approaches to some of these problems have shown promising results, but mainly with different types of sensors, in automotive applications, single-target environments, or relatively simple multiple-target environments without many potential clutter targets in the FoV of the radar [13,2,57]. For practical applications, it is necessary that the solutions can work in cluttered indoor environments, with multiple human targets, on a low-cost radar sensor, and with limited compute requirements. ...
Due to the progress in communication technologies and the ever-shrinking form-factors of electronic devices, sensors are becoming increasingly ubiquitous in daily life. In smart-home applications especially, detecting the number of humans present and determining their positions allows for intelligent controls of heating, ventilation, lighting, and entertainment systems, potentially saving significant amounts of energy and CO2 emissions in the process. However, while recent progress through deep learning in computer vision enables cameras to detect human targets accurately, privacy concerns and their dependency on good sight conditions prevent their adoption. Contrary to that, radar sensors are privacy-preserving, work independent of sight conditions, and allow for the estimation of range, velocity, and angle information of targets. This work explores the use of millimeter-wave frequency continuous wave radar sensors for estimating the number of human targets and detecting their positions in indoor environments. To this date, most methods in literature on radar-based tracking, detection, and counting, focus on automotive applications and are mostly based on traditional signal processing-based methods. The work presented in this thesis shows that deep learning enables a low-cost radar sensor to succeed in said tasks in complex multi-target indoor environments, where signal-processing-based methods reach their limits due to multi-path reflections, clutter, and ghost targets. To this end, new training schemes, network architectures, and loss functions were developed to enable performance gains similar to what was seen with the advent of deep learning in computer vision.
... On the other hand, DNNs, accompanied by the use of micro-Doppler signatures [35], [36], are also an alternative approach for the classification of ground targets [37]. This approach has proven effective in several applications targeting human recognition and activity classification [38]- [41], as well as human-robot classification [42]. ...
In this paper, an approach for ground-moving target classification with an FMCW radar is proposed. In particular, data are collected using a low-cost 24 GHz off-the-shelf FMCW radar, combined with an embedded Raspberry Pi device for data acquisition and processing. An FFT-based processing scheme is then applied to obtain a sequence of range-Doppler maps, which are provided in input to different convolutional neural network (CNN) architectures for classifying the targets (cars, motorcycles, or pedestrians) eventually passing in front of the radar. Specifically, two approaches have been followed and compared. In the first one, single range-Doppler maps are processed alone using a convolutional neural network, and then a voting mechanism is applied to select the target classes. In the second approach, a sequence of range-Doppler maps is processed using a time-distributed layer feeding a recurrent neural network. The CNNs are deployed on the Raspberry Pi providing the target classification on a low-cost embedded device. The obtained results show that the proposed approaches allow for effectively detecting the different types of targets running on an embedded device in less than one second.
... Great efforts have been made to advance MMW radar object classification performance. Existing researches are mainly based on three kinds of radar data, including micro-Doppler signatures (Villeval et al., 2014;Angelov et al., 2018;Held et al., 2019), point clouds (Feng et al., 2019;Zhao et al., 2020) and range-Doppler (RD) maps or rangeazimuth maps (Major et al., 2019;Palffy et al., 2020). Since RD maps can be easily obtained in engineering and maintain rich Doppler and object motion information, this paper focuses on object classification based on RD maps. ...
To improve the cognition and understanding capabilities of artificial intelligence (AI) technology, it is a tendency to explore the human brain learning processing and integrate brain mechanisms or knowledge into neural networks for inspiration and assistance. This paper concentrates on the application of AI technology in advanced driving assistance system. In this field, millimeter-wave radar is essential for elaborate environment perception due to its robustness to adverse conditions. However, it is still challenging for radar object classification in the complex traffic environment. In this paper, a knowledge-assisted neural network (KANN) is proposed for radar object classification. Inspired by the human brain cognition mechanism and algorithms based on human expertise, two kinds of prior knowledge are injected into the neural network to guide its training and improve its classification accuracy. Specifically, image knowledge provides spatial information about samples. It is integrated into an attention mechanism in the early stage of the network to help reassign attention precisely. In the late stage, object knowledge is combined with the deep features extracted from the network. It contains discriminant semantic information about samples. An attention-based injection method is proposed to adaptively allocate weights to the knowledge and deep features, generating more comprehensive and discriminative features. Experimental results on measured data demonstrate that KANN is superior to current methods and the performance is improved with knowledge assistance.
... The network is made of multiple convolutional layers followed by fully connected ones. [107] performs classification of moving targets on Doppler signatures. Results are compared across 3 architectures of neural networks: VGG-like [133], ResNet, and a fully convolutional network. ...
MmWave (millimeter wave) Frequency Modulated Continuous Waves (FMCW) RADARs are sensors based on frequency-modulated electromagnetic which see their environment in 3D at a long-range. The recent introduction of millimeter-wave RADARs with frequencies from 60 GHz to 300 GHz has broadened their potential applications thanks to their improved accuracy in angle, range, and velocity. MmWave FMCW RADARs have better resolution and accuracy than narrowband and ultra-wideband (UWB) RADARs. In comparison with cameras and LiDARs, they possess several strong advantages such as long-range perception, robustness to lightning, and weather conditions while being cheaper. However, their noisy and lower-density outputs even compared to other technologies of RADARs, and their ability to measure the targets’ velocities require specific algorithms tailored for them. Working principles of mmWave FMCW RADARs are presented as well as the separate ways to represent data and their applications. This paper describes algorithms and applications adapted or developed for these sensors in automotive applications. Finally, current challenges and directions for future works are presented.
