Figure - available from: Multimedia Tools and Applications
This content is subject to copyright. Terms and conditions apply.
RGBD facial images (the first row presents RGB images, and the second row presents depth images) of six basic expressions captured with Kinect 2.0
Source publication
Facial expression recognition (FER) is an important means for machines to understand the changes in the facial expression of human beings. Expression recognition using single-modal facial images, such as gray scale, may suffer from illumination changes and the lack of detailed expression-related information. In this study, multi-modal facial images...
Similar publications
This paper presents an effective method for automated 3D/4D facial expression recognition based on Mesh-Local Binary Pattern Difference (mesh-LBPD). In contrast to most of existing methods, the proposed mesh-LBPD is based on a unified set of geometric and appearance features of different facial regions. Indeed, multiple features are combined into a...
Objectives: The emotion detection is one of the important fields in computer-human interaction and this study plays significant a role for identification facial expression from the images. To identify the single emotion, need various variability of human shapes such as pose, color, texture, expression, posture, and orientation. In this study, we im...
Abstract - humans make use of facial expression to communicate in their day to day interactions with each other, which comes
naturally without much effort. Facial expression is essentially a communication and interaction between humans and where
other information like speech is not available; it becomes what one can depend on to transmit emotion or...
Citations
... For small databases, the RF model can present good prediction results for the data [28]. RF model has been widely used in many fields, such as image recognition [39] and corrosion [40,41], due to their simple structure, easy implementation, anti-overfitting and suitability for small sample data and non-linear datasets [42]. Pei et al. [28] investigated the effect of atmospheric corrosion on carbon steel using the RF model and identified a hierarchy of environmental factors on the initial atmospheric corrosion. ...
... After all, face detection is already a mature direction developed separately, and various excellent algorithms are constantly being proposed. These face landmark detectors [23][24][25] based on region segmentation and thus subsequent feature extraction efforts, however, degrade performance due to the erratic behavior of facial key point detection, and such anomalous behavior is possible in the presence of occlusions and challenging scenes with pose and lighting changes. In this paper, we do not rely on landmark detectors and simply use the most common averaging of the image into different subblocks. ...
To address the problem that the features extracted by CNN‐based facial expression recognition (FER) do not consider structural information, a region adaptive correlation deep network (RACN) is proposed. The network consists of two branches. In one branch, the features obtained by applying CNN to facial sub‐blocks are used as the input of the proposed second‐order region correlation network (SRCN), which obtains structural features by adaptively learning the correlation of facial regions. Furthermore, they are fused with the parallel branch‐extracted global features to obtain a comprehensive high‐semantic feature representation. Finally, weights are assigned to the two features through the channel attention mechanism for more accurate expression classification. Experimental results show that our method can effectively extract expression features in an end‐to‐end manner, improve the accuracy of FER, and achieve competitive performance without relying on any a priori knowledge. And the region‐adaptive correlation feature extraction branch RACN can be applied to other deep learning networks to extract discriminative structural‐adaptive features. To the best of our knowledge, our work is the first to enrich the feature representation for end‐to‐end static FER by adaptively obtaining more discriminative regional adaptive correlation feature vectors via the autocorrelation matrix combined with CNN compared to the existing literature.
... However, Suwa [10] did a preliminary investigation in 1978. Even though many academics feel the six basic emotions are culturally unique and not universal, the categorization approach based on the six fundamental expressions has been widely accepted by researchers and has aided the growth of expression recognition [11]. ...
With advances in computer vision and artificial intelligence technology, facial expression recognition research has become a prominent topic. Current research is grappling with how to enable computers to fully understand expression features and improve recognition rates. Most single face image datasets are based on the psychological classification of the six basic human expressions used for network training. By outlining the problem of facial recognition by comparing traditional methods, deep learning, and broad learning techniques, this review highlights the remaining challenges and future directions of deep learning and broad learning research. The deep learning method has made it easier and more effective to extract expression features and improve facial expression recognition accuracy by end-to-end feature learning, but there are still many difficulties in robustness and real-time performance. The broad learning system (BLS) is a broad network structure that is expanded by increasing the number of feature nodes and enhancement nodes appropriately to reinforce the structure and is also effective in facial expression recognition. However, outliers and noises in unbalanced datasets need BLS to solve in the future. Finally, we present several problems that still need to be addressed in facial expression recognition.
... However, due to the sensitivity to environmental changes (e.g., illumination changes [21]), facial appearance information is not stable enough. Most researchers have focused on facial depth information which provides complementary information to improve the performances [22,23]. Compared to the existing DDAL, we need to design an efficient multi-modal semi-supervised deep network in which not only facial appearance information is used, but also facial depth information is exploited. ...
As an effective deep semi-supervised network, deep discrimination association learning has achieved impressive performance in facial expression recognition (FER). However, the instability of facial appearance (i.e. illumination change) is still a big challenge for FER. To this point, we propose a novel multi-modal deep discriminative association (MDDA) framework to better exploit facial depth information and unlabeled data. Firstly, the facial depth map is generated via 3D face reconstruction and encoded in three channels to learn more representative features. Secondly, we design a novel deep multi-loss semi-supervised network based on association learning and exploit multi-modal information through cross fusion mechanism. We evaluate the proposed method on RaFD and Oulu-CASIA datasets and achieve accuracies of 95.64% and 66.75%, respectively. Compared to the existing deep discrimination association learning approach, the accuracies are increased by 1.01% and 4.88% with the encoded facial depth map information. Moreover, extensive experiments confirm that the proposed approach has comparable performance compared to existing deep networks and is more robust to illumination changes.
... It is challenging to compare different ML and deep-learning-based facial recognition strategies due to the different setups, datasets and machines used [95,96]. However, the latest comparison of different approaches is presented in Table 3. Tables 3 and 4, deep-learning-based approaches outperform conventional approaches. ...
Facial emotion recognition (FER) is an emerging and significant research area in the pattern recognition domain. In daily life, the role of non-verbal communication is significant, and in overall communication, its involvement is around 55% to 93%. Facial emotion analysis is efficiently used in surveillance videos, expression analysis, gesture recognition, smart homes, computer games, depression treatment, patient monitoring, anxiety, detecting lies, psychoanalysis, paralinguistic communication, detecting operator fatigue and robotics. In this paper, we present a detailed review on FER. The literature is collected from different reputable research published during the current decade. This review is based on conventional machine learning (ML) and various deep learning (DL) approaches. Further, different FER datasets for evaluation metrics that are publicly available are discussed and compared with benchmark results. This paper provides a holistic review of FER using traditional ML and DL methods to highlight the future gap in this domain for new researchers. Finally, this review work is a guidebook and very helpful for young researchers in the FER area, providing a general understating and basic knowledge of the current state-of-the-art methods, and to experienced researchers looking for productive directions for future work.
... The included datasets are explained as follows. (Yang et al. 2017) CK database (Jung et al. 2015a) was the first version of CK+ database having 486 images from 97 subject sequences which are FACS coded images. CK+ database is composed of both non-posed and posed expressions images from 123 subjects. ...
Facial expression plays an important role in communicating emotions. In this paper, a robust method for recognizing facial expressions is proposed using the combination of appearance features. Traditionally, appearance features mainly divide any face image into regular matrices for the computation of facial expression recognition. However, in this paper, we have computed appearance features in specific regions by extracting facial components such as eyes, nose, mouth, and forehead, etc. The proposed approach mainly has five stages to detect facial expression viz. face detection and regions of interest extraction, feature extraction, pattern analysis using a local descriptor, the fusion of appearance features and finally classification using a Multiclass Support Vector Machine (MSVM). Results of the proposed method are compared with the earlier holistic representations for recognizing facial expressions, and it is found that the proposed method outperforms state-of-the-art methods.
... Yurtkan and Demirel (2014) proposed a novel feature selection program which uses three-Dimensional (3D) geometric facial feature positions to recognize basic facial expressions with high recognition rate. Yang et al. (2018) proposed a strategy of fusing dual features (including deep geometric features and local appearance features) to maximize the use of complementary facial information. Tsai and Chang (2018) proposed a new technology of facial expression recognition (FER) based on support vector machine (SVM), which develops a face detection method that combines the Haar-like features method with the self-quotient image (SQI) filter. ...
Human facial expression recognition has been treated as a multi-class classification problem in the field of artificial intelligence. The main difficulty lies in how to distinguish the different categories of expression features. In this paper, we identify common facial expressions by fusing multiple weak classifiers. It compensates for the disadvantage of single classifier in weak generalization ability and low recognition rate for different datasets and different environments. This paper integrates the prediction results of each classifier through improved weighted mean value method and proposes an expression feature extraction method based on keypoint detection. Classifier fusion methods enable each classifier to perform at its best in order to improve overall expression recognition. Keypoint detection is used to improve the model’s attention on the expression features. Convolution neural network is selected as the model for feature extraction and classification, and the model structure is adjusted. Experiments show that the recognition accuracy of this method used on datasets FER 2013 and CK+ are 70.7% and 95.4% respectively, which are better than that of a single classifier, which shows that the keypoint extraction feature and classifier fusion method used in this paper have a good effect on facial expression recognition.
... This section describes three different types of datasets for the facial expression recognition namely JAFFE dataset [30], FER-2013 dataset [31] and CK + dataset [32]. The details regarding the datasets are described in the following section. ...
Recently, Encryption technology plays a vital role in providing secure information. In the multimedia sector, several images and posts are broadcasted via the internet on daily basis; in which the images are transmitted on social media and are subjected to several security attacks. Therefore it is necessary to protect the images from illegal or forbidden access. This paper aims in developing a novel SKECA-EMFO based encrypted transmission system. Here, an effective encryption transmission system is designed using a Bayes minimum risk classifier to secure the sensitive information during the transmission processes. In addition to this, the SM4 encryption algorithm is employed to perform high speed encrypted transmission as well as to achieve intelligent recognition. The novel SKECA-EMFO approach is employed in demonstrating facial expression recognition thereby obtaining an optimal feature set. Finally, seven benchmark functions are utilized to examine and evaluate the effectiveness of the newly developed proposed approach. The comparative analysis is carried out with few approaches and the results reveal that the proposed approach provides better performances when compared with other approaches.
... This section describes three different types of datasets for the facial expression recognition namely JAFFE dataset, 37 FER-2013 dataset 38 and CK + dataset. 39 The details regarding the datasets are described in the following section. ...
Recently, encryption technology plays a vital role in providing secure information. In the multimedia sector, several images and posts are broadcasted via the internet on a daily basis; in which the images are transmitted on social media and are subjected to several security attacks. As a result, it is essential to secure the images from illegal or forbidden accessing. This paper aims to develop a novel Supervision Kernel Entropy Component Analysis‐enhanced moth flame optimization (SKECA‐EMFO)‐based encrypted transmission system. Here, an effective encryption transmission system is designed using a Bayes minimum risk classifier to secure sensitive information during the transmission processes. Furthermore, the SM4 encryption algorithm performs high‐speed encrypted transmission speed and achieves intelligent recognition performance. The novel SKECA‐EMFO approach is employed in demonstrating facial expression recognition thereby obtaining an optimal feature set. Finally, seven benchmark functions are utilized to examine and calculate the efficiency of the newly established proposed approach. The comparative analysis is carried out with a few approaches, and the results reveal that the proposed approach provides superior performances than other existing techniques. A Bayes minimum risk classifier is designed to quantify the tradeoffs thus constructing an information gain and entropy to obtain a secure encryption transmission. A novel SKECA‐EMFO approach is proposed to demonstrate facial expression recognition thereby obtaining an optimal feature set. SM4 encryption algorithm is demonstrated to perform high‐speed encrypted transmission as well as to achieve intelligent recognition.
... In the next years, Local Binary Patterns (LBP) [45] is introduced as another image texture descriptor. Different variations of LBP are employed for appearance-based facial representation [18,22,31,60]. In [54], boosted version of LBP descriptor (Boosted-LBP) and different classifiers are used for facial expression recognition. ...
Facial expression is a powerful way for human emotional communications. According to various applications, automatic facial expression recognition becomes an interesting problem for researchers in different areas. An automatic facial expression recognition system recognizes the expressed emotion in input facial image using several processing stages. Feature extraction is a vital step in facial expression recognition. One of the most widely used techniques for feature extraction in machine vision is utilizing Gabor filter which is sensitive to lines at various orientations. The disadvantage of Gabor filters is their computational cost and large feature vector length. Inspiring the human vision system and stimulations of complex cells, in this paper, firstly, facial image is convolved with Gabor filters. Then, the achieved convolution matrices are properly coded based on the maximum and minimum responses. Finally, the feature vector is obtained by calculating the histogram of these codes. The length of achieved histogram for 16 and 8 Gabor filters are 240 and 56, respectively, which is considerably less than keeping all Gabor responses. The proposed method is (person-independently) evaluated on four facial expression recognition datasets including CK+, SFEW, MMI, and RAF-DB. The experimental results show that the proposed method outperforms existing image texture descriptors in facial expression recognition in both controlled and uncontrolled images.
