FIGURE 2 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Multi-culture facial expression recognition framework Dataset Preparation
Source publication
Facial expressions convey exhaustive information about human emotions and the most interactive way of social collaborations, despite differences in ethnicity, culture, and geography. Due to cultural differences, the variations in facial structure, facial appearance, and facial expression representation are the main challenges to the facial expressi...
Similar publications
Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state of the...
This proposed work deals with the uses and techniques of 3D range images for facial expression recognition. A 3D range image is basically a depth image (also called a 2.5D image), which contains depth information at each (x, y) pixel of the image. In the future, computer vision will become a part of our everyday life because of all of its extensive...
The massive adoption of social networks has made available an unprecedented amount of user-generated content, which may be analyzed in order to determine people's opinions and emotions on a large variety of topics. Research has made many efforts in defining accurate algorithms for the analysis of emotions conveyed by texts, however their performanc...
Despite their continued popularity, categorical approaches to affect recognition have limitations, especially in real-life situations. Dimensional models of affect offer important advantages for the recognition of subtle expressions and more fine-grained analysis. We introduce a simple but effective facial expression analysis (FEA) system for dimen...
Citations
... One position taken is that the facial expressions of the six basic emotions relate to Western interpretations of these, but are not representative for instance, of East Asian culture (Jack et al., 2012). While often universality of facial expressions of the six basic emotions is assumed, several studies tackled cross-cultural FER (Dailey et al., 2010;Benitez-Garcia et al., 2017;Ali et al., 2020) and highlighted culture-specific differences in automated detection. The features used in this work are based on a model that was pre-trained on AffectNet data without accounting for culture-specific differences. ...
Student’s shift of attention away from a current learning task to task-unrelated thought, also called mind wandering, occurs about 30% of the time spent on education-related activities. Its frequent occurrence has a negative effect on learning outcomes across learning tasks. Automated detection of mind wandering might offer an opportunity to assess the attentional state continuously and non-intrusively over time and hence enable large-scale research on learning materials and responding to inattention with targeted interventions. To achieve this, an accessible detection approach that performs well for various systems and settings is required. In this work, we explore a new, generalizable approach to video-based mind wandering detection that can be transferred to naturalistic settings across learning tasks. Therefore, we leverage two datasets, consisting of facial videos during reading in the lab (N = 135) and lecture viewing in-the-wild (N = 15). When predicting mind wandering, deep neural networks (DNN) and long short-term memory networks (LSTMs) achieve F $$_{1}$$ 1 scores of 0.44 (AUC-PR = 0.40) and 0.459 (AUC-PR = 0.39), above chance level, with latent features based on transfer-learning on the lab data. When exploring generalizability by training on the lab dataset and predicting on the in-the-wild dataset, BiLSTMs on latent features perform comparably to the state-of-the-art with an F $$_{1}$$ 1 score of 0.352 (AUC-PR = 0.26). Moreover, we investigate the fairness of predictive models across gender and show based on post-hoc explainability methods that employed latent features mainly encode information on eye and mouth areas. We discuss the benefits of generalizability and possible applications.
... The claim of 96% accuracy is impressive but lacks a comprehensive discussion on potential challenges, such as the model's performance with real-world variations in lighting conditions, diverse demographics, or facial expressions that deviate from regular patterns. The proposed approach in [23] introduces ANN, involving the adoption of a pre-trained model, replacing its dense upper layer for compatibility with emotion recognition, and finetuning it with facial emotion data. This work generates an accuracy of 89.47%, however, the success of theensemble approach relies heavily on the availability and representativeness of the training data. ...
The necessity of identifying an individual's intentions solely through facial expressions, without verbal communication, is crucial in daily life. Various approaches employing Artificial Intelligence interventions are utilized to recognize emotions. Consequently, the present study aims to propose an optimal image processing approach named as Hybrid Optimization Based Artificial Neural Network (HOBANN) for the recognition of facial expression. In this process, the implementation of Adaptive Gabor filter is preferred to remove the noise contents in the input image while preserving edge information effectively. Following this, Cascaded Fuzzy C-Means (CFCM) approach is employed to segment the image into multiple sections, enhancing accuracy across an extensive range. In the process of feature extraction, an efficient method known as Gray-Level Co-occurrence Matrix (GLCM) is employed. This method offers several benefits, including reduced complexity and optimal computations of pixel variance. Additionally, a novel hybrid optimization technique called Artificial Bee Colony-Cuckoo Search (ABC-CS) is introduced. This hybrid optimization is applied in the final classification process using an Artificial Neural Network (ANN) classifier, aiming to recognize expressions with maximum accuracy effectively. The Matlab simulink is preferred to validate the entire study and the simulated outcomes have demonstrated that the overall performance of this approach is significantly high as it delivers optimal accuracy rate of 97.93%.
... Therefore, constructing FER databases and annotations must prioritize label correlation and discrepancies with careful consideration. In the future, young researchers focusing on FER should emphasize multimodal emotion recognition and take into account factors such as ethnicity, race, and cultural behavior [6]. ...
In the contemporary era, Facial Expression Recognition (FER) plays a pivotal role in numerous fields due to its vast application areas, such as e-learning, healthcare, marketing, and psychology, to name a few examples. Several research studies have been conducted on FER, and many reviews are available. The existing FER review paper focused on presenting a standard pipeline for FER to predict basic expressions. However, previous studies have not given an adequate amount of importance to FER datasets and their influence on affecting FER system performance. In this systematic review, 105 papers retrieved papers from IEEE, ACM, Science Direct, Scopus, Web of Science, and Springer from the years 2002 to 2023, following systematic review guidelines. Review protocol and research questions are also developed for the analysis of study results. The review identified that the accuracy of the FER system in controlled and spontaneous facial expression datasets is being affected, along with other challenges such as illumination, pose, and scale variation. Furthermore, this paper comparatively analyzed the FER model in both machine and deep learning techniques, including face detection, pre-processing, handcrafted feature extraction techniques, and emotion classifiers. In addition, we discussed some unresolved issues in FER and suggested solutions to enhance FER system performance further. In the future, multimodal FER systems need to be developed for real-time scenarios, considering the computational efficiency of model performance when integrating more than one model and dataset to achieve promising accuracy and reduce error rates.
... Two independent encoders taken the input for these two branches and then fused by two decoders. In another work [25] the GANs consists of generator and discriminator. Generator takes the expression related features and Impose on a sample face. ...
Facial Expressions are quite personalized and may look different for different individuals. Whereas there are certain facial muscles which shows some common features for certain human expressions across the cultures and facial shapes. Convolution Neural Network have shown tremendous success in Facial Expression Recognition task. In recent past many researchers have proposed multiple models with manageable size solution to Facial Expression Recognition task. In the current work, we have considered shape, complexion and other identity related information separate from certain specified muscle movements which are specific for emotion recognition. This is done by a novel Emotion-Generative Adversarial Network. This saves a lot of effort and simplifies the Facial Expression Recognition process. We then apply Scale Invariant Feature Transformation and vola john's face extraction method for pre-processing and face image extraction from background. This enables us to train our model accurately irrespective of scale, orientation, illumination etc and with very less training samples accurately. We feed the feature extracted facial image to an attention-based Convolutional Neural Network. This will ensure more emphasis on critical areas for expression recognition of facial image. Finally, we have used Local Binary Pattern for classification of the input image to a particular emotion class. We have tested our model on CK+, OULU-Casia and FER-2013 datasets and it is at par with performance of all major state-of-art models. Proposed model may be utilized by various automated interactive systems, such as robot to human communication, automated customer care systems etc. The proposed work may also be quite useful for observing reaction of viewers to a particular advertisement or article automatically and use this information for various purposes like user's interest, product feedback etc.
... Ali et al. proposed a method to recognize the multicultural facial expression and used the combination of different datasets including JAFFE, TFEID, and RaFD to make a multi-cultural dataset. The result of their work concluded that facial expressions are natural across different cultures with small differences [229]. The frequency of different models used is shown in Fig. 28. ...
Human ideas and sentiments are mirrored in facial expressions. Facial expression recognition (FER) is a crucial type of visual data that can be utilized to deduce a person’s emotional state. It gives the spectator a plethora of social cues, such as the viewer’s focus of attention, emotion, motivation, and intention. It’s said to be a powerful instrument for silent communication. AI-based facial recognition systems can be deployed at different areas like bus stations, railway stations, airports, or stadiums to help security forces identify potential threats. There has been a lot of research done in this area. But, it lacks a detailed review of the literature that highlights and analyses the previous work in FER (including work on compound emotion and micro-expressions), and a comparative analysis of different models applied to available datasets, further identifying aligned future directions. So, this paper includes a comprehensive overview of different models that can be used in the field of FER and a comparative study of the traditional methods based on hand-crafted feature extraction and deep learning methods in terms of their advantages and disadvantages which distinguishes our work from existing review studies.This paper also brings you to an eye on the analysis of different FER systems, the performance of different models on available datasets, evaluation of the classification performance of traditional and deep learning algorithms in the context of facial emotion recognition which reveals a good understanding of the classifier’s characteristics. Along with the proposed models, this study describes the commonly used datasets showing the year-wise performance achieved by state-of-the-art methods which lacks in the existing manuscripts. At last, the authors itemize recognized research gaps and challenges encountered by researchers which can be considered in future research work.
... Ali, et al. [7] proposed an artificial neural network-based ensemble classifier for multiculture expression analysis. The ensemble classifier consisted of three levels of classifiers: base-level, meta-level, and predictor. ...
Facial expression recognition has become a hot issue in the field of artificial intelligence. So, we collect literature on facial expression recognition. First, methods based on machine learning are introduced in detail, which include image preprocessing, feature extraction, and image classification. Then, we review deep learning methods in detail: convolutional neural networks, deep belief networks, generative adversarial networks, and recurrent neural networks. Moreover, the advantages and limitations of different facial expression recognition methods are compared. In addition, 20 commonly used facial expression datasets are collected in this paper, and the types of expressions and the number of images contained in each dataset are summarized. Finally, the current problems and future development of facial expression recognition are concluded.
... The traditional feature detection methods include Local Binary Pattern (LBP) [12][13], histogram of oriented gradients [14], facial action units [15], Principal component analysis [16], and so on for detecting the relevant features from the face for expression classification. An artificial neural network-based ensemble classifier is employed in [17] for analyzing the multicultural facial expression where Local Binary Pattern (LBP), Principal Component Analysis (PCA), and uniform Local Binary Pattern (LBP) are applied for representing the facial features. [18] proposes multitask cascaded convolutional networks which have cascade detection features for complete face identification and captured face coordinates are sent to the facial expression classification model to recognize the facial emotions. ...
Facial expression helps to communicate between the people for conveying abundant information about human emotions. Facial expression classification is applied in various fields such as remote learning education, medical care, and smart traffic. However, due to the complexity and diversity of the facial emotions, the present facial expression recognition model causes a low recognition rate and it is hard to extract the precise features that are related to facial expression changes. In order to overcome this problem, we proposed Multi-feature Integrated Concurrent Neural Network (MICNN) which is significantly different from the single neural network architectures. It aggregates the prominent features of facial expressions by integrating the three kinds of networks such as Sequential Convolutional Neural Network (SCNN), Residual Dense Network (RDN), and Attention Residual Learning Network (ARLN) to enhance the accuracy rate of facial emotions detection system. Additionally, Local Binary Pattern (LBP) and Principal Component Analysis (PCA) are applied for representing the facial features and these features are combined with the texture features identified by the Gray-level Co-occurrence Matrix (GLCM). Finally, the integrated features are fed into softmax layer to classify the facial images. The experiments are carried out on benchmark datasets by applying k-fold cross-validation and the results demonstrate the superiority of the proposed model.
... With the advancement of computers, facial expression recognition plays a significant role in various applications, such as the modern medical field, including health care [8] and computer-based security. A sufficient number of researches have been carried out for recognizing expressions that help to recognize and classify the basic emotional expressions from the facial pictures [11], such as fear, anger, sadness, surprise and disgust. ...
... RTCRelief-F is clustering based and ordering based ensemble pruning algorithm which clusters the classifiers and resets the fusion order based on classifiers' ability. Ali et al. (2020) developed a multicultural FER model using an ensemble classifier approach. This model proposed a two-level feature extraction using Local Binary Patterns (LBP), Uniform LBP (ULBP) which attains facial features with many dimensions. ...
Facial Expression Recognition (FER) is the basis for many applications including human-computer interaction and surveillance. While developing such applications, it is imperative to understand human emotions for better interaction with machines. Among many FER models developed so far, Ensemble Stacked Convolution Neural Networks (ES-CNN) showed an empirical impact in improving the performance of FER on static images. However, the existing ES-CNN based FER models trained with features extracted from the entire face, are unable to address the issues of ambient parameters such as pose, illumination, occlusions. To mitigate the problem of reduced performance of ES-CNN on partially occluded faces, a Component based ES-CNN (CES-CNN) is proposed. CES-CNN applies ES-CNN on action units of individual face components such as eyes, eyebrows, nose, cheek, mouth, and glabella as one subnet of the network. Max-Voting based ensemble classifier is used to ensemble the decisions of the subnets in order to obtain the optimized recognition accuracy. The proposed CES-CNN is validated by conducting experiments on benchmark datasets and the performance is compared with the state-of-the-art models. It is observed from the experimental results that the proposed model has a significant enhancement in the recognition accuracy compared to the existing models.
... More discriminative features and efficient learning. [111] Visual (Face) ...
The field of Artificial Intelligence (AI) has gained immense traction over the past decade, producing increasingly successful applications as research strives to understand and exploit neural processing specifics. Nonetheless emotion, despite its demonstrated significance to reinforcement, social integration and general development, remains a largely stigmatized and consequently disregarded topic by most engineers and computer scientists. In this paper we endorse emotion’s value for the advancement of artificial cognitive processing, as well as explore real-world use cases of emotion-augmented AI. A schematization is provided on the psychological-neurophysiologic basics of emotion in order to bridge the interdisciplinary gap preventing emulation and integration in AI methodology, as well as exploita- tion by current systems. In addition we overview three major subdomains of AI greatly benefiting from emotion, and produce a systematic survey of meaningful yet recent contributions to each area. To conclude, we address crucial challenges and promising research paths for the future of emotion in AI with the hope that more researchers will develop an interest for the topic and find it easier to develop their own contributions.
