Conference PaperPDF Available

ECG-based Emotion Recognition: Overview of Methods and Applications

Authors:

Abstract

This paper presents an overview of recent methods for recognition of human emotions based on Electrocardiogram (ECG) signals and related applications. The major challenges in emotion modeling (affective computing) from ECG data are finding representations that are invariant to inter-and intra-subject differences, as well as the inherent noise associated with the ECG data recordings. The most common invariant features (in frequency and time domain) extracted from the raw ECG signals are outlined. The reviewed studies reveal the great potential of ECG to decode basic human emotional states such as joy, sadness, anger, fear in combination with other physiological signals and facial expression. Major application areas cover patient monitoring, marketing, car driving.
ECG-based Emotion Recognition: Overview of
Methods and Applications
Desislava Nikolova
Petya Petkova
Agata Manolova
English Language Faculty of
Faculty of Telecommunications
Faculty of Telecommunications
Engineering
Technical University of Sofia
Technical University of Sofia
Technical University of Sofia
Sofia, Bulgaria
Sofia, Bulgaria
Sofia, Bulgaria
desislava.v.nikolova@gmail.com
amanolova@tu-sofia.bg
petya.petkova@tu-sofia.bg
Petia Georgieva
DETI/IEETA
University of Aveiro
Aveiro, Portugal
petia@ua.pt
AbstractThis paper presents an overview of recent methods for
recognition of human emotions based on Electrocardiogram (ECG)
signals and related applications. The major challenges in emotion
modeling (affective computing) from ECG data are finding
representations that are invariant to inter- and intra-subject
differences, as well as the inherent noise associated with the ECG
data recordings. The most common invariant features (in frequency
and time domain) extracted from the raw ECG signals are outlined.
The reviewed studies reveal the great potential of ECG to decode
basic human emotional states such as joy, sadness, anger, fear in
combination with other physiological signals and facial expression.
Major application areas cover patient monitoring, marketing, car
driving.
Keywords ECG, emotion recognition, machine learning, neural
networks, humanmachine interface, Cyber Physical Systems with
Human in the Loop
I. INTRODUCTION
The heart is one of the most critical organs in the human
body, and electrocardiography (ECG) is considered to be one
of the most powerful diagnostic tools in medicine that is
routinely used for the assessment of the functionality of the
heart. ECG being a physiological signal is used as the
conventional method for noninvasive interpretation of the
electrical activity of the heart in real time. But is usefulness
not only in analyzing the heart’s activity it can be also used for
emotion recognition. Some of the physiological signals are
highly used for classifying the human emotional state are:
electroencephalogram (EEG), electrocardiogram (ECG),
electromyogram (EMG), electrooculogram (EoG), skin
conductive resistance (SCR), skin temperature (ST), and
respiration rate (RR). Among these physiological signals, ECG
and EMG play a vital role in developing portable, non-
intrusive, reliable, and computationally efficient emotion
recognition systems. Understanding human emotions using
physiological signals is one of the active research areas on
developing intelligent humanmachine interface (HMI)
systems and Cyber Physical Systems with Human in the Loop.
One of the advantages of recognizing emotions and feelings
using physiological signals is that these are unconscious
responses of the human body, and therefore are very difficult
to conceal.
The electrical cardiac signals are recorded by an external
device, by attaching electrodes to the outer surface of the skin
of the patient’s thorax. These currents stimulate the cardiac
muscle and cause the contractions and relaxations of the heart
The electrical signals travel through the electrodes to the ECG
device, which records them as characteristic waves. Different
waves reflect the activity of different areas of the heart which
generate the respective flowing electrical currents.[1] On Fig
1. Is illustrated a schematic representation of a normal ECG
and its various waves, where P-waves represent atrial
depolarization, T-wave represents ventricular repolarization,
the first deflection in the complex, if it is negative, is called a
Q wave, the first positive deflection in the complex is called an
R wave, a negative deflection after an R wave is called an S
wave and finally U waves are thought to represent
repolarization of the Purkinje fibers [2].
Figure 1. Normal ECG with the relevant waves
Source: https://upload.wikimedia.org/wikipedia/commons/a/ae/Qrs.png
Emotion recognition from ECG signal has become an
important research topic in the field of affective computing.
The future of facial expression recognition is the multimodal
emotion sensing. An accurate and real-time emotion
recognition system cannot be based only on images and video.
It is important to use all the available modalities: voice, touch,
physiological signals etc.
Potential application of emotion recognition based on
physiological signals can be in the areas of marketing,
intelligent gaming, health care and Ambient Assisted Living
(AAL) [3]. One can only imagine the possibilities for health
care such as analysis of emotions or detection of changes in
mood as the disease progresses or as therapies kick in.
Marketing specialist could better gauge how viewers respond
to their products and ads by following the changes in the heart
rate. Smart cars might alter directions and stop if they perceive
that driver is upset, confused or angry.
In this paper, we introduce recent advances in research on
emotion recognition based on ECG signals. We examine the
state of-the-art results that have not been reviewed in previous
survey papers. The rest of this paper is organized as follows.
Section 2 describes the related scientific papers in the field.
Section 3 provides a detailed review of practical scenarios and
applications. Section 4 discusses some of the challenges and
opportunities in this field and identifies potential future
directions.
II. OVERVIEW OF ECG-BASED EMOTIONS RECOGNITION
METHODS
In this section we discuss the most successful recent
methods used to distinguish emotions extracted from the ECG
signals. Detecting emotions through ECG signal is beneficial
because the heart responses are involuntary and if we could
hide our emotions behind the face, it is difficult to mask
spontaneous heart reactions caused by emotions.
Emotion recognition procedure using ECG data undergoes
a number of steps as illustrated in Fig. 2. Appropriate selection
and combination of different machine learning methods
applied at step 3 and step 4 define the recognition success. The
addition step of feature selection (step 3.1) often is crucial for
the correct emotional recognition. The following review is
focused on steps 3, 3.1 and 4 as they have crucial implication
on the correctness of the recognition.
Fig. 2 ECG-based Emotion recognition process
Many affective computing methods are focused into binary
classification of two emotions - Joy and Sadness. Most
typically frequency domain features are extracted.
Continuously Wavelet Transform (CWT) and Discrete
Wavelet Transform (DWT) are used in [4, 5, 6] for feature
extraction. Afterwards feature selection methods were applied
in order to select the best correlated features for the emotion
recognition. Binary Particle Swarm Optimization (BPSO) and
Hybrid Particle Swarm Optimization (HPSO) were combined
with Genetic Algorithm (GA) in [7] for achieving best
accuracy results of 92.10% for Joy, 100% for Sadness and
92.60% for Joy and Sadness in combination with HPSO.
Improved Binary Particle Swarm Optimization (IBPSO)
and Improved Genetic Algorithm (IGA) were applied in [8]
and K-Nearest Neighbor (KNN) and Fisher classifier have
been used to distinguish between Joy, Sadness and both
emotions, as the best performance is showed by the Fisher
classifier with 88.90%, 88.70% and 88.89% respectively.
Automatic location of P-QRS-T wave and Discrete
Wavelet Transform (DWT) defines the feature set in [9]. Tabu
Search Algorithm (TS) has chosen the best features
combinations to perform higher classification results. A
Combination of Fisher and KNN classifiers has been used for
distinguishing emotions, as well as to compare the
performance between KNN and Fisher-KNN classifiers.
Obtained results in distinguishing the two emotions Joy and
Sandiness with 81.29% and 90.63% recognition rate
respectively. Fisher-KNN classifier performed better
recognition rate of 85.78% than KNN classifier (75.85%).
A combination of time and frequency domain features has
been studied in [10]. Feature extraction was done by applying
non-linear transformation on the 1st derivative of ECG
(automatic location of QRS Complex) for feature extraction.
Frequency domain features were extracted by the Fast Fourier
Transform (FFT) of R-R, T-T and P-P heart rates. The best
feature combinations have been selected by the means of Tabu
search. Fisher Projection Classifier has been chosen to classify
the recognized emotions.
For distinguishing between three emotional states Joy, Anger
and Sadness, Local Pattern Description (LPD) methods are the
preferred method in many recent papers. Local Binary Pattern
(LBP) and Local Ternary Pattern (LTP) are used in [11] for
feature extraction from the ECG Signal. To test and verify the
performance a 10-fold cross validation for KNN is applied. Using
LTP combined with KNN obtained better recognition accuracy
(Joy 85.75%; Anger 82.75%; Sadness 95.25%), than the
other combination LBP with KNN (Joy 85.75%; Anger
77.50%; Sadness 89.25%). The combination of 10-fold cross
validation with LPD methods has been studied to evaluate the
real-time emotion recognition from ECG using LTP. It
validates how much the real-time emotion recognition can
accurately detect user-experience emotions in real time
process.
The authors of [13] aim to assess five different human
emotions (happiness, disgust, fear, sadness, and neutral) using
heart rate variability (HRV) signals derived from an
electrocardiogram (ECG). The emotions were induced via
video clips on the 20 healthy students with age of 23 years old.
ECG signals were acquired using 3 electrodes and were
preprocessed using a Butterworth 3rd order filter to remove
noise and baseline wander. The Pan-Tompkins algorithm was
used to derive the HRV signals from ECG. Discrete wavelet
transform (DWT) was used to extract statistical features from
the HRV signals using four wavelet functions: Daubechies6
(db6), Daubechies7 (db7), Symmlet8 (sym8), and Coiflet5
(coif5). The k-nearest neighbor (KNN) and linear discriminant
analysis (LDA) were used to map the statistical features into
corresponding emotions. KNN provided the maximum average
emotion classification rate compared to LDA for five emotions
(sadness − 50.28%; happiness − 79.03%; fear − 77.78%;
disgust − 88.69%; and neutral − 78.34%).
The non-linear approach proposed in [14] distinguishes six
emotions - happiness, sadness, fear, disgust, surprise and
neutral induced by audio visual stimuli. The Hurst feature is
computed using Rescaled Range Statistics (RRS) and Finite
Variance Scaling (FVS) methods. Then new ones are proposed
as a combination of existing RRS and FVS with Higher Order
statistics (HOS). The Emotional State classification is made
through Bayesian Classifier, Regression Tree, KNN and Fuzzy
K-nearest neighbor. The results showed that RRS and FVS
methods have similar classification accuracy, while the
features obtained by combination of FVS and HOS performed
better for classifying the six emotional states using random and
subject independent validation respectively.
The ECG signals are also used in combination with other
physiological signals such as skin conductance (SC), abdomen
expansion, blood volume pulse (BVP) and skin temperature in
the context of collaboration with a partner [14]. During the
communication people tend to react to the partner’s emotion
through mechanisms of empathy and emotion contagion.
A combination of ECG and Galvanic Skin Responses
(GSR) were collected and analyzed in [15] through three
dictionaries, including Coiflets wavelet (Coif5) at level 14,
Daubechies wavelet (db4) at level 8, and discrete cosine
transform (DCT). Matching pursuit coefficients were
calculated from the normalized GSR and ECG signals and
Principal Component Analysis (PCA), Linear Discriminant
Analysis, and Kernel PCA were applied for dimensionality
reduction and feature selection. Probabilistic Neural Network
(PNN) was applied, in subject dependent and subject
independent modes, to classify emotional states in two-
dimensional (valence-arousal) emotion space.
An automated emotion recognition approach based on
different bio signals is proposed in [16]. EMG, ECG, RR, and
electro dermal activity (EDA) are processed and evaluated. In
this case Support Vector machine (SVM) and Adaptive Neuro-
Fuzzy Inference System (ANFIS) have been used for
classification of four emotional states high stress, low stress,
disappointment and euphoria. The overall classification rates
achieved by using 10-fold cross validation are 79.3% and
76.7% for the SVM and the ANFIS, respectively.
In [17] the authors use ECG and EEG for analysis of
emotional properties in this case passive valence /arousal model.
They propose s solution based on the short Fourier transform for
the recognition of dynamically developing emotion patterns on
ECG and EEG. Features extractions that are used in this paper are
Kernel Density Estimation (KDE) and Mel-frequency cepstral
coefficients (MFCC). The classifier employed in this work is
Multi-layer Perceptron (MLP), classification features are based on
the valence and arousal. The results show that the ECG signal has
direct relationship with the arousal factor rather than the valence
factor.
A very recent research paper [18] deals with detection of
emotions from ECG and EDA signals. The raw signals are
directly fed into deep neural networks that perform the so-
called end-to-end learning of the emotion. The motivation
behind this idea is that, the network learns an intermediate
representation of the raw input that better suits the task at
hand, and hence leads to improved performance as the authors
prove by comparing the results with the challenging REmote
COLlaborative and Affective database.
Table 1 summarizes the most common methods used for
emotions recognition. Note that the feature selection step
reduces the complexity and the processing time of the
classification step.
TABLE I.
OVERWVIEW OF METHODS OF ECG-BASED EMOTIONS
RECOGNITION
Feature selection
Classifier
Continuously
Binary Particle
Genetic Algorithm -
Wavelet Transform
Swarm Optimization
(BPSO)
Hybrid Particle
k-Nearest Neighbor
Swarm Optimization
(KNN)
(HPSO)
Improved Binary
Fisher classifier
Particle Swarm
Optimization
(IBPSO)
Improved Genetic
Algorithm (IGA)
Discrete Wavelet
Daubechies6 (db6)
k-Nearest Neighbor
Daubechies7 (db7),
(KNN)
Transform (DWT)
Symmlet8 (sym8)
Linear Discriminant
Coiflet5 (coif5).
Analysis (LDA)
Tabu Search
Combination of
Algorithm (TS)
KNN and Fisher
Local Patern
Local Binary Pattern
k-Nearest Neighbor
Description (LPD)
(LBP)
(KNN)
methods
Local Ternary Pattern
10-fold Cross-
(LTP)
validation and KNN
Hurst
Rescaled Range
Bayesian Classifier
Statistics (RRS)
Regression Tree
Finite Variance
K- Nearest Neighbor
Scaling (FVS)
Fuzzy K-nearest
Higher Order
neighbor
statistics (HOS)
Fast Fourier
Fisher Projection
Transform (FFT)
Tabu search (TS)
Classifier
Non-linear
Transformation on
ECG
Adaptive Neuro-
Fuzzy Inference
Statistical features
System (ANFIS)
Support Vector
Machine (SVM)
Convolutional
Neural Network,
Recurrent Neural
Network, Fully
Raw signal
Connected Layer
Discrete Cosine
Principal Component
Probabilistic Neural
Transform (DCT)
Analysis (PCA)
Network (PNN)
Linear Discriminant
Analysis (LDA)
Kernel PCA
III. ECG BASED EMOTION RECOGNITION APPLICATIONS
Emotion modeling and recognition has drawn extensive
attention from disciplines such as psychology, cognitive
science, and, lately, engineering. Emotion recognition is part
of the Affective computing - a domain that focuses on user
emotions while interacting with computers and applications
[19] The methods summarized in Table 1 are subject to diverse
applications. In this section are discussed the most underlying
approaches.
There are alternative ways to identify human emotional states,
however the ECG signal is known to provide more realistic
results. Fig. 3 represents the applications of emotions
recognition together with their core sources.
Fig. 3 Emotion Recognition applications
Most applications use a combination of different data
sources. The most common combinations are biological
signals and facial expression. Powerful machine learning
algorithms are able to achieve very high recognition rates both
in subject dependent [20] and in subject independent emotion
recognizer’s cases, [21]. In this section are considered major
application areas and more specifically the ones that use ECG
signal as a main source of data for emotions recognition.
A. Monitoring
Real time ECG data processing can be used during car races.
The emotions and psychological situation generally affect the
driver’s behavior and reactions, which can lead to accidents.
The authors of [16] make a study on the car’s setting and
development, based not only on subjective questionnaires
filled by the driver but also on the driver’s emotional state
which may affect the car’s performance. The car performance
have an impact on the driver’s emotional state, therefore the
observer can give additional advice and guidance.
The proposed system can be also used in specific medical
applications. It can support clinical diagnosis related to cases
when the patient’s capability to feel and express emotions is
limited or totally absent. In some clinical treatments, such as
Parkinson’s disease, Huntington’s disease, cortical lesion, stroke,
doctors need to know the psychological condition of the
patients. The goal is to evaluate the emotional state of the
patient. However, the use of specific drugs temporarily
normalizes or even decreases the facial muscular activity.
Using the proposed system, the response of the patient to the
specific drugs, adjusting and optimizing the dosages
prescribed, can be achieved.
B. Medicine
The ECG-based human emotion recognition is helpful in
applications involving patients with autism and other
intellectual disabilities and human computer interaction. It is
useful due to the fact that ECG signals are an activity of the
autonomic nervous system (ANS) and reflect the underlying
true emotional state of a person [11].
The goal in [22] is to recognize emotion states at each
moment and not to detect several prototypic emotions. The
authors make a conclusion that the present work still cannot
meet the real life needs because we exhibit non-basic, subtle
and rather complex emotional states, which cannot be fully
expressed by one category emotion label [23].
The health care system proposed in [21] focuses on
emotional aspects and integrates facial expression with ECG
signals for identifying users’ emotions and providing
appropriate services. The authors’ study showed that the real-
time emotion recognition from ECG signal is beneficial and
effective for detection and analysis of negative emotions and
providing immediate assistance.
C. Marketing
A web-based system to support the users in different
workspaces can provide access by web browsers and recognize
the user’s emotion. In [8] is presented a system that can
suggest taking a break if the emotions, detected by the webcam
and ECG sensors, are classified as negative.
The authors of [24] focus on two areas of application in
software engineering: usability testing and development
process improvement. Usability testing is defined through four
testing steps: 1) impression test which aims to differentiate the
user’s interest from boredom; 2) task-based approach
dedicated to help the users performing specific tasks and its
purpose is to differentiate frustration from empowerment; 3)
interaction test is based on free interaction with application,
which is supposed to evaluate overall user experience and its
objective is the distinction of engagement from
discouragement. 4) Comparative test combines the previous
tests. Development process improvement tried to verify the
hypothesis that emotions have significant impact on software
quality and developers’ productivity.
IV. CONCLUSIONS
The reviewed studies reveal the great potential of ECG to
decode basic human emotional states such as joy, sadness,
anger, fear, etc. However, the current state of the art in
affective computing reveals also that further research is
required in order to achieve realistic recognition of complex
human emotions.
Major application domains are in medicine, monitoring and
marketing. Yet, the combination of ECG with other
physiological signals and facial expression is far more
promising approach to follow.
Understanding the non-verbal cues from people to infer the
emotional states of others is central to our daily social
interactions from very early in life [25]. We as humans are
capable to distinguish happiness and stress as early as babies
and we get better with age to recognize sadness, surprise,
anger and disgust. For the machines to understand human
emotion is totally different task. It is difficult to describe an
emotion with quantitative characteristics. As a result of
improvements in wearable sensor technologies and artificial
intelligence techniques, machines are poised to take a big leap
in their ability to understand humans from their facial
expressions, physiological signals, gestures, gait and posture.
In [26] the authors give a very accurate description of
the issues for the future of emotion recognition such as the
multimodal emotion sensing. Emotion recognition cannot be
based only on images and video it is important to use all the
available modalities: voice, touch, bio signals etc. One of the
main goals of this paper is to understand the challenges in
emotion recognition based on ECG signals in real life contexts
and continue the research in this direction.
ACKNOWLEDGMENT
This work was supported in part by the contract DN 07/22
15.12.2016 for research project: ”Self-coordinating and
Adaptive Wireless Cyber Physical Systems with Human in the
Loop “ of the Bulgarian Research Fund of the Ministry of
Education and Science.
REFERENCES
[1] "ECG- simplified. Aswini Kumar M.D." LifeHugger. Retrieved 11
February 2010.
[2] Konar, A. and Chakraborty, A., 2014. Emotion recognition: A pattern
analysis approach. John Wiley & Sons.
[3] Koleva, P., Tonchev, K., Balabanov, G., Manolova, A. and Poulkov, V.,
2015, October. Challenges in designing and implementation of an
effective ambient assisted living system. In Telecommunication in
Modern Satellite, Cable and Broadcasting Services (TELSIKS), 2015
12th International Conference on (pp. 305-308). IEEE.
[4] C. Jing, L. Guangyuan, H. Min, “The Research on Emotion Recognition
from ECG Signal,” 2009 International Conference on Information
Technology and Computer Science, pp.497-500.
[5] W. Wan-Hui, Q. Yu-Hui, L. Guang-Yuan, “Electrocardiography
Recording, Feature Extraction and Classification for Emotion
RecognitionI,” IEEE Computer Society, World Congress on Computer
Science and Information Engineering, 2009, pp. 168172.
[6] S. Tivatansakul, M. Ohkura, “Emotion Recognition using ECG Signals
with Local Pattern Description methods”, International Journal of
Affective Engineering, 2016, Vol.15 No2 pp.51-61
[7] Xu Ya, Liu Guang-Yuan, A Method of Emotion Recognition Based on
ECG Signal, IEEE Computer Society, International Conference on
Computational Intelligence and Natural Computing, 2009, pp. 202205
[8] Xu Ya, L. Guangyuan, H.Min, W. Wanhui, H. Xiting, Analysis of
Affective ECG Signals Toward Emotion Recognition, journal of
electronics(china), 2010, Vol.27 No.1, pp.9-14
[9] [9] M. Murugappan, S. Murugappan, B. Siao Zheng, “Frequency Band
Analysis of Electrocardiogram (ECG) Signals for Human Emotional
State Classification Using Discrete Wavelet Transform (DWT)”, Journal
of Physical Therapy Science, 2013, 25(7): pp. 753759
[10] [10] J. Selvaraj, M. Murugappan, K. Wan, S. Yaacob,” Classification of
emotional states from electrocardiogram signals: a non-linear approach
based on hurst”, BioMedical Engineering Online, 2013, PMCID:
PMC3680185, PMID: 23680041
[11] G. Chanel, S.Avry, G. Molinari, M. Bétrancourt, T. Pun., “Multiple
users’ emotion recognition: improving performance by joint modeling of
affective reactions”, 2017 Seventh International Conference on Affective
Computing and Intelligent Interaction (ACII), pp.92-97
[12] Goshvarpour, A., Abbasi, A. and Goshvarpour, A., 2018. An accurate
emotion recognition system using ECG and GSR signals and matching
pursuit method. Biomedical Journal.
[13] S. Tivatansakul, M.Ohkura, Emotion Recognition Using ECG Signals
With Local Pattern Description Methods”, 2016, International Jornal of
Affective Engineering Vol.15 No.2 (Special Issue), pp. 51 61
[14] Picard R (2010) Affective computing: From laughter to IEEE. IEEE
Transactions on Affective Computing 1 pp. 11-17
[15] F. Agrafioti, D. Hatzinakos, A. K. Anderson, “ECG Pattern Analysis for
Emotion Detection”, IEEE Transactions on affective computing, vol. 3,
no. 1, january-march 2012, pp. 102-115
[16] C. D. Katsis, N. Katertsidis, G. Ganiatsas, D. I. Fotiadis, “Toward
Emotion Recognition in Car-Racing Drivers: A Biosignal Processing
Approach”, IEEE Transactions on systems, man, and cybernetics—part
a: systems and humans, vol. 38, no. 3, may 2008, pp. 502-512
[17] Amjad M.R. Alzeer Alhouseini, Imad Fakhri Al-Shaikhli, Abdul Wahab
bin Abdul Rahman, Mariam Adawiah Dzulkifli, Emotion Detection
Using Physiological Signals EEG & ECG, International Journal of
Advancements in Computing Technology(IJACT) Volume8, Number3,
June 2016, pp.103-112
[18] Keren, G., Kirschstein, T., Marchi, E., Ringeval, F. and Schuller, B.,
2017, July. End-to-end learning for dimensional emotion recognition
from physiological signals. In Multimedia and Expo (ICME), 2017 IEEE
International Conference on (pp. 985-990). IEEE.
[19] Selvaraj, J., Murugappan, M., Wan, K. and Yaacob, S., 2013.
Classification of emotional states from electrocardiogram signals: a non-
linear approach based on hurst. Biomedical engineering online, 12(1),
p.44.
[20] L. Chao, J. Tao, M. Yang, “Long Short Term Memory Recurrent Neural
Network based Multimodal Dimensional Emotion Recognition”,
AVEC'15, October 26 2015, Brisbane, Australia, ACM. ISBN 978-1-
4503-3743-4/15/10, pp. 65-72
[21] H. Gunes and M. Pantic, Automatic, dimensional and continuous
emotion recognition,” Int. J. Synthetic Emotions, vol. 1, no. 1, pp. 68-99,
2010.
[22] S. Tivatansakul, M. Ohkura, “Healthcare system focusing on emotional
aspect usinf augmented reality” Transaction of japan Society of Kansei
Engineering, 13(1), pp. 191-201, 2014
[23] Kolakowska, A.; Landowska, A.; Szwoch, M.; Szwoch, W.; Wrobel,
M.R.: “Emotion recognition and its applications”, W: Human-Computer
Systems Inter-action. Backgrounds and Applications vol. 3, Advances in
Soft Computing se-ries, Springer-Verlag Co, pp. 51-62.
[24] O Georgieva, S Milanov, P Georgieva, IM Santos, AT Pereira, CF Silva,
Learning to decode human emotions from event-related potentials,
Neural Computing and Applications 26 (3), 573-580, 2015.
[25] Helen Rodger, Luca Vizioli, Xinyi Ouyang, Roberto Caldara, “Mapping
the development of facial expression recognition”, Developmental
Science (2015), pp 114, DOI: 10.1111/desc.12281
[26] Hatice Gunesa, Hayley Hung, “Is automatic facial expression
recognition of emotions coming to a dead end? The rise of the new kids
on the block”, Image and Vision Computing Volume 55, Part 1,
November 2016, Pages 68
... Figure 3, the GEM method assessed the unique covariate points of a single individual under test. These points are distinct from the rest of the scattered random data [25]. After the given iterations, the error rate approaches a minimum; the points are extracted. ...
Article
Full-text available
p>Investigating human subjects is the goal of predicting human emotions in the stock market. A significant number of psychological effects require (feelings) to be produced, directly releasing human emotions. The development of effect theory leads one to believe that one must be aware of one's sentiments and emotions to forecast one's behavior. The proposed line of inquiry focuses on developing a reliable model incorporating neurophysiological data into actual feelings. Any change in emotional affect will directly elicit a response in the body's physiological systems. This approach is named after the notion of Gaussian mixture models (GMM). The statistical reaction following data processing, quantitative findings on emotion labels, and coincidental responses with training samples all directly impact the outcomes that are accomplished. In terms of statistical parameters such as population mean and standard deviation, the suggested method is evaluated compared to a technique considered to be state-of-the-art. The proposed system determines an individual's emotional state after a minimum of 6 iterative learning using the Gaussian expectation-maximization (GEM) statistical model, in which the iterations tend to continue to zero error. Perhaps each of these improves predictions while simultaneously increasing the amount of value extracted.</p
... Системи розпізнавання емоцій мають багато потенційних застосувань, охоплюючи охорону здоров'я, розваги, електронне навчання, маркетинг, моніторинг людей і безпеку. Згідно з [3], існувало три основних застосування систем розпізнавання емоцій конкретно за допомогою сигналів ЕКГ: ...
Article
Дана стаття зосереджена на аналізі та класифікації стратегій розпізнавання емоцій на основі фізіологічних сигналів, а також на дослідженні доступних баз даних для мультимодального аналізу психологічних станів з використанням інформації фізіологічних сигналів. У статті представлено класифікацію методів розпізнавання, проведено аналіз впливів факторів модальності і стратегії злиття на результати розпізнавання та визначено важливість вибору фізіологічних сигналів для точного розпізнавання емоцій. Також досліджено переваги моделей глибокого навчання та практичні застосування розпізнавання емоцій у реальних сценаріях.
... One of the main benefits of studying explainability techniques is that they provide insight into how the different features of a model contribute to its outcomes. These approaches are intuitive and can help to understand the decision-making process of a black box model and explain its behaviour, [23,24]. This systematic review also demonstrates the need for the involvement of medical stakeholders in the development process of any type o ML, Deep Learning (DL) or XAI methods for the medical domains. ...
Article
Advancement of technologies has enabled the creation of smart solutions to improve healthcare. The primary objective is to help healthcare practitioners make efficient and fast decisions and diagnoses. Vital signs provide meaningful information while monitoring patient health, which allows assessing patients’ emotional well-being and adjust therapy. This study introduces an AI-powered patient monitoring system and healthcare platform. The proposed healthcare platform uses Internet of Medical Things (IoMT) and machine learning to monitor and classify emotional states using ECG signals. Thus, a label transformation and Random Forest classifier are used to identify patient emotions as positive or negative based on selected features from Heart Rate Variability (HRV) provided by ECG sensor. Moreover, a data augmentation technique is used to expand the primary used dataset, DREAMER which is afterward integrated with SWELL and WESAD datasets to increase data variety. The model presents emotion identification on the platform with an accuracy of 99.89%, outperforming most existing related works.
Article
Full-text available
Emotions encompass physiological systems that can be assessed through biosignals like electromyography and electrocardiography. Prior investigations in emotion recognition have primarily focused on general population samples, overlooking the specific context of theatre actors who possess exceptional abilities in conveying emotions to an audience, namely acting emotions. We conducted a study involving 11 professional actors to collect physiological data for acting emotions to investigate the correlation between biosignals and emotion expression. Our contribution is the DECEiVeR (DatasEt aCting Emotions Valence aRousal) dataset, a comprehensive collection of various physiological recordings meticulously curated to facilitate the recognition of a set of five emotions. Moreover, we conduct a preliminary analysis on modeling the recognition of acting emotions from raw, low- and mid-level temporal and spectral data and the reliability of physiological data across time. Our dataset aims to leverage a deeper understanding of the intricate interplay between biosignals and emotional expression. It provides valuable insights into acting emotion recognition and affective computing by exposing the degree to which biosignals capture emotions elicited from inner stimuli.
Chapter
Automated emotion detection and analysis have become one of the most important domains in recent years because of its applicability in health care, education, entertainment industry, robotics, and marketing sectors. Emotion recognition gives an accurate estimation of the mental condition of a person and indicates the inherent activity or thinking state of the mind. In the field of emotion recognition, artificial intelligence-based analysis has become an essential part of research along with the domains of medical science, cognitive science, computer science, and neuroscience. Emotion of any person can be judged from the gesture, face recognition, and body movements. But these are not conclusive enough because of the ability of a person to suppress these responses at will. The present paper presents a computationally simple approach to estimate and classify the emotional states of happy and sad based on a single feature calculated from a single lead PPG signal available in DEAP dataset. Classification is carried out using a binary classification rule and does not require any complex classifier. Results show the justification of the proposed feature and the validity of the approach. The present method may be applied in real-time systems for the detection of sadness or mental state of any individual owing to its simplicity in operation and implementation.
ResearchGate has not been able to resolve any references for this publication.