Figure 2 - uploaded by Chip-Jin Ng
Content may be subject to copyright.
The red dots are the 68 facial landmarks tacked for each image. The action units are the ones being indicative of pain in the past literature. Lastly, it shows the various parameterization of the 68 facial landmarks that we compute as video features to be used in this work. Facial Action Coding System photos(http://www.cs.cmu.edu/ face/facs.htm)
Contexts in source publication
Context 1
... i.e., an instance of constrained local model, essentially involves three major technical components: point distribution model (describing the position of feature points in an image), local neural field patch experts (layered unidirectional graphical model), and optimization fitting approach (non-uniform regularized landmark mean shift fitting). By applying CLNF, we then are able to track the 68 feature points (Figure 2), e.g., around face, eyes, and nose contour, for each patient in each image of the recorded video session. ...
Context 2
... Audio-Only Video-Only Multimodal Fusion (early-fusion / late-fusion) (Figure 2). In this work, instead of recognizing these facial action units, we compute features characterizing these expressions directly from the tracked key points' (x, y) position, • Eyebrows (7): the distance of inner eyebrows divided by the distance of outer eyebrows (1), the quadratic polynomial coefficients of the right and the left eyebrows (6) • Nose (2): the normalized distance between nose and philtrum (1), and of nasolabial folds (1) • Eyes (5): the outer eye corner opening (2), the distance between the inner eye corners divided by the distance of outer eye corners (1), the distance of upper and lower eyelids divided by the distance from the head to the corner of the eyes (2) • Mouth (14): the quadratic polynomial coefficients from the shape of upper lip, including outer and inner part, and lower lip, including outer and inner part (12), the two-sided mouth corners opening angles (2) There are a total of 28 features per frame extracted from the face to represent the facial expression of the patient. ...
Context 3
... this work, instead of recognizing these facial action units, we compute features characterizing these expressions directly from the tracked key points' (x, y) position, • Eyebrows (7): the distance of inner eyebrows divided by the distance of outer eyebrows (1), the quadratic polynomial coefficients of the right and the left eyebrows (6) • Nose (2): the normalized distance between nose and philtrum (1), and of nasolabial folds (1) • Eyes (5): the outer eye corner opening (2), the distance between the inner eye corners divided by the distance of outer eye corners (1), the distance of upper and lower eyelids divided by the distance from the head to the corner of the eyes (2) • Mouth (14): the quadratic polynomial coefficients from the shape of upper lip, including outer and inner part, and lower lip, including outer and inner part (12), the two-sided mouth corners opening angles (2) There are a total of 28 features per frame extracted from the face to represent the facial expression of the patient. Figure 2 also shows a schematics of the features being extracted in this work. ...
Similar publications
This paper describes a study of neural networks without weights application for recognizing emo- tions. Task is performed from facial expressions image analysis. Eight situations were considered: anger, con- tempt, disgust, fear, happiness, neutral, sadness, and surprise. Experiments were made from TFEID (Taiwanese Facial Expression Image Database)...
![Figure 4. Original layout and generated heat map of Auckland hospital [32]](profile/Jiamou-Liu/publication/346266898/figure/fig3/AS:961857934327815@1606336210270/Original-layout-and-generated-heat-map-of-Auckland-hospital-32_Q320.jpg)
Citations
... Taking images or videos requires special equipment and a prepared environment to ensure the quality and clarity of the frames. Still, detection by facial expressions could be useful in emergency VOLUME 11, 2023 waiting rooms to replace the traditional triage classification system [39] 2) Linguistic analysis Natural language processing (NLP) is a type of machine learning that allows for the analysis and processing of freeform text. When applied to medical records, it can assist in predicting patient outcomes, improve emergency triage systems, and develop a conversational chatbot for patients to ask questions and obtain relevant information [40]. ...
Pain assessment traditionally relies on self-report, but it is subjective and influenced by various factors. To address this, there's a need for an affordable and scalable objective pain identification method. Current research suggests that pain has physiological markers beyond the brain, such as changes in cardiovascular activity and electrodermal responses. Utilizing these markers, real-time pain detection algorithms were developed using the BioVid Heat Pain dataset, consisting of 86 healthy individuals experiencing acute pain. Three physiological signals were collected (ECG, GSR, EMG). Various machine learning models were employed to lay the foundation for future advancements in creating sophisticated pain categorization algorithms. The goal is to develop a machine learning model capable of accurately classifying levels of pain experienced based solely on physiological signals. The proposed method produced an accuracy score of 87% for binary classification and 52% accuracy for multi-class classification, with the highest-performing machine learning model being Random Forests. These results suggest that the PainMeter can be deployed in field settings using wearable sensors, offering real-time, unbiased pain sensing and management capabilities.
... In general, if the individual modalities demonstrate sufficiently strong predictive performance, their fusion tends to yield improved results. This has been demonstrated in various studies, including the combination of facial expression and head pose [19,20]; EDA, ECG, and sEMG [20,21]; video, EDA, ECG, and sEMG [20,22]; video, RSP, ECG, and remote PPG [23]; video and audio [24]; and MoCap and sEMG [25]. ...
Pain assessment is a critical aspect of healthcare, influencing timely interventions and patient well-being. Traditional pain evaluation methods often rely on subjective patient reports, leading to inaccuracies and disparities in treatment, especially for patients who present difficulties to communicate due to cognitive impairments. Our contributions are three-fold. Firstly, we analyze the correlations of the data extracted from biomedical sensors. Then, we use state-of-the-art computer vision techniques to analyze videos focusing on the facial expressions of the patients, both per-frame and using the temporal context. We compare them and provide a baseline for pain assessment methods using two popular benchmarks: UNBC-McMaster Shoulder Pain Expression Archive Database and BioVid Heat Pain Database. We achieved an accuracy of over 96% and over 94% for the F1 Score, recall and precision metrics in pain estimation using single frames with the UNBC-McMaster dataset, employing state-of-the-art computer vision techniques such as Transformer-based architectures for vision tasks. In addition, from the conclusions drawn from the study, future lines of work in this area are discussed.
... In contrast, the authors of [50] focused on acute MI pain and achieved the highest overall accuracy using their CNN model. Other studies [14,57] followed patients for one hour to assess their pain levels and conduct an additional audio session, suggesting that the nature of their pain was acute (Table 1). Accurate pain assessment is critical for effective pain management, and improvements in the precision of pain assessment can reduce the risk of incorrect management, especially in opioid prescription, which carries a risk of addiction and can pose a threat to patients' lives [8,58,59]. ...
... In 2016, Tsai et al. [57] conducted a study evaluating their framework against the NRS. They used the audio-video system in combination with the NRS and various measures (age, vital signs, pain levels, pain levels predicted using SVM models), as well as the two clinical outcome variables of painkiller prescription and disposition. ...
... The study indicated a correlation between the concurrent self-reported pain intensity and the quantifiable speech biosignal attributes, similar to the findings of the other two studies which demonstrated the effectiveness of prosodic low-level descriptors for spectral LLDs in identifying pain [14,57]. ...
Pain is a complex and subjective experience, and traditional methods of pain assessment can be limited by factors such as self-report bias and observer variability. Voice is frequently used to evaluate pain, occasionally in conjunction with other behaviors such as facial gestures. Compared to facial emotions, there is less available evidence linking pain with voice. This literature review synthesizes the current state of research on the use of voice recognition and voice analysis for pain detection in adults, with a specific focus on the role of artificial intelligence (AI) and machine learning (ML) techniques. We describe the previous works on pain recognition using voice and highlight the different approaches to voice as a tool for pain detection, such as a human effect or biosignal. Overall, studies have shown that AI-based voice analysis can be an effective tool for pain detection in adult patients with various types of pain, including chronic and acute pain. We highlight the high accuracy of the ML-based approaches used in studies and their limitations in terms of generalizability due to factors such as the nature of the pain and patient population characteristics. However, there are still potential challenges, such as the need for large datasets and the risk of bias in training models, which warrant further research.
... Several databases have been designed and released for automatic pain recognition methods in computer vision and machine learning domains, which ranked from oldest to newest: COPE Database [14], UNBC-McMaster Shoulder Pain Database [15], BioVid Heat Pain Database [16], BP4D-Spontaneous Database [17], YouTube Database [18], BP4D+ Database [19], IIIT-S ICSD [20], SenseEmotion Database [21], Multimodal EmoPain Database [22], Mint PAIN Database [23], and X-ITE Pain Database [24]. Most methods of pain recognition used single modality: [7,25] used video, [26,27] used audio signal, and [28][29][30] used physiological signals. The recent methods used multiple modalities [31][32][33] that can improve the performance and flexibility of pain recognition. ...
... Most methods of pain recognition used a single modality [44,50] used video, [51,52] used audio signal, and [10,12,33] used physiological signals. The recent methods used multiple modalities [3,4,14,16,[53][54][55] that can improve the performance and flexibility of pain recognition. ...
Pain is a reliable indicator of health issues; it affects patients’ quality of life when not well managed. The current methods in the clinical application undergo biases and errors; moreover, such methods do not facilitate continuous pain monitoring. For this purpose, the recent methodologies in automatic pain assessment were introduced, which demonstrated the possibility for objectively and robustly measuring and monitoring pain when using behavioral cues and physiological signals. This paper focuses on introducing a reliable automatic system for continuous monitoring of pain intensity by analyzing behavioral cues, such as facial expressions and audio, and physiological signals, such as electrocardiogram (ECG), electromyogram (EMG), and electrodermal activity (EDA) from the X-ITE Pain Dataset. Several experiments were conducted with 11 datasets regarding classification and regression; these datasets were obtained from the database to reduce the impact of the imbalanced database problem. With each single modality (Uni-modality) experiment, we used a Random Forest [RF] baseline method, a Long Short-Term Memory (LSTM) method, and a LSTM using a sample weighting method (called LSTM-SW). Further, LSTM and LSTM-SW were used with fused modalities (two modalities = Bi-modality and all modalities = Multi-modality) experiments. Sample weighting was used to downweight misclassified samples during training to improve the performance. The experiments’ results confirmed that regression is better than classification with imbalanced datasets, EDA is the best single modality, and fused modalities improved the performance significantly over the single modality in 10 out of 11 datasets.
... To mitigate the biases caused by the conventional evaluation methods, automatic evaluation of pain is potential to construct an objective and unified standard. In recent studies, models for automatic detection of pain have been investigated and proposed based on multiple modalities, including facial expression [184,185,186], body gestures, and motion descriptors [187,188]. As an important factor of evaluating the physiological health like the cardiovascular system [189] and the mental health such as depression [190], voice is potential to evaluate the pain level. ...
Automatically recognising audio signals plays a crucial role in the development of intelligent computer audition systems. Particularly, audio signal classification, which aims to predict a label for an audio wave, has promoted many real-life applications. Amounts of efforts have been made to develop effective audio signal classification systems in the real world. However, several challenges in deep learning techniques for audio signal classification remain to be addressed. For instance, training a deep neural network (DNN) from scratch is time-consuming to extracting high-level deep representations. Furthermore, DNNs have not been well explained to construct the trust between humans and machines, and facilitate developing realistic intelligent systems. Moreover, most DNNs are vulnerable to adversarial attacks, resulting in many misclassifications. To deal with these challenges, this thesis proposes and presents a set of deep-learning-based approaches for audio signal classification. In particular, to tackle the challenge of extracting high-level deep representations, the transfer learning frameworks, benefiting from pre-trained models on large-scale image datasets, are introduced to produce effective deep spectrum representations. Furthermore, the attention mechanisms at both the frame level and the time-frequency level are proposed to explain the DNNs by respectively estimating the contributions of each frame and each time-frequency bin to the predictions. Likewise, the convolutional neural networks (CNNs) with an attention mechanism at the time-frequency level is extended to atrous CNNs with attention, aiming to explain the CNNs by visualising high-resolution attention tensors. Additionally, to interpret the CNNs evaluated on multi-device datasets, the atrous CNNs with attention are trained in the conditional training frameworks. Moreover, to improve the robustness of the DNNs against adversarial attacks, models are trained in the adversarial training frameworks. Besides, the transferability of adversarial attacks is enhanced by a lifelong learning framework. Finally, the experiments conducted with various datasets demonstrate that these presented approaches are effective to address the challenges.
... Depending on the amount and diversity of sensors used during the data collection phase, several signals have been assessed and evaluated in various settings for the development of pain assessment systems. Some of the most prominently used signals constitute of the audio signal (e.g., paralinguistic vocalizations) (Tsai et al., 2016(Tsai et al., , 2017Thiam and Schwenker, 2019), the video signal (e.g., facial expressions) (Rodriguez et al., 2017;Werner et al., 2017;Tavakolian and Hadid, 2019;Thiam et al., 2020b), specific bio-physiological signals such as the Electrodermal Activity (EDA), the Electrocardiogram (ECG), the Electromyography (EMG), or the Respiration (RSP) signal Campbell et al., 2019;Thiam et al., 2019a), and also bodily expression signals (Dickey et al., 2002;Olugbade et al., 2019;Uddin and Canavan, 2020). ...
Traditional pain assessment approaches ranging from self-reporting methods, to observational scales, rely on the ability of an individual to accurately assess and successfully report observed or experienced pain episodes. Automatic pain assessment tools are therefore more than desirable in cases where this specific ability is negatively affected by various psycho-physiological dispositions, as well as distinct physical traits such as in the case of professional athletes, who usually have a higher pain tolerance as regular individuals. Hence, several approaches have been proposed during the past decades for the implementation of an autonomous and effective pain assessment system. These approaches range from more conventional supervised and semi-supervised learning techniques applied on a set of carefully hand-designed feature representations, to deep neural networks applied on preprocessed signals. Some of the most prominent advantages of deep neural networks are the ability to automatically learn relevant features, as well as the inherent adaptability of trained deep neural networks to related inference tasks. Yet, some significant drawbacks such as requiring large amounts of data to train deep models and over-fitting remain. Both of these problems are especially relevant in pain intensity assessment, where labeled data is scarce and generalization is of utmost importance. In the following work we address these shortcomings by introducing several novel multi-modal deep learning approaches (characterized by specific supervised, as well as self-supervised learning techniques) for the assessment of pain intensity based on measurable bio-physiological data. While the proposed supervised deep learning approach is able to attain state-of-the-art inference performances, our self-supervised approach is able to significantly improve the data efficiency of the proposed architecture by automatically generating physiological data and simultaneously performing a fine-tuning of the architecture, which has been previously trained on a significantly smaller amount of data.
... With the release of publicly available pain databases (e.g., the UNBC-McMaster Pain Archive) and advancements in computer vision and machine learning, automatic assessment of pain from behavioral measures (e.g., facial expression) has emerged as a possible alternative to manual observations [10]. Using either spatial features or spatio-temporal features [10], researchers have automatically detected pain in the flow of behavior [1,16,19], differentiated feigned from genuine pain [2,16,17], detected ordinal pain intensity [11,12,15,[22][23][24][25][26][27]29] and distinguished pain from expressions of emotion [3,13,14] (see [10,28] for a detailed review). ...
The standard clinical assessment of pain is limited primarily to self-reported pain or clinician impression. While the self-reported measurement of pain is useful, in some circumstances it cannot be obtained. Automatic facial expression analysis has emerged as a potential solution for an objective, reliable, and valid measurement of pain. In this study, we propose a video based approach for the automatic measurement of self-reported pain and the observer pain intensity, respectively. To this end, we explore the added value of three self-reported pain scales, i.e., the Visual Analog Scale (VAS), the Sensory Scale (SEN), and the Affective Motivational Scale (AFF), as well as the Observer Pain Intensity (OPI) rating for a reliable assessment of pain intensity from facial expression. Using a spatio-temporal Convolutional Neural Network - Recurrent Neural Network (CNN-RNN) architecture, we propose to jointly minimize the mean absolute error of pain scores estimation for each of these scales while maximizing the consistency between them. The reliability of the proposed method is evaluated on the benchmark database for pain measurement from videos, namely, the UNBC-McMaster Pain Archive. Our results show that enforcing the consistency between different self-reported pain intensity scores collected using different pain scales enhances the quality of predictions and improve the state of the art in automatic self-reported pain estimation. The obtained results suggest that automatic assessment of self-reported pain intensity from videos is feasible, and could be used as a complementary instrument to unburden caregivers, specially for vulnerable populations that need constant monitoring.
... Apart from facial expressions, attempts have been made to use other modalities either individually or in combination for automatic pain detection. For example, Aung et al. [27] examined the use of body posture, body motion, and muscle activity for detecting patterns in body movements that could be indicative of pain during physical exercise; Tsai et al. [28] combined facial expressions and acoustic features to detect pain intensities in emergency cases; Werner et al. [29] investigated the use of facial expressions, electromyogram (EMG) recorded from trapezius muscles, and autonomic signals such as skin conductance and electrocardiogram (ECG) to automatically detect the different heat pain stimulus levels. Attempts have also been made to investigate the use of brain activation-acquired via either electroencephalography (EEG) [30] [31] or functional imaging [32]-for automatic pain assessment. ...
... Zhang et al. [76] averaged the probabilities of selected pain-related AUs to calculate the pain intensity estimate. [126] statistical features from sequence of facial landmark distances and quadratic polynomial coefficients of mouth shape Tsai et al. [28] bag of words from k-means based clusters of sequence of geometric features (facial landmark distances and quadratic polynomial coefficients of mouth shape) ...
... Tsai et al. [28] textural HOG from Three Orthogonal Planes (HOG-TOP) Chen et al. [75] LBP from Three Orthogonal Planes (LBP-TOP) Kaltwang et al. [113] combinations of LBP-TOP, LPQ-TOP, BSIF-TOP Yang et al. [104] energy from optical flow Ghasemi et al. [74] time [130] Almost all classification and regression tasks were supervised. Ground truth in the form of pain or AU labels, and discrete or continuous-valued pain or AU intensities, were used to train the machine learning models. ...
Pain sensation is essential for survival, since it draws attention to physical threat to the body. Pain assessment is usually done through self-reports. However, self-assessment of pain is not available in the case of noncommunicative patients, and therefore, observer reports should be relied upon. Observer reports of pain could be prone to errors due to subjective biases of observers. Moreover, continuous monitoring by humans is impractical. Therefore, automatic pain detection technology could be deployed to assist human caregivers and complement their service, thereby improving the quality of pain management, especially for noncommunicative patients. Facial expressions are a reliable indicator of pain, and are used in all observer-based pain assessment tools. Following the advancements in automatic facial expression analysis, computer vision researchers have tried to use this technology for developing approaches for automatically detecting pain from facial expressions. This paper surveys the literature published in this field over the past decade, categorizes it, and identifies future research directions. The survey covers the pain datasets used in the reviewed literature, the learning tasks targeted by the approaches, the features extracted from images and image sequences to represent pain-related information, and finally, the machine learning methods used.
... Pain evaluation [3], [4] is to collect all the information related to pain, such as Physical senses, severity, physical examination results Related special examination results Including the results of various treatments That has been before Interpret the results to diagnose the cause and mechanism of pain. Because each type of pain responds differently to painkillers. ...
This research is to design and build a system for managing and monitoring drug use for pharmacies based on the principles of the database system and database connection with the program for drug administration and monitoring consists of a drug inventory management system storefront system for pharmacies and drug use tracking system using pain score. The results of the research can be designed to manage and monitor drug use systems for pharmacies. This system can function according to the design and construction objectives in all respects.
