AAM based tracking of landmark coordinates for subject 52 in the CK + database (©Jeffrey Cohn). From left to right: Onset (1st frame), half-apex (5th frame) and apex 

Contexts in source publication

Context 1

... m represents the number of parameters in the statistical model and L m ( K ) denotes the maximized value of the log likelihood function. Several K values are tested, and the value that minimizes the expression (8) is selected. Once a GMM is fit to the set of neutral face shapes, the neutral face shape dictionary is formed from the mean vectors μ k , k = 1 , . . . , K of each of the K Gaussian mixture components. The covariance matrices Σ k will represent the variation of the face shapes around the mean shapes. In summary, the steps of the dictionary estimation are as follows, the flowchart of which is shown in Fig. 6. 1: Input: All neutral frames in the database, and the CBF features that represent the shape for each face. 2: Perform rigid alignment on all neutral face shapes to minimize in-plane rotation, scale and translation differences. 3: Fit a Gaussian mixture model (GMM) to the aligned shapes, where the number of mixture densities K is determined automatically using Akaike’s information criterion. 4: Output: The mean vectors ( μ k ) and the covariance matrices ( Σ k ) of the K mixture densities form the dictionary of neutral face shapes. In this section we first explain how the best fitting neutral face shape is selected from the pre-trained dictionary for a given expressive face. Then, the estimated neutral face shape is used to extract the motion vectors of facial landmarks, which will be the geometric features to be used for facial expression recognition. Next, we propose an efficient method to fuse the geometric features with appearance based features, the details of which are presented below. Given an expressive face image, such as the last frame of a video clip shown in Fig. 2, we want to decompose the shape vector of the expressive face, which is shown by the black landmarks, into two components. That is, we assume that an expressive shape vector s n , i that belongs to image i of n th sequence can be decom- posed as ...

Context 2

... convey when we are trying to express ourselves with words. Therefore, facial expressions are very important in human-to-human communication to enhance our messages and to display emotions. It is expected that facial expression based emotion recognition will be a part of many human–computer interaction scenarios in the future [1]. Below, we first give a brief overview of the state-of-the-art on facial expression recognition. Then, we state and motivate the problem that we address in this paper. There is a plethora of recent works on automatic facial expression and emotion recognition, some of which have been high- lighted in several survey papers [2–6]. Automatic facial expression recognition is still a challenging computer vision problem. The main issues to tackle while developing a facial expression recognition algorithm are: i) choosing an emotion model ii) extracting suitable facial features iii) choosing a classification method and iv) choosing a suitable database to report facial expression recognition results. Below, we give an overview of each of them briefly. There are two emotion models used while recognizing emotions from facial expressions: the categorical model and the dimensional model . In the categorical model, the facial expression is recognized as one of the distinct labels from a set of emotions. A popular categorization of emotions consists of six basic emotions: happiness, sadness, anger, fear, surprise and disgust, which have been suggested by Ekman and Friesen and have been shown to be uni- versal [7]. Recently, dimensional and continuously labeled emotion recognition has also become popular that use a space with three dimensions [4]. The first dimension ( evaluation or valence ) measures pleasure–displeasure (positive-negativeness) attribute of the emotion. The second dimension activation or arousal measures the likelihood of a person to take an action in the emotional state and it ranges from sleepiness to extreme excitement [4]. The third dimension dominance or power represents the sense of control over the emotion. Using the dimensional model, it is possible to an- notate a facial expression on a continuous scale rather than a categorical scale to describe the wide range of emotions that we face in our daily lives. On the other hand, the dimensional model is not as intuitive as the categorical model [8]. In this paper, we address the problem of categorical emotion recognition. The facial features that can be extracted from a face image generally falls into one of two categories: geometrical features or appearance based features . Geometrical features deal with the shape and motion of facial landmarks located around salient regions such as eyelids, eyebrows and lips. Appearance-based features on the other hand, can quantify texture variations due to facial expressions such as wrinkles between the eyebrows and the forehead. It has been shown that using geometric and appearance features together improve the facial expression recognition performance [6]. A varying number of landmarks on the face can be detected and tracked to serve as geometrical features . Various methods have been used for this task including active appearance models [9,10], constrained local models [11–13], GentleBoost [14] and mixtures of trees [15]. Some appearance based features that have been shown to be effective for facial expression recognition (FER) are Gabor filters [14], local binary patterns (LBP) [16–19], local phase quantization features [17,20], and the scale-invariant feature transform (SIFT) [21,22]. Many different classifiers have been utilized for FER. Sup- port vector machines (SVMs) [14,17,18,23,24] are among the most popular ones. Other classifiers used for FER include neural networks [25], sparse representation classifiers [26,27], hidden Markov models [28], conditional random fields [29,30] and Bayesian networks [31]. Some efforts have also been reported that propose novel classifiers. In [32], Chew et al. propose a novel su- pervised classifier namely the modified correlation filter (MCF), which is shown to have superior generalization performance due to its robustness to noisy training examples. In order to test and compare automatic facial expression recognition algorithms, emotional databases which are open to researchers are needed [6]. There are a number of 2D [6,10,33–35] and 3D [36–39] facial expression databases in the literature. In this work, we utilize three widely-used facial expression databases: CK + [10], MMI [34] and eNTERFACE [40]. The Cohn–Kanade (CK) database [41] has been a very popular one, which consists of facial clips containing the six basic emotions as shown in Fig. 1. The CK database has recently been extended to include more subjects, a new facial expression class (contempt) and facial tracking data and is called the CK + database [10]. The face tracking data provided with the CK + database consists of the locations of 68 facial landmarks, which can be seen in Fig. 2. In the CK + database, there are a total of 123 subjects and 327 sequences with emotion labels. The emotions were posed in a laboratory by the participants who had different ages ranging from 18 to 50 and 69% of them were female, 81% Euro-American, 13% Afro-American and 6% from other ethnic groups. The first frame of each sequence in CK + database shows a neutral expression (onset frame) and the last frame shows an expression at its apex (peak frame) as can be observed in Fig. 2. The seven emotion categories in the database are happiness, sadness, anger, fear, disgust, surprise , and contempt . Another database that we used in this study is the MMI database [33,34]. This database contains recordings of full temporal patterns of facial expressions covering neutral, onset, apex and off- set phases. The facial expressions consist of the six basic emotions as well as expressions with a single action unit, which were posed in a laboratory environment using either natural lighting or two high intensity lamps with reflective umbrellas. Some naturalistic expressions have also been added to the database [34]. There are currently 75 subjects in the whole database aged between 19–62, who have European, Asian or South American ethnic backgrounds. The third database that we used in this work is eNTER- FACE’05 [40], which is an audio–visual database. We used this database to test the generalization potential of the proposed method to less constrained data containing lip motion due to speech. The eNTERFACE database contains a total of 1287 audio– visual clips of 44 subjects (81% men 19% women) from 14 different nationalities. The database contains recordings of sentences ut- tered in English in an acted way, reflecting the six basic emotions, i.e., happiness, sadness, anger, fear, disgust and surprise. Each clip is expected to reflect one of these emotions from the beginning to the end. Therefore, neutral expression does not exist in the database. The visual component of eNTERFACE dataset is quite challenging since the actors are not professional and some emotions are very hard to detect visually, even for humans (e.g. fear). Moreover, 31% of the subjects wear glasses and 17% have beard. An expressive face image carries information about both the facial expression, and identity (including age and gender) of that person. If shape-based features are used to describe the locations of salient points (around eyes, lips etc.), suppressing the identity- related information has been shown to increase the facial expression recognition rate. One way to do this is to subtract the baseline [4,42], that is, the neutral face shape of the person from the expressive shape, and to work with the “motion” information. This solution is feasible only if the neutral face image (or shape) is available, for example as in the CK + database, which is always the first frame of a sequence [10]. If the neutral face image is not available, one approach [43] uses averaging all the frames of a sequence assuming that the average will give us an estimate of the neutral face shape. However, this assumption does not hold if all the frames of a video are strongly expressive. In this work, we propose a general solution to the baseline problem described in the previous paragraph. That is, we present a general method for estimating the neutral (i.e. expressionless) face shape for a given expressive face. First, a dictionary of neutral face shapes is generated. Then, the best fitting dictionary entry is selected as the estimated neutral shape for a given expressive face, which is expected to model the identity-related component of the expressive face. We also propose an efficient method of fusing geometric and appearance based features. Stand-alone and cross- database experiments on the CK + , MMI and eNTERFACE databases demonstrate that the proposed method increases the accuracy of facial expression recognition significantly, when the actual neutral face image is unavailable. In Section 2, the method proposed for estimation of a neutral face shape dictionary using Gaussian Mixture Models is presented. In Section 3, the facial expression recognition method, which uses the neutral shape dictionary is explained in detail. The fusion of geometric and appearance based features is also explained in Section 3. Experimental results are presented in Section 4 and conclu- sions are given in Section 5. In this section, we give the details of estimation of a neutral face shape dictionary using Gaussian Mixture Models (GMM). First, we describe the geometrical facial features that we use, then give the details of GMM fitting method. The geometrical features we use are based on the x and y coordinates of 68 landmark points on the face. Examples of these 68 points can be seen in Fig. 2 and Fig. 3(a). In the CK + database, these points have been manually marked at keyframes, and were tracked automatically for the remaining frames using an Active Appearance Model ...

Context 3

... in this paper. There is a plethora of recent works on automatic facial expression and emotion recognition, some of which have been high- lighted in several survey papers [2–6]. Automatic facial expression recognition is still a challenging computer vision problem. The main issues to tackle while developing a facial expression recognition algorithm are: i) choosing an emotion model ii) extracting suitable facial features iii) choosing a classification method and iv) choosing a suitable database to report facial expression recognition results. Below, we give an overview of each of them briefly. There are two emotion models used while recognizing emotions from facial expressions: the categorical model and the dimensional model . In the categorical model, the facial expression is recognized as one of the distinct labels from a set of emotions. A popular categorization of emotions consists of six basic emotions: happiness, sadness, anger, fear, surprise and disgust, which have been suggested by Ekman and Friesen and have been shown to be uni- versal [7]. Recently, dimensional and continuously labeled emotion recognition has also become popular that use a space with three dimensions [4]. The first dimension ( evaluation or valence ) measures pleasure–displeasure (positive-negativeness) attribute of the emotion. The second dimension activation or arousal measures the likelihood of a person to take an action in the emotional state and it ranges from sleepiness to extreme excitement [4]. The third dimension dominance or power represents the sense of control over the emotion. Using the dimensional model, it is possible to an- notate a facial expression on a continuous scale rather than a categorical scale to describe the wide range of emotions that we face in our daily lives. On the other hand, the dimensional model is not as intuitive as the categorical model [8]. In this paper, we address the problem of categorical emotion recognition. The facial features that can be extracted from a face image generally falls into one of two categories: geometrical features or appearance based features . Geometrical features deal with the shape and motion of facial landmarks located around salient regions such as eyelids, eyebrows and lips. Appearance-based features on the other hand, can quantify texture variations due to facial expressions such as wrinkles between the eyebrows and the forehead. It has been shown that using geometric and appearance features together improve the facial expression recognition performance [6]. A varying number of landmarks on the face can be detected and tracked to serve as geometrical features . Various methods have been used for this task including active appearance models [9,10], constrained local models [11–13], GentleBoost [14] and mixtures of trees [15]. Some appearance based features that have been shown to be effective for facial expression recognition (FER) are Gabor filters [14], local binary patterns (LBP) [16–19], local phase quantization features [17,20], and the scale-invariant feature transform (SIFT) [21,22]. Many different classifiers have been utilized for FER. Sup- port vector machines (SVMs) [14,17,18,23,24] are among the most popular ones. Other classifiers used for FER include neural networks [25], sparse representation classifiers [26,27], hidden Markov models [28], conditional random fields [29,30] and Bayesian networks [31]. Some efforts have also been reported that propose novel classifiers. In [32], Chew et al. propose a novel su- pervised classifier namely the modified correlation filter (MCF), which is shown to have superior generalization performance due to its robustness to noisy training examples. In order to test and compare automatic facial expression recognition algorithms, emotional databases which are open to researchers are needed [6]. There are a number of 2D [6,10,33–35] and 3D [36–39] facial expression databases in the literature. In this work, we utilize three widely-used facial expression databases: CK + [10], MMI [34] and eNTERFACE [40]. The Cohn–Kanade (CK) database [41] has been a very popular one, which consists of facial clips containing the six basic emotions as shown in Fig. 1. The CK database has recently been extended to include more subjects, a new facial expression class (contempt) and facial tracking data and is called the CK + database [10]. The face tracking data provided with the CK + database consists of the locations of 68 facial landmarks, which can be seen in Fig. 2. In the CK + database, there are a total of 123 subjects and 327 sequences with emotion labels. The emotions were posed in a laboratory by the participants who had different ages ranging from 18 to 50 and 69% of them were female, 81% Euro-American, 13% Afro-American and 6% from other ethnic groups. The first frame of each sequence in CK + database shows a neutral expression (onset frame) and the last frame shows an expression at its apex (peak frame) as can be observed in Fig. 2. The seven emotion categories in the database are happiness, sadness, anger, fear, disgust, surprise , and contempt . Another database that we used in this study is the MMI database [33,34]. This database contains recordings of full temporal patterns of facial expressions covering neutral, onset, apex and off- set phases. The facial expressions consist of the six basic emotions as well as expressions with a single action unit, which were posed in a laboratory environment using either natural lighting or two high intensity lamps with reflective umbrellas. Some naturalistic expressions have also been added to the database [34]. There are currently 75 subjects in the whole database aged between 19–62, who have European, Asian or South American ethnic backgrounds. The third database that we used in this work is eNTER- FACE’05 [40], which is an audio–visual database. We used this database to test the generalization potential of the proposed method to less constrained data containing lip motion due to speech. The eNTERFACE database contains a total of 1287 audio– visual clips of 44 subjects (81% men 19% women) from 14 different nationalities. The database contains recordings of sentences ut- tered in English in an acted way, reflecting the six basic emotions, i.e., happiness, sadness, anger, fear, disgust and surprise. Each clip is expected to reflect one of these emotions from the beginning to the end. Therefore, neutral expression does not exist in the database. The visual component of eNTERFACE dataset is quite challenging since the actors are not professional and some emotions are very hard to detect visually, even for humans (e.g. fear). Moreover, 31% of the subjects wear glasses and 17% have beard. An expressive face image carries information about both the facial expression, and identity (including age and gender) of that person. If shape-based features are used to describe the locations of salient points (around eyes, lips etc.), suppressing the identity- related information has been shown to increase the facial expression recognition rate. One way to do this is to subtract the baseline [4,42], that is, the neutral face shape of the person from the expressive shape, and to work with the “motion” information. This solution is feasible only if the neutral face image (or shape) is available, for example as in the CK + database, which is always the first frame of a sequence [10]. If the neutral face image is not available, one approach [43] uses averaging all the frames of a sequence assuming that the average will give us an estimate of the neutral face shape. However, this assumption does not hold if all the frames of a video are strongly expressive. In this work, we propose a general solution to the baseline problem described in the previous paragraph. That is, we present a general method for estimating the neutral (i.e. expressionless) face shape for a given expressive face. First, a dictionary of neutral face shapes is generated. Then, the best fitting dictionary entry is selected as the estimated neutral shape for a given expressive face, which is expected to model the identity-related component of the expressive face. We also propose an efficient method of fusing geometric and appearance based features. Stand-alone and cross- database experiments on the CK + , MMI and eNTERFACE databases demonstrate that the proposed method increases the accuracy of facial expression recognition significantly, when the actual neutral face image is unavailable. In Section 2, the method proposed for estimation of a neutral face shape dictionary using Gaussian Mixture Models is presented. In Section 3, the facial expression recognition method, which uses the neutral shape dictionary is explained in detail. The fusion of geometric and appearance based features is also explained in Section 3. Experimental results are presented in Section 4 and conclu- sions are given in Section 5. In this section, we give the details of estimation of a neutral face shape dictionary using Gaussian Mixture Models (GMM). First, we describe the geometrical facial features that we use, then give the details of GMM fitting method. The geometrical features we use are based on the x and y coordinates of 68 landmark points on the face. Examples of these 68 points can be seen in Fig. 2 and Fig. 3(a). In the CK + database, these points have been manually marked at keyframes, and were tracked automatically for the remaining frames using an Active Appearance Model based approach [9,10,44]. In the MMI database, we track these landmark points fully automatically, using an improved version of the Constrained Local Models based approach [11] (see Fig. 3(b)). The Face Tracker implemented by Saragih et al. [11] is a near real-time, generic face and facial feature tracker based on deformable model fitting by regularized landmark mean-shift. Before the geometrical feature vectors are formed for each frame of a video clip, face shapes as described by ...

Context 4

... and facial tracking data and is called the CK + database [10]. The face tracking data provided with the CK + database consists of the locations of 68 facial landmarks, which can be seen in Fig. 2. In the CK + database, there are a total of 123 subjects and 327 sequences with emotion labels. The emotions were posed in a laboratory by the participants who had different ages ranging from 18 to 50 and 69% of them were female, 81% Euro-American, 13% Afro-American and 6% from other ethnic groups. The first frame of each sequence in CK + database shows a neutral expression (onset frame) and the last frame shows an expression at its apex (peak frame) as can be observed in Fig. 2. The seven emotion categories in the database are happiness, sadness, anger, fear, disgust, surprise , and contempt . Another database that we used in this study is the MMI database [33,34]. This database contains recordings of full temporal patterns of facial expressions covering neutral, onset, apex and off- set phases. The facial expressions consist of the six basic emotions as well as expressions with a single action unit, which were posed in a laboratory environment using either natural lighting or two high intensity lamps with reflective umbrellas. Some naturalistic expressions have also been added to the database [34]. There are currently 75 subjects in the whole database aged between 19–62, who have European, Asian or South American ethnic backgrounds. The third database that we used in this work is eNTER- FACE’05 [40], which is an audio–visual database. We used this database to test the generalization potential of the proposed method to less constrained data containing lip motion due to speech. The eNTERFACE database contains a total of 1287 audio– visual clips of 44 subjects (81% men 19% women) from 14 different nationalities. The database contains recordings of sentences ut- tered in English in an acted way, reflecting the six basic emotions, i.e., happiness, sadness, anger, fear, disgust and surprise. Each clip is expected to reflect one of these emotions from the beginning to the end. Therefore, neutral expression does not exist in the database. The visual component of eNTERFACE dataset is quite challenging since the actors are not professional and some emotions are very hard to detect visually, even for humans (e.g. fear). Moreover, 31% of the subjects wear glasses and 17% have beard. An expressive face image carries information about both the facial expression, and identity (including age and gender) of that person. If shape-based features are used to describe the locations of salient points (around eyes, lips etc.), suppressing the identity- related information has been shown to increase the facial expression recognition rate. One way to do this is to subtract the baseline [4,42], that is, the neutral face shape of the person from the expressive shape, and to work with the “motion” information. This solution is feasible only if the neutral face image (or shape) is available, for example as in the CK + database, which is always the first frame of a sequence [10]. If the neutral face image is not available, one approach [43] uses averaging all the frames of a sequence assuming that the average will give us an estimate of the neutral face shape. However, this assumption does not hold if all the frames of a video are strongly expressive. In this work, we propose a general solution to the baseline problem described in the previous paragraph. That is, we present a general method for estimating the neutral (i.e. expressionless) face shape for a given expressive face. First, a dictionary of neutral face shapes is generated. Then, the best fitting dictionary entry is selected as the estimated neutral shape for a given expressive face, which is expected to model the identity-related component of the expressive face. We also propose an efficient method of fusing geometric and appearance based features. Stand-alone and cross- database experiments on the CK + , MMI and eNTERFACE databases demonstrate that the proposed method increases the accuracy of facial expression recognition significantly, when the actual neutral face image is unavailable. In Section 2, the method proposed for estimation of a neutral face shape dictionary using Gaussian Mixture Models is presented. In Section 3, the facial expression recognition method, which uses the neutral shape dictionary is explained in detail. The fusion of geometric and appearance based features is also explained in Section 3. Experimental results are presented in Section 4 and conclu- sions are given in Section 5. In this section, we give the details of estimation of a neutral face shape dictionary using Gaussian Mixture Models (GMM). First, we describe the geometrical facial features that we use, then give the details of GMM fitting method. The geometrical features we use are based on the x and y coordinates of 68 landmark points on the face. Examples of these 68 points can be seen in Fig. 2 and Fig. 3(a). In the CK + database, these points have been manually marked at keyframes, and were tracked automatically for the remaining frames using an Active Appearance Model based approach [9,10,44]. In the MMI database, we track these landmark points fully automatically, using an improved version of the Constrained Local Models based approach [11] (see Fig. 3(b)). The Face Tracker implemented by Saragih et al. [11] is a near real-time, generic face and facial feature tracker based on deformable model fitting by regularized landmark mean-shift. Before the geometrical feature vectors are formed for each frame of a video clip, face shapes as described by the tracked landmark points are aligned to minimize the scale, rotation, and translation differences that may exist between different subjects and also between different frames of a video clip. For this purpose, the landmarks which are supposed to be affected the least from the facial expressions and are robust to track are chosen, such as the inner corners of the eyes and the nose tip [45]. Translation differences between frames are eliminated by moving the nose tip to the origin. The in-plane head rotation variations are eliminated by making the line connecting the inner corners of the eye parallel to the x -axis. The scale differences are minimized by normalizing the inter-ocular distance to a constant value. An example of face alignment is shown in Fig. 4, which shows the fearful and neutral facial expressions of subject 32 before alignment (a) and after alignment (b). We can see that after the alignment, the nose and the inner corners of the eyes, and the sides of the cheek are aligned well. As can be observed in Fig. 4(c) and (d), the alignment step minimizes the global scale, rotation and translation differences between all frames in the database sim- ilar to affine registration methods in the literature [46,47], while keeping the variations due to facial expressions. The coordinate based features (CBF) are formed by concatenating the x and y coordinates of M aligned landmark points in the peak frame of an image sequence. If the landmark points of the person- specific neutral facial expression are available (which is the first frame in CK + database), they are subtracted from the peak frame and will be named as coordinate based features with neutral subtraction (CBF-NS). Several examples of the CBF-NS features are shown in Fig. 5, which are basically the motion vectors of the facial landmark points. Our goal is to estimate the unknown neutral face shape for any given expressive face aiming to improve the facial expression recognition performance with the help of it. Therefore, we model the density of space of neutral face shapes using Gaussian Mixture Models (GMMs), expecting that the means of each Gaussian component will represent typical face shapes that might be expected in a population, reflecting personal differences (e.g. round face, thin face etc.). The set of mean vectors of the Gaussian components will give us a dictionary of neutral face shapes. In order to fit a GMM to the space of neutral face shapes, we use the neutral images in the CK + database from 123 subjects, which are the first frames of the sequences. Let χ = { s n , 1 } , n = 1 , . . . , N represent the neutral shape data set, where s n , 1 = [ p n 1 , 1 , p n 2 , 1 , . . . , p n M , 1 ] represents the face shape in the first frame of image sequence n , which consists of M normalized coordinates of landmark points after alignment, i.e., p n i , 1 = ( x n i , 1 , y n i , 1 ) . For example, there are M = 68 facial points in the CK + database. The notation will be simplified as χ = { s 1 , . . . , s N } to represent the set of neutral face shapes (i.e. the sample). One way of modeling the distribution of neutral face shapes is to use mixture of densities ...