Expression and communication of doubt/uncertainty through facial expression

Abstract

There is a wide debate on the mental state of doubt/uncertainty; one wonders whether it is a predominantly cognitive or emotional state of mind and whether typical facial expressions communicate doubt/uncertainty. To this purpose,through a role playing procedure, a large sample of expressions were collected and afterwards evaluated through a combination of encoding and decoding procedures,including also FACS (Facial Action Coding System) analysis. The results have partially confirmed our hypothesis, identifying two typical facial expressions of doubt/uncertainty, which share the same facial actions in the inferior part of the face and show differential facial actions in the upper face.

Full text

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
159
Expression and communication of doubt/uncertainty through
facial expression
Pio E. Ricci Bitti
University of Bologna
Luisa Bonfiglioli
University of Bologna
Paolo Melani
University of Bologna
Roberto Caterina
University of Bologna
Pierluigi Garotti
University of Bologna
Abstract
There is a wide debate on the mental state of doubt/uncertainty; one wonders
whether it is a predominantly cognitive or emotional state of mind and whether
typical facial expressions communicate doubt/uncertainty. To this purpose,
through a role playing procedure, a large sample of expressions were collected and
afterwards evaluated through a combination of encoding and decoding proce-
dures, including also FACS (Facial Action Coding System) analysis. The results
have partially confirmed our hypothesis, identifying two typical facial expressions
of doubt/uncertainty, which share the same facial actions in the inferior part of
the face and show differential facial actions in the upper face.
Keywords: facial expressions, doubt, uncertainty, communication, cognition
_____________________________________________________________

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
160
Introduction
Face and facial expression
The face plays an important role in social interaction, both in its static dimen-
sions (structural feature, physiognomy) and in its dynamic dimension (facial ex-
pression), being a rich source of information and interactive signals. The face is in
fact able to send a lot of information concerning age, gender, social status, etc.,
and affects impression of personality through the process of interpersonal percep-
tion. Facial expression on the other hand is an effective signaling system in inter-
personal communication. In combination with other nonverbal signals it has a
strong and immediate impact in expressing emotions such as fear, anger, happi-
ness, sadness… and in communicating interpersonal attitudes such as cordiality,
hostility, dominance, submission and so on; it communicates also other mental
activity such as attention, memory, thinking, etc. Moreover the face takes part ac-
tively in conversation: the “speaker” accompanies his/her words with facial ex-
pression to emphasize or modulate the meaning of verbal communication; the
“listener” during conversation provides a constant feedback through facial expres-
sion. Facial movements take also part in regulating the interpersonal exchanges
and synchronizing the turn taking. The face may finally produce mimic move-
ments that can play the role of adaptive behavior (correlated with the level of
arousal experienced by the individual) or symbolic signals.
Facial expression of cognitive processes
In relation to facial expression of cognitive processes, De Sanctis (1902) an
Italian pioneer in facial expression study, dedicated an entire book to La Mimica del
Pensiero (The Facial Expression of Thought) posing many questions, such as: do
thought processes manifest themselves in facial expressions? Is there a relationship
between mental states such as attention and concentration and the concomitant
facial expression? What relationship exists between the expression of emotions (or
affective states) and the expression of other mental processes, such as attention,
concentration or cognitive engagement? Can the extent and intensity of the facial
movements act as a measure of the degree of concentration? Of course, De Sanc-

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
161
tis’ considerations were mainly the result of observation and speculation; they did
however open a very interesting chapter, which did not subsequently receive the
due attention from researchers.
De Sanctis identified frowning and the movements of the eyebrow region as
the most significant indicators of cognitive processes; in particular he maintains, in
agreement with Duchenne (1876), that the frontalis muscle can be considered as
the muscle that chiefly expresses attention towards external objects (also called
sensory attention or external attention). To this he added the movement used for
internal attention (or reflection), during which a marked reduction of the eyelid
opening is observed. A particular form of integration between the two previous
expressions is represented by the so-called interrogative attention (Cuyer, 1902), in
which both frowning and the relative tightening of the eyelids are observed. An-
other interesting form of facial expression of a specific cognitive process is ob-
served during mnemonic effort: here too, the eyes are narrowed due to the eyelids
tightening and the direction of the gaze can be diverted, upwards for example. Al-
so Darwin (1872) had a certain interest in the so-called “blank” expression of the
eyes, which expresses a kind of “assortment” of the thoughts, an “enchantment”
in which the gaze is blank and the eyelids slightly narrowed.
In short, we can say that the expressive structures of the face that manifest the
various cognitive processes (external attention, reflection, concentration, mne-
monic effort etc.) are mainly located in the upper part of the face and can originate
in the forehead, eyelid and eyebrow muscles. A less significant role is played by the
muscles in the lower part of the face, that are responsible of the movements in-
volving the mouth: in some cases the mouth can be shaped into the movement
resembling a kiss (lip pucker); in other cases the lips are pressed together (lip tight-
ener or lip presser) or pulled inside (lip suck); finally, we can observe a stretching
of the corners of the mouth to produce a sardonic expression (movements similar
to those produced in the facial expression that accompanies physical efforts).
De Sanctis concludes his observations by highlighting, at least on a methodo-
logical level, the distinction between emotional and “intellectual” expression, rec-
ognizing greater complexity and propagation in the former than in the latter,
which involves fewer facial movements and which are also less intense and evident
than emotional expressions. More recently few contributions concern the facial
expression accompanying cognitive process (Ekman, 1979; Fridlund, 1991; Scher-
er, 1992; Baron-Cohen et al., 1997; Pelachaud & Poggi 2002; Ricci Bitti, in press);
some of them agree with De Sanctis arguing that the facial expressions of cogni-
tive processes are mainly located in the upper part of the face.

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
162
Facial expression and interpersonal communication
It is clear to all that the face takes an active part in the communication process-
es that occur during conversation, through the gaze direction and the movements
of the forehead, eyebrows, eyes, and the lower part of the face (together with oth-
er signals not discussed here, such as the position and movement of the head and
shoulders, posture, gestures, etc.). Through these movements, the face constantly
accompanies both the speaker and the listener as they take turns speaking in a
conversation.
The speaker constantly accompanies his/her words with facial expressions that
in turn emphasize, underline and modulate the content and meaning of the con-
comitant verbal language, in the same way as he/she uses an array of hand ges-
tures (Ekman, 1976; Ekman & Friesen 1972; Rimé, 1983). To this end, the move-
ments involving the muscles of the forehead, eyebrows and mouth play a signifi-
cant part. The eyebrows in particular provide a great deal of information about
verbal behavior (Costa & Ricci Bitti, 2003) by lifting, lowering or moving together
to varying degree; expressed respectively by Action Unit 1 (AU 1) Inner Brow
Raiser; AU 2 Outer Brow Raiser and AU 4 Brow Lowerer as per the FACS (Ek-
man & Friesen, 1978; Ekman et al., 2002).
From a functional point of view these facial movements, which we can call co-
verbal facial expressions, have specific characteristics distinct from emotional faci-
al expressions. They are quicker and appear at the same time as the concomitant
verbal behavior.
Despite their apparent resemblance and use of the same underlying muscles,
co-verbal and affective (conveying emotions) facial behaviors differ from one an-
other in many ways, such as in their function, form, duration and, probably, in
their activation mechanisms. Co-verbal facial expressions are governed by the ver-
bal system; their precise coordination with the word or phrase is crucial in trans-
mitting the message, which is thus completed and/or modulated. They are charac-
terized by rapid and “individual” movements, unlike affective facial expressions
which consist most often of combinations of facial actions, which are activated,
evolve and end in ways that do not correspond to the clear confines of linguistic
units, as is the case with co-verbal expressions with grammatical functions.
The difference between co-verbal facial expressions with linguistic functions
and affective expressions is also supported by neuropsychological research. It was
demonstrated that the two types of expression involve the activation of different
neural structures: affective expression are processed mainly in the right hemi-
sphere, while co-verbal expression with linguistic function are chiefly processed in
the left hemisphere. Further proof of these differences is provided by studying

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
163
aphasic patients: damage to specific areas of the left hemisphere can cause deterio-
ration of co-verbal facial expressions with linguistic function without interfering
with affective facial expressions, while damage to the right hemisphere with con-
sequent deterioration of affective expressions leaves co-verbal facial expressions
with linguistic functions intact (Adolphs et al., 1996; Borod et al., 1998; Burt &
Perret, 1997; Campbell, 1986).
Doubt/uncertainty and its communication
Communication of doubt/uncertainty is necessary to mutual understanding in
dialogue and develops dynamically and continuously; the analysis of processes and
mechanisms underlying communication of interlocutors’ uncertainty allows to ex-
plain how people detect and resolve misunderstandings (Stone & Oh, 2008). In
the developmental psychology, several studies highlight that speaker certainty is an
important cue for learning and that nonverbal signals which reflect meta-cognitive
aspects are the latest to develop. Empirical studies show that children 3–4 years
old are competent at identifying accurate and inaccurate informants (e.g., Bisanz et
al., 1975; Koenig et al., 2004).
Doubt/uncertainty: issues from psychological literature.
Doubt/uncertainty has recently been examined in various disciplines such as neu-
roscience (Zaretsky, 2010), informatics (Stone & Oh, 2008), linguistics (Carberry
et al., 2002), philosophy (Goldie, 2009). Psychological literature discusses several
topics concerning doubt. In particular, different studies try to deepen the follow-
ing issues:1) is doubt a cognitive rather than an affective or emotional condition?;
2) is doubt associated with affective states (positive or negative)?; 3) how do indi-
viduals express doubt ?; 4) how do individuals learn to express and interpret epis-
temic modality (degrees of certainty) both in mother language and in second lan-
guage? Moreover, it is suggested to distinguish different types of
doubt/uncertainty; we can here consider among others at least three of them: a)
the doubt/uncertainty related to the specific internal condition concerning deci-
sion making; b) the “doubt condition” that concerns the attitude assumed by the
individual in the pursuit of truth or what is right (about philosophical, religious,
moral, political issues); c) finally, the doubt/uncertainty associated with a specific
case of interpersonal communication, e.g. in case of “Wh questions” (e.g. Where,
What, Why, Who, etc.) when the speaker, trying to reply to a question, is not cer-
tain to know the answer.

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
164
As far as the first kind of doubt/uncertainty related to the specific internal
condition concerning decision making, we note that, in this case, the state of
doubt/uncertainty can be considered as an attitude with cognitive, affective and
behavioral components. Specifically, cognitive component involves thoughts and
cognitive activity related to the decision; affective components involves positive or
negative emotions related to the doubt/uncertainty and, finally, behavior-
al/motivational component involves the action tendencies. We can assume that
the facial expression of this type of doubt depends both on different emotions and
cognitive processes related to the management of doubt. This specific state of
doubt/uncertainty, which tends to last a long time until the subject has not taken a
decision, doesn't have specific and unambiguous facial expressions. The non-
verbal signals depend on both the affective state (e.g., worry, anxiety, etc.) experi-
enced by the individual and the cognitive processes activated in order to manage
or resolve the condition of doubt/ uncertainty (e.g., concentration, mental effort,
etc.). The communication literature presents several studies concerning the analy-
sis of the relationship among affective and cognitive components in confusion and
concentration (Rozin & Cohen, 2003; Ellsworth, 2008); to the contrary, how spe-
cific expressions convey uncertainty by signalling specific elements of an interlocu-
tor’s cognitive or affective state is a highly debated issue in studies regarding
doubt.
Another kind of doubt/uncertainty condition concerns the attitude assumed by
the individual in the pursuit of truth or what is right (about philosophical, reli-
gious, moral, political issues); in this case the question is cultivated as a method for
approaching to the knowledge of the true and tends to persist over time. In this
case there are no typical/specific nonverbal signals.
The third kind of doubt/uncertainty we would like to take in consideration is
the doubt/uncertainty associated with a specific case of interpersonal communica-
tion, that occurs when the speaker, trying to reply, for example, to a “Wh ques-
tion”, is not certain to know the answer; in this case the facial expression accom-
panying the verbal message “I am not sure to know it” (or “I doubt to know it or I
am uncertain to know it”) serves as a conversational signal, that externalizes the
cognitive processes related to search/retrieval of a specific information or
knowledge.
Communication of doubt/uncertainty and facial expression.
Maatman
(2005) highlight the importance of backchannel signals of listeners that are strictly
correlated to speaker behaviour and, into this kind of cues, frown and gaze direc-
tion are often linked to speaker communication of doubt/uncertainty.

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
165
Eibl-Eibesfeldt (1979) highlighted that eyebrows movements have an im-
portant role in interpersonal interaction and examined different meanings of eye-
brows raising. In particular, he distinguishes between emotional signals and con-
versational signals and includes cues associated with doubt, question, and empha-
sis among conversational signals. Moreover, Eibl-Eibesfeldt considers the original
meaning of eyebrow raising as a consequence of eyes opening related to attention
and suggests that individuals use this nonverbal signal to display surprise both in
the case of negative and positive stimuli.
Poggi and Pelachaud (2000) propose to study eyebrow opening as a polyse-
mous nonverbal signal with a restricted number of different meanings related to
attention or violation of expectation. They also outline that, in the specific case of
doubt/uncertainty, opening eyebrows is a direct consequence of violation of ex-
pectations.
Data collected in a study through judges’ rating of a speaker’s apparent certain-
ty task, show that there are reliable behavioral cues for degrees of
doubt/uncertainty and that these clues seem to be present in the nonverbal behav-
iors of people detected from video or audio or both (audio-video) but not from
information contained in the text only (Oh et al., 2007); Furthermore, these data
suggest that both facial movements and head and eye movements linked to cogni-
tive state seem to contribute to viewers’ judgments of the degree of certain-
ty/uncertainty of the speaker.
In a developmental perspective, Krahmer and Swerts (2005) have compared
the ability in signalling and detecting uncertainty in audio-visual speech by adults
(20 –50 years old) and second grade school children (7–8 years old). The study
suggests that there are specific cues (e.g., filler usage) that are gradually learned,
and not yet fully developed in 7-8 years old children.
Carberry, Lambert, and Schroeder (2002) have analysed the attitude (a disposi-
tion, or mental state, toward a particular proposition or situation) of
doubt/uncertainty concerning particular proposition expressed by another agent
in the framework of studies about artificial agents. They propose that artificial
agents are able to “play” a role of cooperative partners in problem-solving and de-
cision-making if they can interact with users in an ecological and natural way; spe-
cifically they have to recognize and exhibit the same attitudes (like attitude of
doubt) as human agents. Carberry, Lambert, and Schroeder highlight that a collab-
orative artificial agent must also both recognize a user’s attitude toward a proposi-
tion expressed in a dialogue and generate an utterance that convey doubt.
Givens (2001) reports several nonverbal cues associated with uncertain-
ty/doubt that include facial expression (frowns, eye movements, lip-pouting, lip-

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
166
pursing), head movements (side-to-side head-shakes, head tilts) and gestures like
self-touch gestures, palm-up gestures, and the shoulder-shrug.
Swerts and Krahmer (2005) have analysed facial expression of speaker’s
doubt/uncertainty founding specific visual cues like eyebrow movements, smiling,
diverted gaze, and marked facial expressions.
Stone and Oh (2008) have examined displays of uncertainty in face-to-face
conversation to simulate those behaviors in the Rutgers University talking Head
(RUTH), an ECA (Embodied Conversational Agent) that represents a real-time
facial animation system able to animate conversational facial displays and head
movements in synchrony with speech and lip movements. Results show that
movements of the head and eyes seem to signal what an interlocutor is doing to
contribute to the conversation (e.g., listening, planning an utterance, presenting
information, questioning or revising previous contributions) while other displays
seem to serve to appraise how well that on-going activity is proceeding. They also
suggest that head and eye movements are indicators of uncertainty and highlight
the presence of a specific nonverbal signals used by participants to communicate
uncertainty: the “facial shrug” formed by “pressing the lips together and raising
the chin, arching the upper lip and allowing the lower lip to bulge outward” (Stone
& Oh, 2008, pag. 67).
Aim
This study concerns the facial expression that accompanies the communication
of doubt/uncertainty; in particular we have analysed the facial expressions in-
volved in case a person communicates doubt/uncertainty related to a specific in-
formation or knowledge of her/his own in answering a question of this kind: “Do
you know what…?”, “Do you know who...?”, “Do you know whether…?”, “Do
you know when…?”, “Do you know where…?”. The aim of this study was to ana-
lyse facial expression and other non-verbal signals involved in two different cases
of doubt/uncertainty communication:
1. “I don't know” as: “I'm sure that I don't know!”; (in this case the answer
does not communicate a doubt but rather a certainty; the certainty of not
knowing the answer).
2. “I don't know” as: “ I'm not sure to know it, ...may be I could know… I can
try to retrieve this information…” .

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
167
Our hypothesis is that the two types of facial expression share the same AU com-
bination (AUc) in the inferior part of the face, but show differential facial actions
in the upper face, that has the specific function to express the involved cognitive
processes.
Method and procedure
Since we considered two possible nonverbal answers that correspond to the
verbal answer “I do not know…..” in front of different “Wh questions”, we real-
ized a four steps study, where we adopted a combination of encoding and decod-
ing procedure.
First step
In this step, with the help of 12 volunteer encoders (6 Females and 6 Males;
mean age 41.2; range 34-68) we carried out an individual session of role playing,
lasted approximately 10 minutes where we got facial expressions accompanying
answers like, “I don't know”… “I am not sure to know it”… “I doubt to know
it”… “I am uncertain to know it”… “I don't know but I can try to retrieve this
information…”
in response to 5 “Wh questions”. Our interest was focused only
on facial expression; other nonverbal signals take part in fact in this kind of com-
munications differentiating the two types of answer: e.g., in the first type of an-
swer the facial expression can be accompanied by the head shake, as negation sig-
nal. Out of the 60 photos 36 respond to the including criterion. Twenty-four an-
swers were unable to produce minimum required technical characteristics (E.g. the
participants move partially out of the shot angles of the camera…).
Second step
We presented the 36 facial expressions to 30 decoders (19 females and 11
males; mean age 36.3; range 24-41), that evaluated them through two different
tasks: in the first case they had to evaluate each expression through a free verbal
answer (during the pre-set time of 10 second); the analysis of the different free
verbal definitions of the facial expressions found out three possible “families” of
feelings/attitudes expression:

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
168
1. “I don’t know”: that means certainty of not knowing.
2. “I’m not sure”: that really means uncertainty.
3. “I am thinking about it”: that means “I am uncertain, but I can try to re-
trieve this information in my mind”.
In the second task they had to assess the degree of correspondence of each facial
expression, on a five points likert scale (from 1-Absolutely disagree to 5-
Absolutely agree), to the three “families” of answers listed above.
On the basis of the results, we chose 4 images (2 males and 2 female) of facial
expressions for each of the three meanings above mentioned; we selected the im-
ages with the highest correspondence to the three “families” of meaning (see ta-
ble 1).
Table 1. Degree of correspondence to the three “families” of meaning.
I don’t know
I’m not sure
I am thinking
about it
Mean
(SD)
Mean
(SD)
Mean
(SD)
Female Images
4,11
1,07
4,47
0,87
4,53
0,50
Female Images
4,03
0,99
4,42
0,76
4,42
0,79
Male Images
4,03
0,55
4,31
1,10
4,64
0,48
Male Images
4,00
1,43
4,
25
0,95
4,19
0,99
Third step
The 12 images of facial expressions were included in random order in a set of
36 images; the other 24 images represented male and female encoders expressing
by facial movements the following six emotions: anger, disgust, fear, happiness,
sadness, surprise. Then we realized two decoding sessions where we administered
these 36 images of facial expressions to two groups of decoders recruited volun-
tarily among psychology university students’:
• the first group of 66 decoders (53 females and 13 males, mean age 22.3;
range 19-44) had the task to evaluate and describe each facial expression
through a free verbal label (adjectives, nouns, sentences ...);
• the second group of 66 decoders (54 females and 12 males, mean age 21.9;
range 19-41) had the task to attribute to each facial expression a specific mean-

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
169
ing through one of the following verbal labels: anger, disgust,
doubt/uncertainty, fear, happiness, sadness, surprise.
Fourth step
In the last step of the study we analyzed the three types of facial expression ob-
tained in the second step through the Facial Action Coding System (FACS; Ek-
man & Friesen 1978; Ekman et al., 2002); two independent judges decoded the
different Action Units (AUs) involved in the 12 facial expressions.
Results and discussion
The verbal labels (adjectives, nouns, sentences, etc.) attributed to the 12 facial
expressions in the free label task of the second step of the study were grouped by
“families”; e.g., in relation to “doubt/uncertainty”, we grouped terms such as:
doubt, doubtful, uncertain, hesitant, uncertainty, indecisive, without conviction, I
do not know, no idea, indecision, irresolute…
The degree of recognition concerning the 12 facial expressions of
doubt/uncertainty through the task with free verbal definition was 63.51% (see
figure 1), which is particularly high, significantly higher than chance accuracy and
not substantially lower than the degree of recognition obtained in the same task by
the facial expressions of basic emotions (mean percentage: 71.02%).
4,67%
1,39%
11,24%
3,22%2,78%
63,51%
4,04%
0,25%1,64%
7,26%
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
no answer
happiness
sedness
surprise
anger
doubt/uncertainty
disgust
fear
contempt
other
Figure 1. Free label task: recognition degree (%)

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
170
The degree of recognition obtained by the second group of decoders was
79.61% (see figure 2), which is significantly higher than chance accuracy and not
substantially lower than the degree of recognition obtained in the same task by the
facial expressions of basic emotions (mean percentage: 85.00%).
1,45% 0,13%
8,40% 1,70% 1,58%
79,61%
6,50% 0,63%
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
no answer
happiness
sadness
surprise
anger
doubt/uncertainty
disgust
fear
Figure 2. Fixed label task: recognition degree (%)
Chi square analysis through the three different types of facial expression of
doubt/uncertainty (1. “I don’t know”; 2. “I’m not sure”; 3. “I am thinking about
it”) showed significant differences between the first type and the other two; (2)
= 19.229, p= .000 (see figure 3). This result confirms that these two facial expres-
sions are considered more specific representation of doubt/uncertainty.
Figure 3. Comparison of the three expressions of doubt/uncertainty
66,10%
89,20%
83,52%
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
90,00%
I don't Know I'm n ot sure I'm th inking
about it

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
171
The FACS analysis of the three types of facial expressions of
doubt/uncertainty was performed by two independent FACS coders. Scoring
agreement was quantified with Cohen's Kappa for the presence of single AU in
the three types of doubt/uncertainty (see table 2).
Table 2. Kappa's coefficients means value for single Action Units in the 12 selected images
Single
Action Units
I don’t know
Occurrence
I’m not sure
Occurrence
I’m thinking about it O
c-
currence
AU1
Np*
0.88
0.73
AU2
Np*
0.89
0.75
AU4
Np*
Np*
0.88
AU7
Np*
Np*
0.79
AU15
0.90
0.91
0.89
AU17
0.87
0.92
0.93
*Not present
Conclusions
These results partially confirmed our hypothesis, identifying two typical facial
expressions of doubt/uncertainty, which share the same facial actions in the infe-
rior part of the face and show different facial actions in the upper face. In the low-
er part of the face all the three types of expression involved the presence of the
AUc 15+17 (Lip Corner Depressor + Chin Raiser); in the upper part of the face
we found the following differences: while the first type of expression (correspond-
ing to the verbal message “I do not know, and I am certain I do not know”) does
not present any specific AU (see figure 4), the second type of expression (corre-
sponding to the verbal message “I am not certain to know it”) shows the AUc
1+2 (Inner Brow Raiser + Outer Brow Raiser; see figure 5); finally the third type
of expression (corresponding to the verbal message “I do not know, but I can try
to retrieve this information…”) shows in addition to the AUc 1+2 also the AUc
4+7 (Brow Lowerer + Lid Tightener) and a deviation of gaze direction that are
both signal of mental concentration (see figure 6).
Our results show that the communication of doubt/uncertainty activates most-
ly the facial expressions involved in cognitive processes and that the study of facial

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
172
expression must devote more attention to expressions involved in conversation
and externalization of cognitive processes.
Figure 4. Facial expression for “I do not know”.
Figure 5. Facial expression for “I am not sure to know it”.
AUc 1+2
AUc 15+17
AUc 15+17

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
173
Figure 6. Facial expression for “I do not know, but I can try to retrieve this information”.
References
Adolphs, R., Damasio, H., Tranel, D., Damasio A. R. (1996). Cortical systems for
the recognition of emotion in facial expression. The Journal in Neuroscience, 16,
7678-7687.
Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997). Is there a “language of the
eyes”? Evidence from normal adults and adults with autism or Asperger syn-
drome. Visual Cognition, 4, 311–331.
Bisanz, G. L., Vesonder, G.T., & Voss, J. F. (1978). Knowledge of one’s own re-
sponding and relation of such knowledge to learning: A developmental study.
Journal of Experimental Child Psychology, 25, 116–128.
Borod, J. C., Obler, L. K., Erhan, H. M., Grunwald, I. S., Cicero, B. A., Welko-
witz, J., Santschi, C., Agosti, R. M., & Whalen, J. R. (1998). Right hemisphere
emotional perception: evidence across multiple channels. Neuropsychology, 12,
446-458.
AUc 4+7
AUc 15+17

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
174
Burt, M. D., Perrett, D. I. (1997). Perceptual asymmetries in judgments of facial
attractiveness, age, gender, speech and expression. Neuropsychologia, 35, 685-
693.
Campbell, R. (1986). The lateralization of lip-read sounds: a first look. Brain and
Cognition, 5, 1-21.
Carberry, S., Lambert, L.,& Leah Schroeder. (2002). Toward Recognizing and
Conveying an Attitude of Doubt Via Natural Language. Applied Artificial Intelli-
gence 16(7), 495-517.
Costa, M., Ricci Bitti, P. E. (2003). Il chiasso delle sopracciglia. [The din of the
eyebrows]. Psicologia Contemporanea, 176, 38-47.
Cuyer, E. (1902). La mimique. [The facial expression]. Paris, France: Doin
Darwin, C. (1872). The Expression of Emotions in Man and Animals. London, Eng-
land: Murray.
De Sanctis, S. (1902). La Mimica del Pensiero. [The facial expression of thought]. Pa-
lermo: Sandron.
Duchenne, G. B. (1876). Mecanisme de la Physiologie Humain [Mechamisms of Hu-
man Physiology] II Edition. Paris, France: Baillière
Eibl-Eibesfeldt, I. (1979). Somiglianze interculturali tra movimenti espressivi. In
Diodato, L., (Ed.), Il corpo parla. Gli altri linguaggi, (pp.110-121). Bari, Italy:
Laterza.
Ekman, P. (1979). About brows: Emotional and conversational signals. In J.
Aschoff, M. von Carnach, K. Foppa, W. Lepenies, & D. Plog (Eds.). Human
ethology (pp. 169-202). Cambridge, England: Cambridge University Press.
Ekman, P. & Friesen, W. V. (1972). Hand Movements. Journal of Communication, 22,
353-374.
Ekman, P. & Friesen, W. V. (1978). Manual For The Facial Action Coding System. Palo
Alto, L.A.: Consulting Psychologists Press.
Ekman, P. & Friesen, W. V. & Hager J. C. (2002). New version of the Facial Action
Coding System. Retrieved from
http://www.face-and-emotion.com/dataface/facs/new_version.jsp.
Ekman, P. (1976). Movements with precise meaning. Journal of Communication, 26,
14-26.
Ellsworth, P. C. (2003). Confusion, concentration and other emotions of interest.
Emotion, 3, 81–85.
Fridlund, A. J. (1991). The sociality of solitary smiles: Effects of an implicit audi-
ence. Journal of Personality and Social Psychology, 60, 229-240.
Givens, D. B. (2002). The Nonverbal Dictionary of Gestures, Signs and Body Language
Cues. Spokane, Washington, DC: Center for Nonverbal Studies Press.

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
175
Goldie, P. (2009). The Oxford Handbook of Philosophy of Emotion. Oxford, England:
Oxford University Press.
Koenig, M., Clément, F., & Harris P. L. (2004). Trust in testimony; Children’s use
of true and false statements. Psychological Science 15(10), 694–698.
Krahmer, E., Swerts, M. (2005). How Children and Adults Produce and Perceive
Uncertainty in Audiovisual Speech. Language and Speech 48(1), 29-53.
Maatman, M., Gratch, J. & Marsella, S. (2005). Natural behavior of a listening
agent. Proceedinggs of 5th International Working Conference on Intelligent
Virtual Agents. Kos, Greece. In, Themis P., Gratch, J., Aylett, R., Ballin, D.,
Olivier, P., & Rist, T. (Eds.). Lecture notes in Computer Science, (pp.25-36). Berlin,
Germany: Springer-Verlag.
Oh, I., & Stone M. (2007). Understanding RUTH: Creating believable behaviors
for a virtual human under uncertainty. In Michael L., Sebe, N., Huang, T. S., &
Bakker E. M. (Eds.). Lecture notes in Computer Science: Vol. 4796. Computer Vision
in Human Computer Interaction. IEEE International Workshop, HCI. Berlin, Ger-
many: Springer-Verlag.
Oh, I., Frank, M., & Stone, M. (2007). Face-to-face communication of uncertainty:
expression and recognition of uncertainty signals by different levels across
modalities. In ZiF'06 Proceedings of the Embodied communication in humans and ma-
chines, 2nd ZiF research group international conference on Modeling communication with
robots and virtual humans (pp. 57-76). Berlin, Germany: Springer-Verlag.
Pelachaud, C., & Poggi, I. (2002). Subtleties of facial expressions in embodied
agents. Journal of Visualization and Computer Animation, 13, 301-312.
Poggi, I., & Pelachaud C. (2000). Emotional meaning and expression in Animated
Faces. In Paiva, A., (Ed.). Affective Interactions (pp.182-195). Berlin, Germany:
Springer-Verlag.
Ricci Bitti, P.E., (in press). Facial expression and social interaction. In Müller, C.,
Cenki, A., Fricke, E., Ladewig, S.H., McNeil, D., & Tessendorf, S. (Eds.).
Handbook body-language communication. Berlin, Germany: Mouton De Gruyter.
Rimé, B. (1983). Langage et Communication. In Moscovici, S. (Ed.). Psychologie So-
ciale (pp.415-446). Paris, France: P.U.F.
Rozin, P., & Cohen A. B. (2003). High frequency of facial expressions correspond-
ing to confusion, concentration and worry in an analysis of naturally occurring
facial expressions in Americans. Emotion, 3, 68–75.
Scherer, K. R. (1992). What does facial expression express? In Strongman, K.
(Ed.). International Review of Studies on Emotion, Vol. 2, (pp. 139-165). Chichester,
England: Wiley.

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
176
Stone, M., & Oh., I. (2008). Modeling Facial Expression of uncertainty in Conver-
sational animation. In Wachsmuth I., & Knoblich, G. (Eds.). Communication
with Robots and Virtual Humans (pp. 57-76). Berlin, Germany: Springer-Verlag.
Zaretsky, M., Mendelsohn A., Mintz M., & Hendler, T. (2010). In the Eye of the
Beholder: Internally Driven Uncertainty of Danger Recruits the Amygdala and
Dorsomedial Prefrontal Cortex. Journal of Cognitive neuroscience, 22(10), 2263-
2275.
Pio E. Ricci Bitti is Professor of Psychology at Bologna University. His main re-
search interests are: expression and recognition of emotions; emotional regulation;
health psychology; nonverbal communication; time perspective. He is author or
editor of several books; among the scientific contributions he published in nation-
al and international journals, several papers concern facial expression of emo-
tions.
Contact: pioenrico.riccibitti@unibo.it
Luisa Bonfiglioli has a background in Psychology (PhD in General and Clinical
Psychology, University of Bologna) and Music (Degree in Piano, Conservatoire of
Music, Bologna). She completed her PhD at the University of Bologna with a re-
search projects on emotions induced by music in music therapy. She received a
postdoctoral fellowship on the EU Project MIROR (Musical Interaction Relying
On Reflexion). Her main research interests include facial expression of emotion,
emotion regulation, emotion in music. Contact:
luisa.bonfiglioli@unibo.it
Paolo Melani, PhD in Psychology, is research fellow at the Department of Psy-
chology, University of Bologna; his main interests include health psychology, non-
verbal expressiveness of emotion, emotion regulation, and interpersonal commu-
nication in the clinical setting. On these topics he has published some contribu-
tions in the national and international scientific literature.
Contact:
paolo.melani@unibo.it
Roberto Caterina is associate professor of General Psychology at the University
of Bologna. He is member of the Escom (European Society for the Cognitive Sci-
ence of Music). He is interested in the emotional regulation processes especially in
therapeutic contexts. Art therapy is one of his specific interests. On that subject he
has published several papers in the international as well as in the Italian scientific
literature. Contact:
roberto.caterina@unibo.it

Ricerche di Pedagogia e Didattica – Journal of Theories and Research in Education 9, 1 (2014). Special
Issue. Communicating certainty and uncertainty: Multidisciplinary perspectives on epistemicity in everyday life. Edited by
Andrzej Zuczkowski and Letizia Caronia
Pio E. Ricci Bitti, Luisa Bonfiglioli, Paolo Melani, Roberto Caterina, Pierluigi Garotti –
Expression and communication of doubt/uncertainty through facial expression
177
Pier Luigi Garotti is associate professor of Interpersonal communication tech-
niques and Psychology of Sport at the University of Bologna. His main research
fields are related to: communication and regulation of emotion; psychology of
sport; physician-patient relationship. He has published several papers in the inter-
national as well as in the Italian scientific literature. He is member of: A.I.S.C.N.V.
(Italian Society on Non Verbal Communication); A.I.P (Italian Society of Psychol-
ogy); SIPSA (Italian Health Psychology Society).
Contact:
pierluigi.garotti@unibo.it