Fig 3 - uploaded by Tomoko Koda
Content may be subject to copyright.
Examples of six combined facial expressions created from images in Fig. 2. From top left: happy eyes and neutral mouth (HN), happy eyes and sad mouth (HS), neutral eyes and happy mouth (NH); from bottom left: neutral eyes and sad mouth (NS), sad eyes and happy mouth (SH), sad eyes and neutral mouth (SN).
Source publication
This paper reports the preliminary results of a cross-cultural study on facial regions as cues to recognize the facial expressions
of virtual agents. The experiment was conducted between Japan and Hungary using 18 facial expressions of cartoonish faces
designed by Japanese. The results suggest the following: 1) cultural differences exist when using...
Context in source publication
Context 1
... agents are frequently used in virtual worlds, online applications and (serious) games. In the current state of development of the virtual agent technology, virtual agents can express emotions in their bodily behavior, first of all, by displaying facial expressions. The culture of the virtual agent is relevant in relation to the culture of the real human interlocutor. Moreover, there are training and simulation applications emerging where the cultural identity and particularly, the bodily behavior is the major learning component of the training application [1]. Such agents have been designed under the assumption that their expressions are interpreted universally among all cultures. The basis for this assumption is the early finding about the universality of the 6 basic expressions – joy, surprise, fear, sadness and disgust - by Ekman [2]. Later works by Ekman and colleagues indicated cultural differences in perceived intensity of emotions [3, 4]. However, recent research indicates cultural differences in recognizing human facial expressions. Elfenbein et. al. [5] have coined the term cultural dialects of facial expressions: the cultural dialect, unlike a personal idiosyncratic variant, is a well identifiable specific usage of some facial signal. Such cultural dialect can be seen as an in-group advantage in human facial expression recognition, whereby human facial expression recognition is generally more accurate for perceivers from the same cultural group as expressers. These works all have been using photographs, where the cultural identity of the face was clear. Ruttkay [6] discusses further empirical findings and provides a detailed scheme of the possible factors of cultural differences in interpreting facial emotions of virtual agents. When designing virtual agents, it is a challenging and not very much exploited possibility to use non-realistic faces. There are two motivations for going for cartoon- like faces. On the one hand, it has been shown that the more realistic the design is, the more critical the human perceivers are. As the realism increases, the “uncanny valley” effect occurs [7, 8]. However, in most application contexts it is the “suspension of disbelief” which is to be achieved, not the full realism. Further on, when using non- realistic faces there are additional means of expressivity (exaggeration, usage of non- realistic features or additional signals). Hence it is interesting to study possible cultural variations in perception of cartoon-like faces. Do findings from psychology on interpreting realistic facial expressions carry over to cartoon-like faces? How do the drawing style and familiarity with non-realistic facial expressions (e.g. in the tradition of comics) influence the interpretations of cartoon-like facial expressions? Koda’s cross-cultural study [9] on the recognition of cartoon-like agent facial expressions drawn by Asian and Western designers suggests that the recognition accuracy of facial expressions is higher for virtual agents designed by the same cultural group as the subjects. E.g. Japanese cartoon-like facial expressions are recognized most accurately by Japanese, and Western cartoon-like facial expressions are recognized most accurately by western countries. The results suggest an in-group advantage is also applicable to cartoon-like facial expressions of virtual agents. Recent psychological study also investigates the cultural differences of facial expressions by focusing on the facial regions. Research on human eye movements to interpret photo realistic human facial expressions showed East Asian participants mostly focused on the eyes, and Western participants scanned the whole face [10]. Yuki et al. used pictograms and photorealistic human facial images and suggest that Americans tend to interpret emotions based on the mouth, while Japanese tend to focus on the eyes [11]. Yuki et al. state this cultural difference arises from cultural norms: that people in cultures where emotional subduction is the norm (such as Japan) would focus on the eyes, and those in cultures where overt emotional expression is the norm (such as U.S.) would focus on the mouth shape. This study applies the findings of [10, 11] to animated cartoon-like virtual agent faces to improve the culturally effective facial expression design of virtual agents. Such findings can be used not only to derive design guidelines when aiming at users of a single culture, but as adaptation strategies in applications with multicultural users. E.g. in an ATM machine or on-line shop, if the user’s cultural identity is established; the virtual agent’s facial expressions may be fine-tuned for optimal recognition for the given culture. Animating virtual agents’ facial expression is rather easily done, but much more difficult in case of physical robots. Recent social robots have cartoonish faces with limited facial expressions, e.g., Kismet [12], Nexi MDS Robot [13], iCat [14] when their research focus is not on increasing realism of a humanoid robot such as geminoids [8]. We believe providing research results on the perception of cartoonish virtual agents’ facial expressions is also meaningful in order to minimize the effort to develop social robot’s facial expressions. We investigated cultural differences in using the eye and mouth regions as cues to recognize the facial expressions of cartoonish agent faces between Hungary and Japan. We conducted a web-based survey to confirm the following hypothesis. In cartoonish faces, Japanese weigh facial cues in the eye regions more heavily than Hungarians, who weigh facial cues in the mouth region more heavily than Japanese. Section 2 describes the facial expression design and experiment design, section 3 describes the results, section 4 discusses the results, and section 5 concludes the research. Two Japanese designers designed the three agent faces shown in Fig. 1. Each face design has neutral, happy, and sad expressions. The examples of the three original facial expressions are shown in Fig. 2. The face and facial expressions were created using CharToon [15], a design and animation tool for 2D cartoon faces. The facial expressions were designed by taking the emotional expressions displayed in Ekman’s FACS (Facial Action Coding System) training material. The facial features were discussed in case of happy and sad expression [16]. Pre-evaluation of the original expressions was conducted by ten Japanese and eight Hungarians to validate that each expression conveyed the intended emotions of the designer. The static images of happy and sad expressions of the three face designs were shown randomly to the evaluators after showing the neutral expression in each session. They were asked to select the perceived emotion of the each expression (happy or sad expression) from the following four adjectives: happy, sad, surprised, fear. They wrote an adjective if they don’t find appropriate adjectives from the four provided. As a result of the pre-evaluation, all the original face designs had higher than 90% recognition accuracy among Japanese, higher than 85% among Hungarians, although we used cartoon faces and facial expressions designed by Japanese. Thus, we can assume the happy and sad expressions correctly convey the intended emotions in both countries. We also asked the perceived age of each agent face (discussed in section 4). We then created six static expressions per agent design by combining the eyes and mouths. The six combinations are: happy eyes and neutral mouth (HN), happy eyes and sad mouth (HS), neutral eyes and happy mouth (NH), neutral eyes and sad mouth (NS), sad eyes and happy mouth (SH), and sad eyes and neutral mouth (SN). The total number of combined facial expressions is 18 (six expressions x three agents). Fig. 3 shows the six combined expressions created from the original expressions in Fig. ...
Similar publications
The purpose of this study was to compare the attitude of Iranian and non-Iranian English language students' attitudes towards Computer-Assisted Language Learning (CALL). Furthermore, the relations of gender, education level, and age to their attitude are investigated. A convergent mixed methods design was used for analyzing both quantitative and qu...
Background: Domestic violence against pregnant women is becoming more frequent and intense. Studies have shown that women who were under more physical and psychological violence experienced a lot of stress and depression and had less ability to communicate with their fetuses. International statistics show a high prevalence of violence against Afgha...
In today’s knowledge-based economy, most of the knowledge-sharing studies have investigated the effects of cultural values at the national level. However, individual beliefs and behaviors might also be influenced and modified by individual cultural values. To understand the effects of cultural values, cultural effects at both the individual and cou...
This study aimed at exploring the linguistic mitigating devices of requests used by Saudi EFL learners. The participants of this study were 97 students enrolled in the English program at King Khalid University, Saudi Arabia. The data of this study were collected by Discourse Completion Test (DCT) questionnaire designed for the purpose of this study...
Several empirical studies have used the Rosenberg General Self-Esteem Scale (RGSE) to address issues of cross-cultural differences in mean level of self-esteem. The majority of these studies have mostly focused on the comparison of individual-level factor structures, leaving unanswered the issue regarding the equivalence of the factor structure of...
Citations
... Exploring the design and subsequent perception of enculturated IVAs, numerous studies have investigated the role of different external culture-specific markers, such as appearance (e.g., [20,56]), non-verbal (e.g., [28,29,31,33]) and verbal behaviour (e.g., [39,41]). However, research investigating mixed-cultural backgrounds in IVAs is rare. ...
Racial bias, implicit or explicit, is still a widespread phenomenon in our modern society, negatively affecting the way we interact with foreigners. These biases can also lead to the general tendency to avoid encounters with foreigners, which has a critical impact on society as a whole. This paper presents an approach to reduce implicit and explicit racial bias using two Intelligent Virtual Agents (IVAs) in Virtual Reality (VR). Based on previous research from social psychology and the field of enculturated IVAs, a sympathetic East African-German mixed-cultural IVA, and an antipathetic Ger-man mono-cultural IVA were implemented to interact with the participants in a virtual pub quiz. Pre-and post-intervention measures of participants' implicit and explicit bias showed a significant decrease in both scores. This work demonstrates the capability of
... Therefore, researchers mostly implement culture-specific nonverbal behaviour based on the SIAs intended original culture (e.g., Lugrin et al., 2018c) in order to design convincing mixedcultural SIAs. Research, investigating how culture-specific nonverbal behaviour observed in human-human interaction transfers to human-agent interaction has investigated gestures (e.g., Lugrin et al., 2018a; e) as well as facial expressions (e.g., Koda et al., 2010;Koda and Ruttkay, 2017). The following section presents work on the culture-specific implementation of nonverbal behaviour for SIAs, focusing on gestures, as they are more universally adaptable for social robots, since they often have limited facial expressions. ...
Despite the fact that mixed-cultural backgrounds become of increasing importance in our daily life, the representation of multiple cultural backgrounds in one entity is still rare in socially interactive agents (SIAs). This paper’s contribution is twofold. First, it provides a survey of research on mixed-cultured SIAs. Second, it presents a study investigating how mixed-cultural speech (in this case, non-native accent) influences how a virtual robot is perceived in terms of personality, warmth, competence and credibility. Participants with English or German respectively as their first language watched a video of a virtual robot speaking in either standard English or German-accented English. It was expected that the German-accented speech would be rated more positively by native German participants as well as elicit the German stereotypes credibility and conscientiousness for both German and English participants. Contrary to the expectations, German participants rated the virtual robot lower in terms of competence and credibility when it spoke with a German accent, whereas English participants perceived the virtual robot with a German accent as more credible compared to the version without an accent. Both the native English and native German listeners classified the virtual robot with a German accent as significantly more neurotic than the virtual robot speaking standard English. This work shows that by solely implementing a non-native accent in a virtual robot, stereotypes are partly transferred. It also shows that the implementation of a non-native accent leads to differences in the perception of the virtual robot.
... emotions designed by a Japanese designer are more likely to be interpreted correctly by Japanese participants. They further found that positive emotional display has a wider cultural variance in interpretation than negative ones [Koda et al. 2009] and that, albeit the mouth region seems to be more effective for conveying the emotions of facial expressions than the eye region, Japanese observers weighted facial cues more heavily in the eye region compared to Hungarians, who vice versa weighed facial cues in the mouth region more heavily [Koda et al. 2010]. In a later investigation, the authors investigated culture-specific gaze-patterns and found that Japanese participants preferred agents that showed familiar gaze [Koda et al. 2017]. ...
... , and others from the 1980's onwards, using studies done with human images 35 and confederates. Recent work over the last few years using digital avatars has further challenged 36 the universality of basic "Ekman emotions" on the basis of variations due to culture, context, and 37 age (Yuki et al., 2007;Jack et al., 2009;Koda et al., 2010;Jack et al., 2012). However, that work, 38 based heavily on visual fixation patterns, has been disputed by more recent research (see Section 39 1.2). ...
We report two experimental studies of human perceptions of robotic facial expressions while systematically varying context effects and the cultural background of subjects (n = 93). Except for Fear, East Asian and Western subjects were not significantly different in recognition rates, and, while Westerners were better at judging affect from mouth movement alone, East Asians were not any better at judging affect based on eye/brow movement alone. Moreover, context effects appeared capable of over-riding such cultural differences, most notably for Fear. The results seem to run counter to previous theories of cultural differences in facial expression based on emoticons and eye fixation patterns. We connect this to broader research in cognitive science – suggesting the findings support a dynamical systems view of social cognition as an emergent phenomenon. The results here suggest that, if we can induce appropriate context effects, it may be possible to create
culture-neutral models of robots and affective interaction.
... In a similar manner, a virtual character's expressions can be perceived differently due to the cultural background of the observer. Studies by Koda and colleagues [5], for example, show that observers from different cultures judge the emotions of a virtual character differently based on its facial expressions. Therefore designers of virtual characters should take potential cultural differences into account when simulating natural nonverbal behaviors with virtual characters. ...
Anthropomorphic user interfaces such as virtual agents or humanoid robots aim on simulating believable human behavior. As human behavior is influenced by diversifying factors such as cultural background, research in anthropomorphic user interfaces considers culture background for their behavioral models as well. This paper presents a hybrid approach of creating a culture-specific model of non-verbal behaviors for simulated dialogs based on both: theoretical knowledge and empirical data. Therefore, the structure and variables of a Bayesian network are designed based on models and theories from the social sciences, while its parameters are learned from a video corpus of German and Japanese conversations in first time meeting scenarios. To validate the model a 10-fold-cross-validation has been conducted, suggesting that with the model culture-specific behavior can automatically be generated for some of the investigated behavioral aspects.
... To advance the science of social communication and inform socially intelligent and culturally aware avatars and companion robots with precise knowledge of social information [111][112][113][114][115][116][117], it is necessary to objectively identify the low-dimensional face information, for example, facial expression patterns, that subtends different social judgments, and we shall consider how this may be done in the next section. ...
As a highly social species, humans frequently exchange social information to support almost all facets of life. One of the richest and most powerful tools in social communication is the face, from which observers can quickly and easily make a number of inferences - about identity, gender, sex, age, race, ethnicity, sexual orientation, physical health, attractiveness, emotional state, personality traits, pain or physical pleasure, deception, and even social status. With the advent of the digital economy, increasing globalization and cultural integration, understanding precisely which face information supports social communication and which produces misunderstanding is central to the evolving needs of modern society (for example, in the design of socially interactive digital avatars and companion robots). Doing so is challenging, however, because the face can be thought of as comprising a high-dimensional, dynamic information space, and this impacts cognitive science and neuroimaging, and their broader applications in the digital economy. New opportunities to address this challenge are arising from the development of new methods and technologies, coupled with the emergence of a modern scientific culture that embraces cross-disciplinary approaches. Here, we briefly review one such approach that combines state-of-the-art computer graphics, psychophysics and vision science, cultural psychology and social cognition, and highlight the main knowledge advances it has generated. In the light of current developments, we provide a vision of the future directions in the field of human facial communication within and across cultures.
Copyright © 2015 Elsevier Ltd. All rights reserved.
... Koda et al. (2008) investigate the perception of virtual characters' appearance by studying the facial expressions of avatars. In a follow-up study (Koda, Ruttkay, Nakagawa & Tabuchi 2010), they focused on different regions of the face (eyes and mouth) and conducted a cross-cultural study in Hungary and Japan. In their results, the authors report that Japanese participants found facial cues in the eye region more important than Hungarian participants, who concentrated more on facial cues in the mouth region. ...
... In cartoonish faces, (1) Japanese weigh facial cues in the eye regions more heavily than Hungarians and (2) Hungarians weigh facial cues in the mouth region more heavily than Japanese. The result of first experiment using Japanese designed faces is reported in Koda et al. (2010), which confirmed the hypothesis. This paper reports the results of the second experiment using Hungarian designed faces. ...
... Overall, the above analytical results confirmed the hypotheses: (1) Japanese weigh facial cues in the eye region more heavily than Hungarians and (2) Hungarians weigh facial cues in the mouth region more heavily than Japanese. These results with Hungarian designs further and more strongly supported the hypotheses than the ones with Japanese designs reported in (Koda et al. 2010). However, there are exceptions that do not support the hypotheses. ...
This paper reports the results of a cross-cultural study on facial regions as cues to recognize facial expressions of virtual agents. The experiment was conducted between Japan and Hungary using 30 facial expressions of cartoonish faces designed by Hungarians. The results suggest the following: (1) cultural differences exist when using facial regions as cues to recognize cartoonish facial expressions between Hungary and Japan. Japanese weighed facial cues more heavily in the eye region than Hungarians, who weighed facial cues more heavily in the mouth region than Japanese. (2) The mouth region is more effective for conveying the emotions of facial expressions than the eye region, regardless of country. Our findings can be used not only to derive design guidelines for virtual agent facial expressions when aiming at users of a single culture, but as adaptation strategies in applications with users from various cultures.
... East Asians focus more on the eyes, and Westerners more on the mouth) [23]. A number of papers have studied visual fixation patterns as the basis for these putative differences in recent years, related to that hypothesis [24,25]. However, more recent studies have provided evidence countering the use of such visual fixation patterns, noting that people are engaged in a range of information-gathering activities for a variety of purposes (not simply judging affect) when looking at other faces [26][27][28]. ...
We performed an experimental study (n=48) of the effects of context congruency on human perceptions of robotic facial expressions across cultures (Western and East Asian individuals). We found that context congruency had a significant effect on human perceptions, and that this effect varied by the emotional valence of the context and facial expression. Moreover, these effects occurred regardless of the cultural background of the participants. In short, there were predictable patterns in the effects of congruent/incongruent environmental context on perceptions of robot affect across Western and East Asian individuals. We argue that these findings fit with a dynamical systems view of social cognition as an emergent phenomenon. Taking advantage of such context effects may ease the constraints for developing culturally-specific affective cues in human-robot interaction, opening the possibility to create culture-neutral models of robots and affective interaction.
... Koda and colleagues (2008) investigate the perception of virtual characters' appearance by studying the facial expressions of avatars. In a follow-up study (Koda, Ruttkay, Nakagawa & Tabuchi 2010), they focused on different regions of the face (eyes and mouth) and conducted a cross-cultural study in Hungary and Japan. In their results, the authors report that Japanese participants found facial cues in the eye region more important than Hungarian participants, who concentrated more on facial cues in the mouth region. ...
Integrating culture into the behavioural models of virtual characters requires knowledge from very different disciplines such as cross-cultural psychology and computer science. If culture-related behavioural differences are simulated with a virtual character system, users might not necessarily understand the intent of the designer. This is, in part, due to the influence of culture on not only users, but also designers. To gain a greater understanding of the instantiation of culture in the behaviour of virtual characters, and on this potential mismatch between designer and user, we have conducted two experiments. In these experiments we tried to simulate one dimension of culture (Masculinity vs. Femininity) in the behaviour of virtual characters. We created four scenarios in the first experiment, and six in the second. In each of these scenarios the same two characters interact with each other. The verbal and nonverbal behaviour of these characters differ depending on their cultural scripts. In two user perception studies, we investigated how these differences are judged by human participants with different cultural backgrounds. Besides expected differences between participants from Masculine and Feminine countries, we found significant differences in perception between participants from Individualistic and Collectivistic countries. We also found that the user's interpretation of the character's motivation had a significant influence on the perception of the scenarios. Based on our findings, we give recommendations for researchers that aim to design culture-specific behaviours for virtual characters.
