Fig 2 - uploaded by Jens Allwood
Content may be subject to copyright.
Source publication
This paper concerns the different ways in which hesitation, and hesitation related phenomena like uncertainty, doubt and other
phenomena where lack of knowledge is involved are expressed in different cultures. The paper focuses especially on shoulder
shrugging as a signal of hesitation or uncertainty, and starts from the observation that shoulder s...
Similar publications
This work aims to recognize signs which have both manual and non- manual components by providing a sequential belief-based fusion mechanism. We propose a methodology based on belief functions for fusing extracted manual and non-manual features in a sequential two-step approach. The belief functions based on the likelihoods of the hidden Markov mode...
Citations
... It is also evident that the participants used 'ah' to choose and evaluate within themselves the next word that they would utter to continue their speech. Jokinen and Allwood, (2010) stated in their research that the uttering of "ah" during a conversation by the people in most eastern countries such as Japan and Philippines indicates that the speaker is hesitating and is uncertain on what to say or what to express. ...
This study aimed to discover the code switching found in impromptu speeches of the Grade 11 students of UM Peñaplata College. Specifically, the study sought to determine and describe the common words used in code switching in the impromptu speeches of the Grade 11 students and to analyze the types and functions of the common words they code switched. Since there have been no studies related to CS in the Island Garden City of Samal, the researchers are urged to conduct this kind of research. In addition to this, the researchers also want to know the role of CS in the academe since English language proficiency is highly necessary especially in an English classroom. The participants of this study are from UM Peñaplata College, specifically the Grade 11 Senior High School students. Among the 78 respondents, there were 67 purposively selected which were subjected to Textual Analysis. Analysis revealed four most common code-switching words committed in impromptu speeches. These are ahm, ah, kanang, and ay. The types of these code-switching words are intersentential, intra-sentential, intra-word, and tag-switching. Also, the functions of these code-switched words are fillers, hesitations, discourse markers, literary translation, and repetition of words. With these, language educators should continuously find strategies to suit the ever-changing nature of the curriculum and handle the weakness evident in the teaching of the English language.
... This is to indicate uncertainty about QUBI's interpretation of the user's request. Note that not all cultures adhere to shrugging the shoulders as a sign of uncertainty [40], especially in Eastern cultures the shrug is not necessarily interpreted as a sign of uncertainty, as it is in Western cultures. We envision this motion to be used when none of QUBIs interpretations of what was said are above a set confidence threshold (O2), as opposed to when QUBI has two highly likely interpretations (for which we use M7wiggle). ...
People’s physical movement and body language implicitly convey what they think and feel, are doing or are about to do. In contrast, current smart speakers miss out on this richness of body language, primarily relying on voice commands only. We present QUBI, a dynamic smart speaker that leverages expressive physical motion – stretching, nodding, turning, shrugging, wiggling, pointing and leaning forwards/backwards – to convey cues about its underlying behaviour and activities. We conducted a qualitative Wizard of Oz lab study, in which 12 participants interacted with QUBI in four scripted scenarios. From our study, we distilled six themes: (1) mirroring and mimicking motions; (2) body language to supplement voice instructions; (3) anthropomorphism and personality; (4) audio can trump motion; (5) reaffirming uncertain interpretations to support mutual understanding; and (6) emotional reactions to QUBI’s behaviour. From this, we discuss design implications for future smart speakers.
... In our corpus we also noticed a large number of shoulder shrugs performed by both children and adults. This signalshoulders first raised and then going back down to their initial positionis a polysemous item which may assume, depending on the context, quite diverse meanings, such as obviousness, lack of knowledge, or non-commitment (because of either carelessness or powerlessness) (Debras, 2015;Debras & Cienki, 2012;Jokinen & Allwood, 2010). Sometimes, participants (both children and adults) shrugged their shoulders in the initial phase of the guessing, and then came up with a hypothesis in the final phase. ...
Children achieve their first language milestones initially in gesture and prosody before they do so in speech. However, little is known about the potential precursor role of those features later in development when children start using more complex linguistic skills. In this study, we explore how children’s ability to reflect on their degree of uncertainty develops. A total of 40 Catalan preschool children (and 10 adults) took part in a game in which they had to guess the identity of ten objects which they could touch but not see and then state how certain they were about each guess. An analysis of the children’s reflections on their own uncertainty showed that (a) when preschoolers take a stance, they start to use multimodal uncertainty signals accurately before they are able to self-report their own level of (un)certainty; (b) they use epistemic prosodic and gestural markers before they start using lexical cues; and (c) multimodal cues signaling knowledge state become increasingly more complex over time. These findings suggest that preschoolers’ expression of knowledge states through gestures and prosody reflects an early epistemic sensitivity which develops as they get older.
... The first level is as a physical or virtual interaction space-created by the mere presence of interlocutors while the second is at the level of interactional approaches. The hybrid space must exist as a prerequisite for the articulation of cultural difference (Bhabha, 2006) during interaction as interlocutors negotiate meanings, views and identities from the stand point of differences with co-interlocutors because, these differences may influence how interlocutors articulate symbol-forming and subject constituting interactive practices as well as identification during interaction across different cultures (Jokinen & Allwood, 2010). ...
The context of interaction is ever changing but to some extent, investigating these contexts can often be confined to Inter-human interaction for example, understanding how people from different cultures create better superdiverse communities in multicultural cities and actualise participatory design of systems as culturally diverse teams. However, increasing pervasive computing is extending the boundaries of interaction to contexts such as automated checkout systems in supermarkets, satellite navigation systems giving directions to drivers and workers sharing work spaces with intelligent robots. This creates the third space of hybridity or inbetweeness where human interactional culture meets agent interactional culture thus creating a new culture. This paper reports a scoping study that uses interaction strategies to understand the patterns of nonverbal listenership behaviours of speakers of English as a second language (L2) taking instructions from English as first language (L1)-mode devices during assembly tasks. Ten participants (L2 speakers and student teachers of English) were tasked with assembling two Lego models using vague verbal instructions from a computer interface (L1 agent instructor) and one human instructor (L2 speaker of English) using his voice within a 15-minute time limit per iteration. The interface used three voices of which two are synthesised and one is non-synthesised human recording by a voice actor. A 5-hour long multimodal corpus was built and analysed from these interactions. The results indicate that although there are differences in the way humans interact with agent and other humans, it is possible for humans to treat agents the same way that they treat other humans during interaction. The results imply that English Language Teaching (ELT) theories have to understand new functions that emerge from interaction between humans and agents, attention has to be paid accommodation strategies required to enhance communication between them and ELT materials design should be user-driven.
... Since Efron's studies on culture and the production of gestures (Efron, 1941), researchers have investigated the influence of culture on the perception of gestures in intercultural exchanges (Ekman, 1976;Friesen, Ekman, & Wallbott, 1979) and have argued that some may be classified as culture-specific. A shoulder shrug, for example, is seen as a sign of confidence in the Middle East but means the opposite in most Western countries (Jokinen & Allwood, 2010), while a thumbs-up, which is understood as a representation of 'good' in Western countries, is considered obscene in the Middle East (Knapp & Hall, 2006). Gestures have also been linked to individuals being perceived as more composed, more competent (Maricchiolo et al., 2009) and have been found to influence observers' preference as individuals who produce gestures are usually liked more than those who do not (Kelly & Goldsmith, 2004). ...
... Despite the automaticity of the production of illustrators, research has shown that gestural form and use conventions are influenced by cultural norms (Gullberg, 2006). Previous research has also shown that different gestures may be decoded differently by interlocutors from different cultural backgrounds (Jokinen & Allwood, 2010;Knapp & Hall, 2006). Therefore, it is possible, as examiner F stated, that the unique illustrators used by test-taker 6 might have caused distractions during the assessment of her performance in the face-to-face speaking test. ...
This study investigated the role of gestures, smiles, and eye contact on scores assigned to English-as-an-additional-language (EAL) speakers during standardized face-to-face speaking tests. Four English-as-a-first-language examiners and four EAL test-takers participated in simulated IELTS Speaking Tests. Qualitatively, an inductive thematic analysis was conducted. Quantitatively, scores were holistically (overall scores assigned) and analytically (by criterion). Nonverbal cues were examined by the total number of cues produced by all test-takers, the frequency of production by test-taker, the frequency of production of subcategories of nonverbal cues by test-taker, and by production alongside speech or in isolation. Mimicry of nonverbal cues generated by test-takers was investigated. Test-takers’ lexical range was also analyzed vis-à-vis the scores assigned to the criterion lexical resource. Conclusions drawn from the triangulation of data sources indicate that nonverbal cues may have played a role in the assessment of the criteria fluency and coherence and pronunciation. This study adds to the current body of literature on second language assessment, which has suggested that variables other than language proficiency may play a role in scores assigned to test-takers during face-to-face speaking tests.
... SS can be expressed through a multiplicity of nonverbal behavioural cues initiated by the human body as a reaction to the current situation in everyday life. A behavioural cue is an atom action, which is atomic movement that can be described at the limb level (Poppe, 2010), or a micro movement such as a facial expression or head movement (Jokinen & Allwood, 2010). In human-to-human interaction, one person can easily 'connect' all expressed behavioural cues of his/her interlocutor and interprets them as an SS, because the communication between two people is a natural process and human social intelligence allows people to do that. ...
This chapter presents the methodology for transformation of behavioural cues into Social Signals (SSs) in human-computer interaction that consists of three steps: acquisition of behavioural cues, manual and algorithmic pre-selection of behaviour cues, and classifier selection. The methodology was used on the SS class {hesitation, no hesitation} in the interaction between a user and video-on-demand system. The first step included observation of the user during interaction and obtaining information about behavioural cues. This step was tested on several users. The second step was the manual and algorithmic pre-selection of all cues that occurred into a subset of most significant cues. Different combinations of selected cues were then used in verification process with the aim of finding the combination with the best recognition rate. The last step involved the selection of an appropriate classifier. For example, a logistic regression model was obtained in combination with four features.
||FOR FINAL PUBLISHED VERSION OF THE ARTICLE PLEASE SEND ME A REQUEST.||
... They are expressed with nonverbal behavioral cues (gestures and postures, face and eye behavior, etc.). One of the most distinct social signals in this interaction is hesitation, which can be expressed with a facial expression , head movement, shoulder movement, etc. [8] . A review of the social signal processing research domain is given in [7], [9], and [10]. ...
... Another psychological definition describes hesitation as elapsing time between the external or internal stimulation of an organism and his, her or its internal or external response [14]. Hesitation can be expressed through a facial expression, head movement, shoulder movement, prosody and special verbal markers like eh or hm [8]. ...
In this paper, we present the experimental design for the evaluation of the impact of social signal application on a user’s decision making in the area of telecommunications. The aim of the design is to show that user’s social signals are applicable feedbacks in conversational recommender systems. We use user satisfaction (with the system and content) evaluation criteria. During social interaction humans express social signals which provide quick feedbacks required by conversational recommender system. The experimental scenario is hands driven video-on-demand service with a conversational recommender system where the user selects among videos on screen. We limited our experimental scenario to the social signal of hesitation only. User is hesitating, when is faced with a variety of choices to make decisions (he is uncertain). The system adjusts the list of items to be recommended according to the extracted social signal {hesitation, no hesitation}.
... We have also collected, in collaboration with the Doshisha University in Japan, a corpus of group interactions where the three participants are chatting freely about issues that interest them. The corpus has 10 conversations with familiar interlocutors and 10 with unfamiliar interlocutors (see more in Jokinen et al., 2010). The unfamiliar conversations are comparable to the first encounters topic-wise, but differ in that there are three participants instead of two. ...
