Content uploaded by Seul Chan Lee
Author content
All content in this area was uploaded by Seul Chan Lee on Sep 27, 2019
Content may be subject to copyright.
Autonomous Driving with an Agent:
Speech Style and Embodiment
Abstract
A driving agent can be an effective interface to interact with
drivers to increase trust towards the autonomous driving
vehicle. While driving research on agent has mostly focused
on the voice-agent, little empirical findings on the robot-
agent were reported. In the present study, we compared
three different agents (informative voice-agent, informative
robot-agent, and conversational robot-agent) to investigate
their effects on driver perception in Level 5 autonomous
driving. A driving simulator experiment with an agent was
conducted. Twelve drivers experienced a simulated
autonomous driving and responded to Godspeed
questionnaire, RoSAS questionnaire, and social presence.
Drivers rated the conversational robot-agent as significantly
more competent, warmer, and providing higher social
presence than the other two agents. Interestingly, despite
this emotional closeness, drivers’ attitude toward the
conversational robot-agent was contradictory. They mostly
chose the conversational robot-agent as the best option or
the worst option. Findings of the present study are
meaningful as a first step of exploring the potential of
various types of in-vehicle agents in the context of
autonomous driving.
Author Keywords
Autonomous driving; voice agent; robot agent; speech style;
embodiment
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for third-party components of this work
must be honored. For all other uses, contact the Owner/Author.
AutomotiveUI '19 Adjunct, September 21–25, 2019, Utrecht, Netherlands
© 2019 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-6920-6/19/09.
https://doi.org/10.1145/3349263.3351515
Seul Chan Lee
Virginia Tech
Blacksburg, VA 24061, USA
seulchan0926@gmail.com
Harsh Sanghavi
Virginia Tech
Blacksburg, VA 24061, USA
sangjinko@vt.edu
Sangjin Ko
Virginia Tech
Blacksburg, VA 24061, USA
sangjinko@vt.edu
Myounghoon Jeon
Virginia Tech
Blacksburg, VA 24061, USA
myounghoonjeon@vt.edu
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
209
CCS Concepts
Hardware
→
Emerging technologies Analysis and design of
emerging devices and systems, Emerging interfaces
Introduction
As automated systems become more common, the concept
of human-computer interaction has evolved in a cooperative
way to increase convenience in people’s lives. This trend
can also be found in the context of driving. In the past,
drivers were required to perceive information, make
decisions, stay in their lane, and maintain the speed all by
themselves. These days, however, advanced driving
assistant systems enable vehicles to assist drivers in many
sub-tasks required for driving, such as lane keeping, speed
control, and route planning. The development of
autonomous vehicles is further accelerating these changes.
In other words, human-vehicle cooperation is an essential
element of future driving [3].
Nevertheless, many drivers are still reluctant to handover
control to the vehicle because of a lack of trust [5]. They
have doubts about a novel system that they have never
experienced before. Therefore, for the adoption of the
automated driving system, it is important to increase drivers’
trust in the automated system using the techniques of
human-vehicle interaction. One of the effective ways to
increase acceptance is using a tangible interface to show
the intentions of the system and allow users to understand
the system’s behavior. Implementing a physical or virtual
agent to interact with drivers can increase the feeling of co-
presence toward the system and give the drivers confidence
that the system is working properly.
To d a te , voice-agents have been widely implemented and
tested in the context of manual driving for the purpose
mentioned above [7,9]. Recently, researchers have tried to
explore the use of robot-agents in the autonomous driving
context [2,10–12,15,16]. Karatas et al. [10] showed that the
existence of a social robot system, equipped with three
movable heads with one degree of freedom, reduced the
reaction time to accident situations in the context of
autonomous driving. Kraus et al. [11] revealed that driver’s
trust towards automated driving systems can be enhanced
more by using a robot-agent than using a voice-only agent.
Zihsler et al. [16] tried to give information about the system
status of an automated driving vehicle by using a virtualized
avatar. This research, although in its nascent phase, has
shown the possible use cases of having a robot-agent
interact with drivers in an automated driving context.
In the present study, as a first step of exploring the potential
of various types of in-vehicle agents in the context of
autonomous driving, we compared the effects of three
different agents on driver’s perception. The role of the
agents was to give driving-related information so that the
driver can notice the intention and status of the automated
driving system. The informative voice-agent had no physical
entity and conveyed driving information via dry speech. The
informative robot-agent was an embodied robot and
conveyed the same information via dry speech. The
conversational robot-agent provided the same information
with more conversational speech, which is more human like.
Literature shows that drivers prefer interactions with the
agents that show more human like features, giving a
positive effect on the driver’s perception of trust towards the
autonomous driving system [6,11]. Based on these findings,
we hypothesized that drivers would prefer the robot-agent to
the voice-agent and the conversational type to the simple
information type.
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
210
To test these hypotheses, a driving simulator experiment
was designed and conducted. We collected and analyzed
the subjective evaluations from participants after they
experienced an automated driving situation with the agents.
Method
Experimental design A within-subject design was
implemented with three conditions (informative voice-agent
(IVA), informative robot-agent (IRA), and conversational
robot-agent (CRA)). In each condition, participants
experienced an autonomous driving journey in a driving
simulator with one of three agents. Participants ran into
eight driving events in each condition to show that the
automated driving system enables to appropriately control
the driving events (Table 1). Participants were given verbal
feedback from one of the agents in each condition before or
right after the events. Two informative agents gave
information to drivers in a simple manner; in contrast, the
conversational robot-agent conveyed information in a way
that would replicate a usual conversation with a human
collaborator (e.g., “we are entering…”, “I am sorry…”)
(Tables 2 and 3). The driving scenarios were designed
based on a straight and curved road including some traffic,
traffic signals, intersections, and road users (Figure 1). In
order to minimize the learning effects, we used one of three
different driving scenarios designed with the same city map
and the same events, however, the route and the order of
events were different for each condition (approximately 6.5
minutes). The order of the conditions was also randomized
to prevent the learning effects.
Dependent measures To co l le c t d r iv e rs ’ p e rc e pt i on o n t he
in-vehicle agent, subjective ratings of three different agents
were collected using the Godspeed questionnaire (five
factors with 24 items, 5-point Likert scale) [1], RoSAS
questionnaire (three factors with 18 items, 7-point Likert
scale) [4], and social presence (five items, 10-point Likert
scale) [13]. Also, the preference ranking among three
agents was measured.
Participants Twe lv e participants (4 female) between 20 and
42 years old (mean = 30.25, SD = 5.80) with a valid driving
license participated in this study. The mean driving year was
6.92 years and they drove an average of 5.83 times per
week.
Apparatus and stimuli The driving simulator used was a
motion-based driving simulator (NervtechTM, Ljubljana,
Slovenia) as shown in Figure 2. It was equipped with three
48’’ Samsung displays, a steering wheel, an adjustable seat,
gas and brake pedals, and a surrounded sound dome. A
humanoid robot, Nao, was used in the two robot-agent
conditions (V6 standard edition, height: 22.6 in., width: 10.8
in.). Amazon Polly text-to-speech software was used to
design the agent’s voices (https://aws.amazon.com/polly,
name: Sallie, gender: female voice, nationality: USA).
Procedure All the details about the experiment were
explained after welcoming the participants. Then, the
consent form confirmed by the Institutional Review Board
(IRB) of the university was signed and demographic
information was collected. Participants were assumed to be
in a Level 5 autonomous driving car; they were told that they
can do non-driving-related tasks if they want, such as
Internet surfing on smartphone. After every condition was
completed, participants responded to the questionnaires.
When they finished all three conditions, their preference and
the reason was lastly asked. It took approximately 45
minutes to complete all procedures.
Data analysis We could only include a small sample size as
this study is the starting point of the project. Accordingly, a
Table 1: Driving events in the scenario
Event list
1. Exit from the road and enter a new road
2. Road construction
3. Swerving a car
4. Tunnel
5. Jaywalking
6. Waiting for traffic signal
7. Turning left / right
8. All way stop intersection
Table 2: Informative agent’s script
Script list
1. Exit ahead
2. Road construction ahead
3. A car is swerving
4. Tunnel ahead
5. Jaywalker ahead
6. Red traffic light
7. Turning left / right ahead
8.This is a four-way stop intersection
Table 3: Conversational agent’s script
Script list
1. We are entering a new road
2. We are slowing down because of road
construction
3. I am sorry for the sudden slow down. A
car swerved into traffic
4. We are entering a tunnel
5. Are you okay? A man suddenly popped
out onto the road
6. We are waiting for the signal to turn green
7. We are turning left / right
8.We’ve reached a four-way stop
intersection. We are waiting for other cars
to go first
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
211
non-parametric Friedman test was performed to analyze the
differences among the conditions for each measure. If a
significant difference was found, a Wilcoxon signed-ranks
test was used for the pairwise comparisons. The 0.05
significance level was applied. All statistical analyses were
performed by IBM SPSS 25.0.
Results
The results showed there are no significant differences
among the three conditions in measures of Godspeed.
However, drivers rated the CRA as significantly more
competent and warmer than the other two types. In addition,
the social presence of the CRA was significantly higher than
the other conditions. Ta bl e 4 presents the summary of
statistical analyses. Preference rankings for each condition
were not significantly different from each other (Table 5).
Table 4: Subjective evaluation results
Items
Agent condition
Sig.
Pairwise
comparisons
IVA
IRA
CRA
Anthropomorp
hism
3.03
2.95
3.82
p = .166
-
Animacy
2.76
2.68
3.53
p = .063
-
Likeability
3.6
3.42
3.97
p = .307
-
Perceived
intelligence
3.82
3.87
3.98
p = .614
-
Perceived
safety
3.39
3.14
3.61
p = .179
-
Competence
4.61
4.89
5.47
p < 0.01
A = B < C*
Warmth
3.08
3.25
4.93
p < 0.01
A = B < C*
Discomfort
2.54
2.47
2.47
p = .928
-
Social
Presence
4.97
4.92
6.80
p < 0.01
A = B < C**
Note. *: p < 0.05, **: p < 0.01
Table 5: Preference rankings for each condition
Preference
Condition
IVA
IRA
CRA
1st
4
3
5
2nd
5
6
1
3rd
3
3
6
Note. Unit: no. of participants
Discussion and Conclusion
We explored the drivers’ subjective evaluations to three
agents in the context of autonomous driving. We were able
to identify some possibilities of the use of the robot-agent in
the autonomous driving.
First, drivers felt more intimate about the guidance of the
CRA than the informative agents. The close emotional band
is likely to play a key role in the acceptance of the system.
Second, despite this emotional closeness, drivers’ attitude
toward the CRA was contradictory. They mostly chose the
CRA as the best option or worst option (Table 5). In the case
of preferred drivers, “friendly”, “natural”, and “emotional
touch” were reasons. However, in the opposite case, they
pointed out that “it gives too much information” and
“distracted”. Therefore, it is necessary to make sure that the
agent should be familiar to drivers but not irritating drivers.
In future study, we can extend the approach to more
accurately understand the driver’s interaction with an agent.
In the present study, we only include subjective evaluations.
Drivers’ behaviors, such as glance behaviors, can be used
to observe their interaction with an agent. In addition,
different characteristics of the voice-agent suggested by
previous literature [8,14], for example, gender, urgency,
voice emotion, and speaking style (command vs.
suggestive/notification) should be tested in a robot-agent.
Figure 2: Experimental Settings
Figure 1: Screenshot of driving scenarios
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
212
References
1. Christoph Bartneck, Dana Kulić, Elizabeth Croft, and
Susana Zoghbi. 2009. Measurement instruments for the
anthropomorphism, animacy, likeability, perceived
intelligence, and perceived safety of robots. International
Journal of Social Robotics 1, 1: 71–81.
https://doi.org/10.1007/s12369-008-0001-3
2. Xiaojun Bi, Yang Li, and Shumin Zhai. 2013. FFitts law:
Modeling Finger Touch with Fitts’ Law. Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems - CHI ’13: 1363.
https://doi.org/10.1145/2470654.2466180
3. Francesco Biondi, Ignacio Alvarez, and Kyeong Ah Jeong.
2019. Human–Vehicle Cooperation in Automated Driving:
A Multidisciplinary Review and Appraisal. International
Journal of Human-Computer Interaction 35, 11: 932–946.
https://doi.org/10.1080/10447318.2018.1561792
4. Colleen M Carpinella, Michael A Perez, Alisa B Wyman,
and Steven J Stroessner. 2017. The Robotic Social
Attributes Scale (RoSAS): Development and Validation.
ACM/IEEE International Conference of HRI: 254–262.
https://doi.org/10.1145/2909824.3020208
5. Jong Kyu Choi and Yong Gu Ji. 2015. Investigating the
Importance of Trust on Adopting an Autonomous Vehicle.
International Journal of Human-Computer Interaction 31,
10: 692–702.
https://doi.org/10.1080/10447318.2015.1070549
6. Philipp Hock, Johannes Kraus, Marcel Walch, Nina Lang,
and Martin Baumann. 2016. Elaborating Feedback
Strategies for Maintaining Automation in Highly Automated
Driving. In 8th International Conference on Automotive
User Interfaces and Interactive Vehicular Applications.
(AutomotiveUI ’16), 105–112.
https://doi.org/10.1145/3003715.3005414
7. Myounghoon Jeon, Bruce N. Walker, and Thomas M.
Gable. 2015. The effects of social interactions with in-
vehicle agents on a driver’s anger level, driving
performance, situation awareness, and perceived
workload. Applied Ergonomics 50: 185–199.
https://doi.org/10.1016/j.apergo.2015.03.015
8. Myounghoon Jeon, Bruce N. Walker, and Thomas M.
Gable. 2015. The effects of social interactions with in-
vehicle agents on a driver’s anger level, driving
performance, situation awareness, and perceived
workload. Applied Ergonomics 50: 185–199.
https://doi.org/10.1016/j.apergo.2015.03.015
9. Yeon Kyoung Joo and Roselyn J. Lee-Won. 2016. An
Agent-Based Intervention to Assist Drivers Under
Stereotype Threat: Effects of In-Vehicle Agents’
Attributional Error Feedback. Cyberpsychology, Behavior,
and Social Networking 19, 10: 615–620.
https://doi.org/10.1089/cyber.2016.0153
10. Nihan Karatas, Soshi Yoshikawa, Shintaro Tamura, Sho
Otaki, Ryuji Funayama, and Michio Okada. 2017. Sociable
driving agents to maintain driver’s attention in autonomous
driving. In RO-MAN 2017 - 26th IEEE International
Symposium on Robot and Human Interactive
Communication, 143–149.
https://doi.org/10.1109/ROMAN.2017.8172293
11. Johannes Maria Kraus, F Nothdurft, P Hock, D Scholz, W
Minker, and M Baumann. 2016. Human after all: Effects of
mere presence and social interaction of a humanoid robot
as a co-driver in automated driving. In 8th International
Conference on Automotive User Interfaces and Interactive
Vehicular Applications, AutomotiveUI 2016, 129–134.
https://doi.org/10.1145/3004323.3004338
12. Johannes Maria Kraus, Jessica Sturn, Julian Elias Reiser,
and Martin Baumann. 2015. Anthropomorphic agents,
transparent automation and driver personality: towards an
integrative multi-level model of determinants for effective
driver-vehicle cooperation in highly automated vehicles.
Adjunct Proceedings of the 7th International Conference
on Automotive User Interfaces and Interactive Vehicular
Applications - AutomotiveUI ’15: 8–13.
https://doi.org/10.1145/2809730.2809738
13. Kwan Min Lee, Younbo Jung, Jaywoo Kim, and Sang
Ryong Kim. 2006. Are physically embodied social agents
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
213
better than disembodied social agents?: The effects of
physical embodiment, tactile interaction, and people’s
loneliness in human-robot interaction. International Journal
of Human Computer Studies 64, 10: 962–973.
https://doi.org/10.1016/j.ijhcs.2006.05.002
14. Clifford Nass, Ing-Marie Jonsson, Helen Harris, Ben
Reaves, Jack Endo, Scott Brave, and Leila Takayama.
2005. Improving automotive safety by pairing driver
emotion and car voice emotion. May 2014: 1973.
https://doi.org/10.1145/1056808.1057070
15. Satoshi Okamoto and Shin Sano. 2017. Anthropomorphic
AI Agent Mediated Multimodal Interactions in Vehicles. In
9th International ACM Conference on Automotive User
Interfaces and Interactive Vehicular Applications
(AutomotiveUI ’17), 110–114.
https://doi.org/10.1145/3131726.3131736
16. Jens Zihsler, Philipp Hock, Marcel Walch, Kirill Dzuba,
Denis Schwager, Patrick Szauer, and Enrico Rukzio. 2016.
Carvatar: Increasing Trust in Highly-Automated Driving
Through Social Cues. In Proceedings of the 8th
International Conference on Automotive User Interfaces
and Interactive Vehicular Applications Adjunct -
Automotive’UI 16, 9–14.
https://doi.org/10.1145/3004323.3004354
AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands
214
