The RRADS platform: a real road autonomous driving simulator

Abstract

This platform paper introduces a methodology for simulating an autonomous vehicle on open public roads. The paper outlines the technology and protocol needed for running these simulations, and describes an instance where the Real Road Autonomous Driving Simulator (RRADS) was used to evaluate 3 prototypes in a between-participant study design. 35 participants were interviewed at length before and after entering the RRADS. Although our study did not use overt deception---the consent form clearly states that a licensed driver is operating the vehicle---the protocol was designed to support suspension of disbelief. Several participants who did not read the consent form clearly strongly believed that they were interacting with a fully autonomous vehicle.

The RRADS platform provides a lens onto the attitudes and concerns that people in real-world autonomous vehicles might have, and also points to ways that a protocol deliberately using misdirection can gain ecologically valid reactions from study participants.

Full text

The RRADS Platform:
A Real Road Autonomous Driving Simulator
Sonia Baltodano, Srinath Sibi, Nikolas Martelaro, Nikhil Gowda, Wendy Ju
Center for Design Research, Stanford University, 94305 USA
{sbaltoda, ssibi, nikmart, ngowda, wendyju} @ stanford.edu
ABSTRACT
This platform paper introduces a methodology for simulating
an autonomous vehicle on open public roads. The paper
outlines the technology and protocol needed for running these
simulations, and describes an instance where the Real Road
Autonomous Driving Simulator (RRADS) was used to
evaluate 3 prototypes in a between-participant study design. 35
participants were interviewed at length before and after
entering the RRADS. Although our study did not use overt
deception—the consent form clearly states that a licensed
driver is operating the vehicle—the protocol was designed to
support suspension of disbelief. Several participants who did
not read the consent form clearly strongly believed that they
were interacting with a fully autonomous vehicle.
The RRADS platform provides a lens onto the attitudes and
concerns that people in real-world autonomous vehicles might
have, and also points to ways that a protocol deliberately using
misdirection can gain ecologically valid reactions from study
participants.
Author Keywords
On-The-Road Simulation; Experimentation; Wizard-of-Oz;
Research Protocols
ACM Classification Keywords
H.5.2 [Information interfaces and presentation (e.g., HCI)]:
User Interfaces – Prototyping.
INTRODUCTION
In order to design effective interfaces for emerging
autonomous vehicle technology we must study human
interactions with autonomous vehicles. Currently there are few
platforms available to support such research. Virtual lab-based
simulations excel at creating highly structured and controlled
events for studying possible human-vehicular interactions [23].
However, virtual simulations struggle to replicate inertial
forces experienced in real world scenarios. Ecological validity
is difficult to attain with digital projections of pedestrians and
vehicles. This can affect the outcome of simulation studies. For
these reasons, we were interested in developing a low-cost,
safe, and reliable methodology for creating a physical, rather
than a digital, simulation of an autonomous vehicle.
PRIOR WORK
On-Road Autonomous Driving Studies
Based on our research, this is the first published study detailing
a system used to explore autonomous vehicle-driver interaction
using commercially-available on-road cars. (A video of this
platform was published as a HRI video publication in [2].)
Previous studies detailing on-road testing of autonomous
vehicle interface prototypes require costly modifications that
allow control of the car from hidden areas of the car [21]. The
bulk of autonomous car studies focus on sensor system and
algorithm development.
There have been recent reports in the press detailing drives in
functioning autonomous vehicles capable of highway driving
[8]. A description of visual displays employed by the car was
given, including displays of car state (human control vs.
autonomous) and handoff information (time to release control,
time to retake control). However, no mention is made of
experimental interfaces being tested. In addition, we have not
found any reports detailing driver interaction and interface
systems for autonomous cars driving in urban environments as
opposed to straight, cruise control style highway driving.
Wizard of Oz and Driving Studies
The RRADS platform proposes an on road, Wizard of Oz
autonomous car simulation environment. In Wizard of Oz
studies, participants are told that they are interacting with a
computer system through an interface, when in fact a human
operator—the wizard, mediates their interactions. The name
“Wizard of Oz” comes from the novels of L. Frank Baum. The
Wizard is believed by all of the denizens of the Land of Oz to
be a magical being where in fact he is an ordinary man
employing a variety of tricks to project an illusory reality [3].
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
AutomotiveUI '15, September 01-03, 2015, Nottingham, United Kingdom
© 2015 ACM. ISBN 978-1-4503-3736-6/15/09…$15.00
DOI: http://dx.doi.org/10.1145/2799250.2799288
Figure 1 – Driving Wizard, Interaction Wizard, and
Participant locations with sightlines and camera placement.
281

The use of “the wizard in the loop” of the experimental set up
allows both the participant and the experimenters more
freedom of expression, or more systematic constraints, than
would be possible with a real computer-operated system [7].
This technique can be used for testing systems, or also as an
iterative design methodology.
Sometimes the use of the wizard is done with the participant’s
a priori knowledge, and sometimes not. The Wizard of Oz
technique is modified from “experimenter-in-the-loop”
techniques pioneered at John Hopkins, in Human Factor’s
Professor Alphonse Chapanis’ Communications Research Lab
[17]. In early natural-language-processing studies, the use of
the Wizard of Oz technique allowed developers to simulate an
interface and thereby induce participants to generate language
samples in the context of an actual task [18]. In the realm of
design research, Wizard of Oz-style techniques allowed
designers of computer-aided design systems to simulate the
tasks and rules of operation for an interactive drawing system
using only text and graphical communication over closed-
circuit television. “Doing so provides a comparatively cheap
simulator, with the remarkable advantages of the human
operator’s flexibility, memory and intelligence, and which can
be reprogrammed to give a wide range of computer roles
merely by changing the rules of operation.” [6]
In the automotive user interface space, the Wizard of Oz
technique has been used to evaluate user expectations and
develop natural language technology systems. Wizard of Oz
methodology was used by designers of VICO (Virtual
Intelligent Co-Driver) to evaluate user expectations [14], by
developers of speech-based in-car entertainment systems by
researchers at TU Munich [22] and other natural-language in-
vehicle technology systems [19], by researchers developing
gesture-based interfaces for secondary tasks in a car
environment [1], to prototype in-car controls and displays [15]
and by researchers looking at the intermodal differences in
distraction tasks while controlling automotive UIs as shown in
[13].
The RRADS platform employs Wizard of Oz control to create
the illusion of an autonomous vehicle system that is capable of
driving and navigating public roadways. The system can be
adapted to provide information to the driver and can
incorporate interface prototypes directly into the testing
platform. As an example, we used the RRADS platform to
evaluate the effectiveness of haptic pre queuing technology at
increasing trust in drivers.
Haptic Feedback for Automotive Information Interfaces
Haptic feedback through the use of vibration has been explored
as possible driver feedback systems in the context of
improving spatial awareness [20, 10] and collision avoidance
[11, 12]. These studies were conducted in both simulation and
on-road environments, respectively. Our use of on-road, in-
traffic testing aims to provide an ecologically valid
environment for testing novel haptic feedback systems in
autonomous vehicles. Although new feedback mechanisms can
be initially tested in simulation environments, on-road testing
is required to validate that drivers or passengers can
distinguish haptic feedback from vibrations due to road
conditions.
For example, vibro-tactile seat arrays have been tested in
traffic on both brick and smooth roads with drivers being able
to confidently identify various vibration patterns [16]. Hogema
et al. employ the use of a rear seat experimenter to supervise
an automated system triggering the vibration patterns to test
how well drivers could perceive the vibrations.
In previous studies, haptic feedback systems have been
focused on drivers actively engaged in the task of driving. The
RRADS platform allowed us to extend this testing paradigm
by allowing us to focus on drivers who have activated the
autonomous mode feature in their vehicle. These drivers are
essentially passengers of an autonomous car.
THE RRADS PLATFORM
Overview
We developed The Real Road Autonomous Driving Simulator
(RRADS) to explore attitudes and concerns that people may
have in real-world autonomous vehicles. We ran the RRADS
following a traditional Wizard of Oz methodology [7]. The
RRADS involved two Wizards and a single vehicle. The
Driving Wizard, drives the vehicle while the Interaction
Wizard sits in the rear. Three GoPro cameras recorded road
events, the participant’s reactions, and the actions of the
Wizards. A partition made of stiff, opaque material obfuscates
the participant’s view of the driver.
The Wizards
The Driving Wizard
The Driving Wizard must be an experienced and licensed
driver. Before running the study, they must thoroughly
familiarize themselves with the selected roadways. The driving
style of the Driving Wizard must be standardized between
participants. The Wizard must be able to accelerate and
decelerate at a constant rate and remain at stop signs for a pre-
determined amount of time. A running timer prominently
displayed on the console of the vehicle may be a helpful way
to ensure consistency. To aid in suspending belief about the
autonomous capabilities of the car, the presence of the Driving
Wizard should be obfuscated.
The Interaction Wizard
The Interaction Wizard’s primary role is to provide a foil to the
Driving Wizard. Should the participant require assistance or
wish to terminate the study at any time, the Interaction Wizard
is there to support the participant. The Interaction Wizard also
can activate any prototype interfaces that may be under test
and monitor the recording equipment while on the road.
We recommend that Driving and Interaction Wizards work as
a team to practice and learn the course together. If a prototype
interface must be activated during a critical event, such as
decelerating at a traffic light, the Interaction Wizard and the
Driving Wizard must work together to seamlessly integrate the
deceleration of the vehicle and the activation of the prototype.
The Vehicle
Vehicle Characteristics
It may be worth considering the nature of your experiment
when choosing a vehicle. The RRADS can be run in any
vehicle equipped with the necessary sight lines and the ability
to install a partition, but the physical attributes of a specific
282

car, such as the characteristics of the engine sound or
suspension system may strongly color results. There is no such
thing as a “neutral” vehicle, so the biases inherent to a specific
vehicle should be taken into account.
As an example, we ran the RRADS in two vehicles: A 2008
Jeep Compass and a 2012 Infiniti M45. We found that
although the RRADS was effective and provided ecologically
valid results in both vehicles, the participants reported
markedly different qualitative experiences in the two control
cases. The Infinity’s lower suspension system seemed to allow
the car to decelerate more smoothly, while the deceleration in
the Jeep felt more abrupt. This disparity occurred despite
having the same Driving Wizard piloting both vehicles using
the same driving style.
Partition Design
The partition must be designed to prevent the participant from
seeing the Driving Wizard, while still allowing the Driving
Wizard to use all mirrors and to see through the rear passenger
windows. It is imperative to not compromise the wizard’s
driving ability while conducting on-the-road experiments.
We found that the partitions are best deployed in a staggered
configuration. This maximizes visibility for the Driving
Wizard while still keeping hands and head hidden from view.
The partitions illustrated in Figure 2 are installed in the
Infiniti M45 and are calibrated to a Driving Wizard measured
180cm. These partitions successfully prevented participants
who were 180cm or shorter from seeing the Driving Wizard.
Participants exceeding 180cm were able to look over the
partitions and were excluded from the study.
The height of the Driving Wizard may effectively limit the
height of the participants. In this instance, taller partitions
blocked the Driving Wizard’s access to the passenger side
rear-view mirror.
The partitions are made from stiff, 2cm thick foam core board.
They affixed to the interior of the vehicle using gaffer’s tape.
This adhesive prevented the partitions from moving during the
operation of the vehicle, but could be removed if needed. Other
materials, such as particleboard or plywood, would also be
appropriate materials for partitions. It is important to note that
the partition designs in Figure 2 are merely guidelines for
future studies. Every partition pairing must be configured to a
specific vehicle’s console geometry and calibrated to the
height of the Driving Wizard.
Seat Position and Sight Lines
The seat of the Driving Wizard must be slightly forward of the
participant in order to have full view of all the necessary sight
lines. This positioning will also minimize the participant’s
awareness of the Driving Wizard indicated by sightlines shown
in Figure 1.
Passenger Side Alterations
Figure 3 – Passenger seat with steering wheel, tablet, and face
capturing camera.
A steering wheel on the participant’s side of the vehicle,
shown in Figure 3, provides an important queue that they are
something other than a passenger. We found that even a simple
alteration, such as non-functional steering wheel taped to the
dash, can enhance the overall effectiveness of the simulation.
CAMERA RECORDING
Camera Placement
Three HD Go Pro cameras are employed throughout the cabin
of the vehicle in order to record the events of the on-the-road
experiment. Locations of the cameras are shown in Figure 1.
The Participant Camera is affixed to the windshield to the
right of the passenger using a suction mount. The camera is set
up in such a way as to record the facial expressions and hand
motions of the participant. A small visor is installed above the
camera’s lens in order to protect the video from lens flare
when in direct sun.
The Driving Wizard Camera clips to the sun visor directly
above the driver’s head. This attachment point allows a clear
view of the road, as well as the speedometer and steering
wheel maneuvers made by the Driving Wizard.
The Interaction Wizard Camera, mounts to the rear right
corner of the vehicle. It is positioned to record the actions of
the Interaction Wizard as well as the interior cabin events.
Recording
The GoPro Cameras record onto local SD cards, as well as
directly to an in-vehicle laptop. The three cameras are
connected directly to a 4-channel video processor (Gra-Vue
MIO MVS-4HDMI), shown in Figure 4, connected to an
H.264 USB video encoder (Elgato 1080p Game Capture HD
Recorder). The video processor as well as the laptop is
powered by an AC power adapter drawing current from the
vehicle. The Interaction Wizard can monitor the recordings
during the drive via the laptop, ensuring that the
instrumentation is working properly throughout the study.
Figure 2 – Driver and participant partitions.
283

THE RRADS METHODOLGY
The RRADS protocol procedure has three main sections:
Meet-and-Greet, On-the-Road, and Exit Interview. Each one of
these sections supports suspension of disbelief.
The Meet-and-Greet
Consent
At the start of a session, a participant is greeted, the study is
explained, and a consent form is signed. The consent form
outlines the nature of the study and the risks and benefits of
participation.
Overt deception is not necessarily required for participants to
suspend disbelief. The consent form explains that a licensed
driver simulates the autonomous driving of the vehicle. We
also inform the participant that that the research associates
interact with the car as though it is fully autonomous, and ask
that the participant do the same.
Approaching RRADS
After the interview, the researcher leads the participant to the
RRADS, approaching it from the passenger side doors. The
vehicle is parked along the curbside with the Driving Wizard
inside but not visible through the windows of the car while the
Interaction Wizard is standing by the rear passenger door, as
seen in Figure 5. The researcher introduces the Interaction
Wizard as a research associate. The participant is told that the
Interaction Wizard will be monitoring the autonomous system,
and is asked to only interact with the Interaction Wizard in
case of emergency.
Once the Interaction Wizard enters the vehicle, the participant
can take a seat. The participant’s seat should already be located
in the appropriate place for the partitions to effectively screen
the Driving Wizard’s presence. The researcher then asks the
Interaction Wizard if all the car’s systems are working.
This allows the researcher to make sure that the cameras are
capturing the participant’s face and are properly recording. The
statement also helps further the illusion of the autonomous
vehicle.
On-the-Road
Course Selection
The On-the-Road portion of the RRADS protocol provides
both quantitative and qualitative research opportunities on the
open road. For controlled studies, the route that the vehicle will
take must be tested and well known to the Driving Wizard and
the Interaction Wizard. The pre-selected course should be
predictable and safe. Pedestrians, traffic lights, changes in
speed limits, and high-density traffic can be a source of
opportunity or complication to a study design.
Residential neighborhoods in particular may mitigate many of
the unpredictable elements inherent to a study on public
roadways. The low speeds found in these areas can facilitate
consistent driving patterns between participants. Single-lane
roads diminish the chances of unwanted cut-off events or being
forced to accommodate the unexpected maneuvers of other
vehicles.
Returning Home
The researcher should be waiting along the roadside when the
vehicle completes the course. Their presence will draw
participant’s attention as the car comes to a stop. The
researcher should open the participant’s door and engage them
in light conversation to allow time for the Driving Wizard to
drive away without being seen. The Interaction Wizard should
not exit the vehicle at this time.
Avoiding Hazards
While on the road, the Driving Wizard should abide by all
posted signs and follow traffic laws. As with any driving, there
are risks inherent to conducting a study on a public roadway.
Should an emergency occur, another vehicle should be on-call
to retrieve the participant at any time.
Exit Interview
A qualitative exit interview provides an opportunity to uncover
the salient points to the passenger’s experience. Qualitative
pilot phrases, such as “how did the drive go, in general terms?”
can yield in-depth narrative responses that can be mined post
facto.
THE RRADS IN ACTION: A STUDY OF HAPTIC PRE
QUEUE PROTOTYPES
We employed the RRADS platform to evaluate physical pre
queuing systems in autonomous vehicles. Although these
prototypes are not the focus of this method paper, they serve as
an example for the kinds of on-the-road interventions that are
possible using the RRADS.
Figure 5 – Configuration for approaching RRADS.
Researcher
Participant
Interaction Wizard
Driving Wizard (in car)
Figure 4 – Camera views: Interaction Wizard (top right),
Driving (bottom left), Participant (bottom right).
284

Method
Participants
Participants were recruited through advertisements in social
and professional networks in Palo Alto and the surrounding
environs. In total N=35 participants were recruited, ranging in
age from 18 to 36. All participants were licensed, experienced
drivers. Of the 35, 14 were men and 21 were women. A $15
gift certificate was offered as payment for participation.
Participants were run in two different vehicles. A separate
control case was run in each car. The vibration array haptic
device was implemented in the Jeep Compass and the
pneumatic floorboard and shoulder haptic devices were tested
in the Infiniti M45.
Prototype Devices
We used the RRADS platform to evaluate the effect that
physical pre queuing systems in an autonomous vehicle would
have on trust. The study evaluated three physical prototypes in
terms of their comfort and efficacy. The devices were designed
to alert participants to the autonomous vehicles’ starts, stops,
and turns.
The first prototype is a pneumatic base in the foot well of the
car that tilts in the direction that the car is about to move in.
The second prototype is a vibration array embedded into the
passenger’s seat back exhibiting various vibration patterns
corresponding with vehicle movement. The third prototype is a
pneumatic device that displaces the participant’s shoulders to
indicate if the vehicle would turn right or left. This was a
between-participant study.
In all cases the participant was asked to watch a short movie
while in the vehicle to distract them from on-road visual cues
such as stop signs and turning lanes.
The prototype devices were installed in a 2008 Jeep Compass
and a 2012 Infiniti M45. (Due to circumstances beyond our
control, we lost access to the first vehicle midway through the
study and had to switch to the second.) The vehicles were
instrumented as previously described.
Course
The course took approximately 15 minutes to complete. In this
study, the vehicle drove through a residential neighborhood
averaging 25 mph. During the drive, participants encountered
stop signs, traffic lights, construction vehicles, cyclists, and
pedestrians. A total of 14 critical events were pre queued as
shown in Figure 6. We chose to have the Driving Wizard drive
in a conservative, smooth manner, similar to a professional
limo driver, for all conditions. The same Driving Wizard drove
the same course, using the same driving pattern and speed, for
all participants to ensure consistency.
When key moments arrived, such as the acceleration of the
vehicle from a stop sign, an Interaction Wizard activated the
prototype being tested. Only one Interaction Wizard activated
a given prototype for all participants to ensure consistent
experience. In the control cases, no indication of the event was
given, other than the normal revving of the engine and the
motion of the vehicle itself.
Procedure
Pre-drive: After signing the consent form, participants were
asked a series of questions in an interview format that allowed
researchers to better understand their relationship to vehicles
and driving. This was a qualitative diagnostic interview aimed
at identifying the attitudes, expectations and previous
experiences that the participant may have regarding
autonomous driving technology.
Vehicle Introduction: Participants approached the vehicle
following the RRADS protocol. Once seated, the vehicle
greeted the participant using one of the prototype devices, or in
the control case, a revving of the car engine. This process
allowed the researcher to be sure that the prototypes were
working as expected, and that the participant had no objections
to continuing with the study.
On the Road: Participants were asked to use hand gestures to
indicate what they thought the car was about to do during the
drive. If they believed that the vehicle would turn left, they
were to raise their left hand and say, “Left.” If they believed
the vehicle would turn right, they were to raise their right hand
and say “Right.” If they believed that the car was about to stop
or accelerate, they were to raise both hands and say “Stop” or
“Go.” We collected the audio and video recordings of their
responses, the activation of the prototype device, and the
initiation of the critical event.
After initial pilot trials, we found that participants observed
environmental cues well before the Interaction Wizard was
able to provide the haptic cue. In order to discourage the
participants from reading environmental cues, they were asked
to watch a movie on a tablet that was affixed to the passenger
steering wheel, shown in Figure 3. Participants were instructed
to pay attention to the movie and were told that they would be
questioned about the movie.
Exit Interview: Following the On-The-Road portion,
participants were interviewed about their experience using a
series of open-ended prompts. Participants were never asked
directly if they believed the autonomous vehicle was fully
autonomous. Rather, a series of prompts such as “Did you trust
the vehicle?” were used to evaluate how strongly a participant
believed the simulation.
Upon the conclusion of the interview, participants completed a
qualitative and quantitative survey about their experience with
the vehicle and the prototype devices.
Figure 6 – Driven course with marked events.
285

Data Analysis
The aim of the data analysis was two-fold:
1. To establish the effect, if any, that haptic pre-queues had on
a participant’s trust of an autonomous vehicle
2. To determine the effectiveness of the RRADS at producing
ecologically valid results
Prototype Evaluation
We measured the trust and likeability of the haptic devices
through post experiment questionnaires and participant
response time to on road events captured on video. We
selected the 7 most consistent critical events along the drive,
and coded videos of all participants at these events.
We coded the videos for three points of data at each event; the
time of experimenter’s haptic cue, the time of the participant’s
response (using a hand gesture) and the time of the on-road
event. By measuring the difference in time between the device
activation and the participant’s response relative to the critical
event, we were able to infer how effectively a given device
communicated the impending action.
RRADS Evaluation
We employed qualitative research techniques focused on
evaluating the participant’s subjective experience within the
RRADS [5]. The Exit Interviews were transcribed in full, and
specific events referred to in the interviews were validated
with the video footage of the participants in the vehicles.
Interviews were analyzed in terms of Content Analysis and
Thematic Analysis [9]. A custom database was created in
Microsoft Excel and summary themes were entered and
counted.
The analysis of the interviews for major themes in the RRADS
experience was based exclusively on direct quotes. This kept
the researcher’s interpretive role to a minimum. Quotes are
included in the results below with a reference to the
participant’s anonymous identification number to illustrate our
findings [4].
Results
Cars Talk
We found that participants guess before the event was earliest
in the case of the pneumatic floorboard and the latest in the
case of the vibration array. This indicates that the floorboard
provided the most effective early-warning mechanism.
We had initially hypothesized that, on average, the time from
the participant’s response to the on-road event would be
negative in the control case, as they did not receive any haptic
cues. This, however, was not the case, as shown in Figure 7.
We found that environmental cues from the road and from the
vehicle’s motion were always present and played a significant
role in the participants’ guesses. Hence, the average time from
the participant’s response to the on-road event always preceded
the event, despite those participants not having received
prototype cues.
Figure 7 – Event response time for each prototype and control.
Note the differences in control times for each vehicle.
Every Car is Unique
It is important to note that the two control cases have
dissimilar values. We attribute this to the ride quality inherent
to the physical properties of the vehicle itself. The Infiniti was
designed to offer a more luxurious ride and may have isolated
many of the vibrational cues from the transmission system of
the car, providing fewer inherent pre queues from the vehicle’s
internal systems.
Trust Found in Driving Style
Trust was high through all conditions. In fact we did not find
any significant statistical differences between the conditions
despite the fact that several of the pre-cuing systems
(especially the floor boards) were very effective pre-cuing
devices. We hypothesize that this lack of difference can be
attributed to consistent and conservative driving style of the
driver of the RRADS platform. The signal sent from the
driving style may have been so strong that it overwhelmed the
signals from less significant inputs.
Smooth Driving is Safe Driving
The exit interviews revealed a group of interrelated themes
concerned the concepts of smooth and safe driving.
Participants often referenced the car’s safe driving style when
questioned about trust. When answering the prompt “Did you
trust the vehicle?” approximately 30% of answers heavily
correlated with descriptions of smooth driving:
I think so. The main thing was that it drove very smoothly. That
was the big thing. Participant 21
This correlation also emerged when asked to elaborate further
on why they trusted the vehicle:
Because it was smooth and it wasn't too fast or jerky... Participant 8
I think it was a really good driver. It was smooth. Participant 40
It didn't shutter or do anything imperfect that I would have
expected it to do. Participant 43
It wasn't jerky at all, which was good. It wasn't anything sudden
or things that would normally make me go oh my God, this is
scary, stop… Participant 12
It was fairly fluid and everything… Participant 18
The descriptions of smooth driving also correlated with
descriptions of vehicular planning and awareness:
286

Definitely smooth starts and stops. It sort of made it feel like the
vehicle was planning what it was doing [...] when a human stops
really quickly at a stop sign it's usually because they didn't realize
it was there, which I do sometimes. Participant 6
It was just a very sort of calm ride. Just seemed very smooth [...]
something's a bit smoother, you realize that the person or the car
kind of knows what's going on. Participant 37
The Hello Effect
At the start of the study, the vehicle greeted the participant
using one of the prototype devices, or, in the control case, with
a revving of the engine. This was built into the study for purely
practical reasons to verify that the prototype was functioning
properly and did not make the participant feel uncomfortable.
This interaction had an unintended effect on the participant’s
experience of the vehicle. Several participants cited the
greeting as a source of comfort and a sign of amicability:
… I was surprised how much I trusted it. Even from the beginning,
when it said, "Hello," it had enough of a personality. That one
thing gave it enough personality for me to trust it Participant 40
When the car said, "Turn on," or something, and then there was
the air, it just kind of shot up, and I was like, "OK, that's kind of
interesting," but it's like its way of communicating with you, rather
than a voice thing. Participant 29
Trust and Disbelief with RRADS
The RRADS protocol was not designed to employ deception.
The partition separating the Wizard Driver from the participant
was intended to help facilitate the illusion of an autonomous
vehicle, rather than to deceive. However, approximately 25%
of our participants believed that the RRADS was a fully
autonomous vehicle. Another large portion of the participants
believed the vehicle was partially autonomous and remotely
controlled by the Interaction Wizard.
The prompt “Did you trust the vehicle?” was particularly
helpful in uncovering how immersed participants became in
the study:
I guess the computer was pretty cautious, which was pretty
awesome […] It was a much better driver than most humans that I
know. Participant 27
I think, had it been my first time on the road with an automated
car, I would have been terrified, because I wouldn't even know if
this technology worked. Participant 29
[…] It made me feel like even though it wasn't a human, it wasn't
of malicious intent. Participant 40
A few participants who strongly believed the was RRADS was
fully autonomous revealed reservations about autonomous
technology:
I just don't fully trust that car to drive on its own. Even though I
had no bad experiences with this car, it just seems strange to me
still and foreign to me that a car can drive itself. Participant 31
It's more of me communicating […] it would just drift a little bit
and I told it, I said, "You have to pick a lane." (laughs) Thinking
my verbal cues would be helpful but I am a very verbal driver.
Participant 42
During more complicated maneuvers, some participants
ascribed agency to the Interaction Wizard:
There was a construction site […] The guy was waving for me to
move and I was like, I don't know what to do… so I was like, "I
really hope the car does something smart” The car backed up and
then the guy made more hand signals. I don't know if [Interaction
Wizard] or if the car did it…Participant 31
Only 4 participants out of the 35 tested indicated that they
were fully aware of the Driving Wizard. This was a
surprisingly low number of individuals given that the study
was not designed using overt deception.
Yes. I'm not sure if that's completely fair because I knew that I
wasn't alone. I'm not sure how I would feel in a vehicle that was
autonomous. [But] I think I would after having a test drive like
that. Participant 25
I'm in a study and this is probably very safe and there is somebody
actually driving and I am in the passenger seat, so yes I trust that
situation. Participant 35
DISCUSSION
Overall, results suggest that the RRADS may be an effective
way to evaluate prototypes and scenarios specific to open road
human-autonomous vehicle interactions. The RRADS
provided a useful platform for evaluating the three prototype
devices in an ecologically valid situation.
More interestingly, the RRADS platform pointed to influences
that might be greater levers in trust than pre queuing. The
Hello Effect seems to indicate that an autonomous vehicle’s
perceived personality and driving style may be incredibly
strong and salient factors in a user’s trust in a vehicle.
In addition, participants seemed to be actively evaluating the
vehicle’s competence when it encountered complex situations.
The vehicle’s apparent ability to interact with construction
vehicles or bicyclists seemed to reassure participants who were
initially skeptical of autonomous driving technology.
FUTURE WORK
Further validation of the RRADS platform should be
considered. A controlled study focused solely on the effects of
specific elements in the RRADS platform leading to the
suspension of disbelief may result in an even more effective
protocol. Possible research topics might include the effect of:
1. Removing the partitions to reveal the Driving Wizard
2. Isolating the effects of the vehicle’s suspension system
3. Varying language in the consent form to be more or less
explicit about the presence of the Driving Wizard
4. Excluding the Interaction Wizard from the vehicle
5. Varying the vehicle greeting
6. Varying the RRADS driving style
Driving style seems to be of particular importance when
evaluating prototypes focused on trust. It seems worthwhile to
better understand where the speeds and driving styles begin to
erode participant’s trust. This would provide a baseline of
distrust from which to reference the effects of a given
prototype. Further development of the RRADS may involve
devising technology that can quantitatively standardize the
Driving Wizard’s driving style. This will allow for a more
consistent experience between participants, and may facilitate
research on the effects of driving style in human - autonomous
vehicle interactions.
287

In addition, a study with and without a vehicle “greeting” may
provide insights into the effects of personifying autonomous
vehicles. This would be a useful and novel research topic for
automotive interaction.
The RRADS platform may even provide a method by which to
evaluate the effects of specific variations in a between-vehicle
study. Physical vehicular attributes such as ride quality and the
amount of sound coming from the engine can be used as levers
to explore people’s real-world relationships to autonomous
vehicles.
CONCLUSION
The RRADS platform provides an important and low-cost
solution for use in the automotive community when designing
driver interactions and user experience. It can provide useful
insights throughout the whole design process, and indicate
which features are proving to be salient with users and which
are not. In an industry where prototypes take years and
thousands of dollars to develop, it is exceedingly useful to
understand user interaction before large system level decision
are made and developed for new vehicles. The RRADS
platform acts as a low-cost rapid prototyping platform for
autonomous car interaction design and testing.
REFERENCES
1. Alpern, M., & Minardo, K. (2003). Developing a car
gesture interface for use as a secondary task. In Extended
abstracts on Human factors in computing systems
(CHI'03), 932-933.
2. Baltodano, S., Sibi, S., Martelaro, N., Gowda, N., & Ju,
W. (2015). RRADS: Real Road Autonomous Driving
Simulation. In Extended Abstracts of the Tenth Annual
ACM/IEEE International Conference on Human-Robot
Interaction (HRI’15), 283-284.
3. Baum, L. F. (1900). The wonderful wizard of Oz. Books
of Wonder.
4. Braun, V., & Clarke, V. (2006). Using thematic analysis
in psychology. Qualitative research in psychology, 3(2),
77-101.
5. Burgess, M., King, N., Harris, M., & Lewis, E. (2013).
Electric vehicle drivers’ reported interactions with the
public: Driving stereotype change? Transportation
Research Pt F: Traffic Psychology, 17, 33-44.
6. Cross, N (1977). The Automated Architect. Pion Limited.
7. Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993).
Wizard of Oz studies—why and how. Knowledge-based
systems, 6(4), 258-266.
8. Davies, A. (2015.) I Rode 500 Miles in a Self-Driving Car
and Saw the Future. It’s Delightfully Dull. Wired.com,
January 7, 2025. Available at:
http://www.wired.com/2015/01/rode-500-miles-self-
driving-car-saw-future-boring/
9. Elo, S., & Kyngäs, H. (2008). The qualitative content
analysis process. J.Advanced Nursing, 62(1), 107-115.
10. Fels, S., Hausch, R., & Tang, A. (2006). Investigation of
haptic feedback in the driver seat. In IEEE Intelligent
Transportation Systems Conference (ITSC'06), 584-589.
11. Fitch, G. M., Kiefer, R. J., Hankey, J. M., & Kleiner, B.
M. (2007). Toward developing an approach for alerting
drivers to the direction of a crash threat. Human Factors:
The Journal of the Human Factors and Ergonomics
Society, 49(4), 710-720.
12. Fitch, G. M., Hankey, J. M., Kleiner, B. M., & Dingus, T.
A. (2011). Driver comprehension of multiple haptic seat
alerts intended for use in an integrated collision avoidance
system. Transportation Research part F: Traffic
Psychology and Behaviour, 14(4), 278-290.
13. Geiger, M., Nieschulz, R., Zobl, M., Neuss, R., & Lang,
M. (2001). Methods for Facilitation of Wizard-of-Oz
Studies and Data Acquisition. In Proc. of the 9th Intl.
Conf. on Human-Computer Interaction (HCI
International 2001), New Orleans, Louisiana, USA, 5(10),
8-11.
14. Geutner, P., Steffens, F., & Manstetten, D. (2002). Design
of the VICO Spoken Dialogue System: Evaluation of User
Expectations by Wizard-of-Oz Experiments. In Proc.
Third International Conference on Language Resources
and Evaluation (LREC’02). 1588-1593.
15. Green, P., Boreczky, J., and Kim, S. (1990).Applications
of Rapid Prototyping to Control and Display Design. SAE
Technical Paper 900470., doi:10.4271/900470.
16. Hogema, J. H., De Vries, S. C., Van Erp, J., & Kiefer, R.
J. (2009). A tactile seat for direction coding in car driving:
Field evaluation. In Proc. of IEEE Transactions on
Haptics, 2(4), 181-188.
17. Kelley, J. F. (1983) An empirical methodology for writing
user-friendly natural language computer applications.
Proceedings of ACM Human Factors in Computing
systems (CHI’83), 193-196.
18. Kelley, J. F. (1985).CAL – A Natural Language program
developed with the OZ Paradigm: Implications for
Supercomputing Systems”..” In Proceedings of ACM
First International Conference on Supercomputing
Systems. 238–248.
19. Lathrop, B., Cheng, H., Weng, F., Mishra, R., Chen, J.,
Bratt, H., & Shriberg, L. (2005). A Wizard of Oz
framework for collecting spoken human-computer
dialogs: An experiment procedure for the design and
testing of natural language in-vehicle technology systems.
In Proc. 12th World Congress on Intelligent
Transportation Systems (ITS’05), 12(6) 3298-307.
20. Morrell, J., & Wasilewski, K. (2010). Design and
evaluation of a vibrotactile seat to improve spatial
awareness while driving. In Proc. IEEE Haptics
Symposium, 281-288.
21. Schmidt, G., Kiss, M., Babbel, E., & Galla, A. (2008).
The Wizard on Wheels: Rapid Prototyping and User
Testing of Future Driver Assistance Using Wizard of Oz
Technique in a Vehicle. In Proceedings of the FISITA
2008 World Automotive Congress, Munich. F2008-02-001
22. Schuller, B., Lang, M., & Rigoll, G. (2006). Recognition
of spontaneous emotions by speech within automotive
environment. Fortschritte der Akustik (DAGA’06), 32(1),
57-8.
23. Talone, A., Fincannon, T., Schuster, D., Jentsch, F. and
Hudson, I. (2013). Comparing Physical and Virtual
Simulation Use in UGV Research. Proc. of the Human
Factors and Ergonomics Society (HFES’13), 57(1), 2017-
202.
288