Content uploaded by Zhengming Zhang
Author content
All content in this area was uploaded by Zhengming Zhang on Dec 23, 2022
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=hihc20
International Journal of Human–Computer Interaction
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/hihc20
The Comfort of the Soft-Safety Driver Alerts:
Measurements and Evaluation
Zhengming Zhang, Renran Tian, Vincent G. Duffy & Lingxi Li
To cite this article: Zhengming Zhang, Renran Tian, Vincent G. Duffy & Lingxi Li (2022): The
Comfort of the Soft-Safety Driver Alerts: Measurements and Evaluation, International Journal of
Human–Computer Interaction, DOI: 10.1080/10447318.2022.2146324
To link to this article: https://doi.org/10.1080/10447318.2022.2146324
Published online: 22 Nov 2022.
Submit your article to this journal
Article views: 27
View related articles
View Crossmark data
The Comfort of the Soft-Safety Driver Alerts: Measurements and Evaluation
Zhengming Zhang
a
, Renran Tian
b
, Vincent G. Duffy
a
, and Lingxi Li
b
a
School of Industrial Engineering, Purdue University, West Lafayette, IN, USA;
b
Purdue School of Engineering and Technology, Indiana
University-Purdue University Indianapolis (IUPUI), Indianapolis, IN, USA
ABSTRACT
With the development of automated driving systems and V2I (vehicle-to-infrastructure) communi-
cations, soft-safety driver alerts can be implemented to supplement imminent driver alerts. This
type of alert improves drivers’situational awareness of emerging risks over a more extended
period with more detailed incident information and longer response time. However, compared to
the large number of studies focusing on imminent risks, there are insufficient studies on the evalu-
ation and effectiveness of soft-safety alerts. This study proposed an innovative metric to assess
the comfort and safety of soft-safety driver alerts by constructing an ideal speed profile and calcu-
lating the deviation between the actual and ideal profile. We select the highway end-of-queue
event as the experiment scenario, which is a leading cause of fatal highway crashes. Human sub-
jects’experiments are conducted in the driving simulator to validate the proposed metrics. The
results have proved that the proposed metrics have good potential to assess driving comfort
objectively. We also found that soft-safety alerts tend to improve driving comfort. However, there
is insufficient evidence to conclude statistically about the prototype soft-safety alerts implemented
in the experiments.
1. Introduction
Driver alerts have proven efficient and critical to mitigating
risks and improving driving experiences. However, unlike
commonly-discussed driver alert systems, this study focuses
on the human factors issues of the soft-safety driver alerts,
which are the alerts triggered much earlier before the acci-
dent/event to notify drivers about the upcoming traffic situ-
ation. We will first define this type of driver alert and justify
its differences and benefits from related research. Then, the
research questions and hypotheses will be summarized in
the next section.
In modern advanced driver-assistance systems, onboard
sensors can detect imminent risks inside and outside the
cabin and issue driver alerts for immediate responses. Some
examples of these systems include the forward-collision
warning system (Abe & Richardson, 2006), lane departure
warning system (Kozak et al., 2006), or driver alerts based
on the detection of impaired driving states (Lee et al., 2004;
Yekhshatyan & Lee, 2013), like driver distraction, fatigue,
and drowsiness. Different parameters and effectiveness of
these systems have also been well-studied from the human
factors perspective (Campbell et al., 2007; Lee et al., 2002).
Most research in this area tries to optimize the multi-modal
design of driver alerts to efficiently prompt imminent
responses and mitigate dangers usually coming in the next
several seconds.
As reviewed above, current driver alert studies mainly
assumed that events (like crashes or control transitions)
would happen in a relatively short period and emphasized
improving the drivers’information-processing speed and
invoking quick responses. Although such an assumption is
reasonable for many situations, and these systems undoubt-
edly improve safety and driving experiences, their effective-
ness is limited by the maximum response time in
various situations.
One good example is the driver-take-over alert for Level
2 and 3 autonomous vehicles. Control transition between
the automation system and the human driver is a well-
known dilemma impeding the implementation of autono-
mous driving systems. According to some studies (Eriksson
& Stanton, 2017; Merat et al., 2014), this transition process
may take drivers from 2 to more than 20 s to react, take
over control, and fully stabilize the car. An imminent alert
is unsuitable for the transition scenarios due to the intrinsic
short lead time. Confirmation and improvement of driver
readiness are critical to smooth this transition process. Thus,
different studies have been completed to improve the design
of takeover requests, increase situational awareness, and
reduce the driver’s reaction time toward different driving
conditions and road situations (Kim & Yang, 2017). As
summarized in a recent review of 129 randomly sampled
studies, the primary research in the area focuses on design-
ing and investigating multi-modal driver take-over requests
to reduce the reaction time below the time budget of the
CONTACT Renran Tian rtian@iupui.edu Purdue School of Engineering and Technology, Indiana University-Purdue University Indianapolis (IUPUI),
Indianapolis, IN, USA
ß2022 Taylor & Francis Group, LLC
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION
https://doi.org/10.1080/10447318.2022.2146324
vehicle system (Zhang et al., 2019). The timing, method,
and information to efficiently alert occupants of potential
risks and improve the transition process are also top human
factors concerns in autonomous vehicle development
(Seppelt & Victor, 2016).
1.1. Soft-safety driver alert
With the growth of automated driving systems and vehicle-
to-infrastructure communications, there are needs and chan-
ces to improve the current driver alerting capabilities with
longer lead time and more detailed information, referred to
as the soft-safety driver alert in some literature (Chan et al.,
2010; Nowakowski et al., 2012; Work et al., 2008). Because
of the longer lead time and richer alert information, soft-
safety alerts can potentially help design user interfaces for
autonomous driving systems with different automation lev-
els. Especially, these types of alerts intuitively promote driv-
ers’situation awareness by describing the event ahead,
inducing better human-environment interactions
(Stephanidis et al., 2019).
This study defines a soft-safety driver alert as an alert
that provides drivers with emerging risks in a more
extended period that do not require immediate responses. A
soft-safety alert is commonly triggered more than 60 s before
the risky events, while an imminent alert is triggered less
than a few seconds ahead. As a result, it can increase situ-
ational awareness with more detailed road and incident
information and boost safer driving behavior by allowing
more protracted decision-making and responding time. Soft-
safety alerts usually are triggered from half to several
minutes before the incident based on the central control
algorithms and V2I (vehicle to infrastructure) communica-
tions. Due to the difference from the imminent alert, the
soft-safety driver alerts have some intrinsic advan-
tages, including:
The amount of information delivered in the soft-safety
alert can be a lot richer and more complex through an
interactive process;
The soft-safety alert allows significantly longer informa-
tion processing and decision-making time for the drivers
so that a well-planned response can be carried out along
the longer response duration.
1.2. Safety-oriented vs. comfort-oriented evaluation
In vehicle testing and driver alert evaluation, most studies
focus on the safety aspect but pay less attention to the
importance of driving comfort (Butakov & Ioannou, 2015;
Dillen et al., 2020). We argue that although safety is the pri-
mary objective of driver alert, it is also critical to enhance
driving comfort with efficient vehicle feedback through the
alert interface. The goal is to improve driving experiences
further and prevent the drivers from disabling uncomfortable
alerts after unsatisfying experiences (Delle Site et al., 2011;
Souders & Charness, 2016). Comfort-oriented evaluation
metrics are more important for automated driving and soft-
safety alerts. Although actual accidents are critical, they
rarely happen in the road environment. However, driving
experiences and comfort are essential in many cases to affect
driver trust, acceptance, and utilization of new technologies
(Zhang et al., 2023).
Typically, assessing the comfort of driving relies on sub-
jective measurements, like surveys and interviews. Although
the subjective measurement can directly measure the per-
ceived comfort of human drivers via experiments and pro-
vide quantitative results through a Likert scale, the
measuring practice is usually discrete and difficult to apply
in real-time. Such deficiencies limit the evaluation methods
and authenticity. Consequently, the subjective driving com-
fort measurements are insufficient to support computational
algorithms for driver state tracking and efficient alerting.
This study focuses on a quantitative and computation-
efficient metric to resolve the dilemma, which can be
calculated based on driving behaviors and without human
subjects’assessments. The proposed metrics will be com-
pared to the driver’s perceived comfort to prove that it can
replace the direct measurements of driving comfort with
more convenient and real-time calculations.
1.3. Highway end-of-queue scenario
This study selected highway end-of-queue crashes as the
research application. End-of-queue crashes are leading
safety concerns for driving, accounting for about 13% of
all highway fatalities (Mekker et al., 2016). For highway
drivers, low visibility, bad road conditions, and impaired
driver states like distraction and drowsiness can signifi-
cantly increase their chances of getting into these danger-
ous situations. Because of this, end-of-queue alerting
systems have been developed via V2I communications
(Browne & Byrne, 2008) and based on the probe-vehicle
data (Ruan et al., 2019). Usually, highway slow traffic or
traffic queue conditions are tracked in real-time in the
central control server. Then, the end-of-queue alerts would
be given to the drivers approaching these queues from
upstream one to several miles ahead of the incidents.
Some of them use in-cabin alerts (Ruan et al., 2019), while
others use roadside warnings like portable variable message
signboards (Khan, 2007) and roadside special-vehicle-based
warnings (Ullman et al., 2013).
Different studies have shown the benefits of end-of-queue
alerts in smoothing the traffic flow and improving driver
situational awareness (Nowakowski et al., 2012; Tamp
ere
et al., 2009). Our previous work has shown that soft-safety
alerts improve safety when encountering an end-of-traffic
queue (Zhang et al., 2021). However, a systematic study is
lacking to evaluate the effectiveness of the soft-safety alert
on driving comfort under end-of-queue scenarios. Since
highway end-of-queue alert is a typical application of the
soft-safety alert, we use this scenario to design and conduct
simulator experiments to validate the proposed metric for
soft-safety alert about its effectiveness on driving comfort.
2 Z. ZHANG ET AL.
2. Research questions and hypotheses
Although the concept has been mentioned in some literature
(Chan et al., 2010; Work et al., 2008), the design and effect-
iveness of soft-safety driver alerts have rarely been discussed
in the current research frontier. Nowakowski et al. (2012)
used several metrics to evaluate the effects of end-of-queue
alerts on driving safety, including deceleration of active
braking, amount of coasting, RMSE (root mean square
error), peak deceleration rate, and mean deceleration rate.
Among these metrics, using RMSE from the predetermined
speed profile provides promising results to evaluate the soft-
safety alerts. However, the prior work has some limitations:
The RMSE is only used as a safety measure for the final
braking period before the events, which does not reflect
the whole cycle of driving behavior from triggering a
soft-safety alert to the event. Since the nature of a soft-
safety alert is about events happening in the future, it is
not enough to evaluate the alert only with the driving
behavior right before the event.
The predetermined speed profile is arbitrarily calculated
based on the minimum required deceleration to stop,
which is not validated to represent optimal safety or
driving comfort. In addition, the RMSE is not validated
to measure driving safety efficiently, especially consider-
ing that there are different types of soft-safety alerts.
This study proposed an innovative metric, weighted
mean absolute error (WMAE), to measure the effects of
soft-safety alerts on driving comfort during the whole alert
cycle. According to the study by af Wåhlberg (2006), both
significant acceleration and deceleration impair the passen-
gers’comfort. Also, since the soft-safety alert is designed to
inform drivers of potentially dangerous events, a set of
smooth reactions to prevent accidents is preferred.
Therefore, it is reasonable to infer that the smoothness of
speed changes could be considered an indicator of driv-
ing comfort.
Based on this intuition, we construct an ideal speed pro-
file and measure the deviation between the ideal and actual
speed profiles to assess the perceived driving comfort.
Details about the ideal speed profile and the comparison cal-
culation are introduced in Section 3.1. This study compre-
hensively evaluated the soft-safety alert using the proposed
metrics with the specific application in highway end-of-
queue scenarios. We first incorporate the results from the
driving simulator experiments and subjective surveys to val-
idate our proposed metrics and then evaluate the different
designs of soft-safety driver alerts in driving comfort. There
are three main hypotheses:
Hypothesis 1: The measurements using WMAE are related
to the subjective comfort measurements. In other words, a
higher WMAE score indicates less perceived comfort.
Hypothesis 2: The measurements using WMAE are posi-
tively related to the minimum time to collision, indicating
that larger WMAE scores will result in a longer lead time to
collision. This hypothesis proves that WMAE could also
serve as a safety indicator.
Hypothesis 3: The presence of a soft-safety alert improves
comfort when encountering traffic queues during high-
way driving.
3. Method
3.1. The proposed metric
The soft-safety driver alerts have been proven to enhance
driver’s safety by increasing situation awareness under vari-
ous scenarios (driver states and visibility conditions) (Zhang
et al., 2021). However, driving comfort has not been
researched for soft-safety alerts due to the difficulty of col-
lecting subjective measurements. The main objective of this
study is to measure comfort levels during the entire cycle
from the initial alert until the incident (approaching the end
of the queue).
We propose a new metric, weighted mean absolute error
(WMAE), based on the deviation from an ideal speed pro-
file. This new metric has four main innovative
contributions:
The new metric focuses on the whole driving period
from the soft-safety alert to the event.
The ideal speed profile is proposed based on the idea of
one-pedal driving. We validated the constructed meas-
urements using subject testing.
Compared with RMSE, the proposed WMAE provides
more degrees of freedom in customization. The weights
can be fine-tuned for different applications based on the
data to calibrate the metrics.
Also, the proposed WMAE does not overestimate large
or underestimate small deviations due to the inherent
nature of absolute value compared with the squaring cal-
culation used in other metrics. One example can be seen
in Figure 1. Compared with the WMAE calculation,
which provides more proportional scores based on the
deviations, RMSE amplifies the larger deviations while
reducing the smaller ones, with the values hard to under-
stand intuitively.
3.1.1. Ideal speed profile
The conceptualized ideal speed profile is inspired by one-
pedal-driving (OPD), commonly used on recent electric
vehicles to recuperate kinetic energy using regenerative
braking. By the name, OPD refers to driving the car with
only the gas pedal and minimizing unnecessary acceleration.
When the drivers with OPD encounter an end-of-queue
scenario, they will stop pressing the throttle once they reach
the point that the vehicles would coast to stop right in
front of the queue. Based on an online survey (van Boekel
et al., 2015), OPD has been proven to be highly preferred
by most drivers as a comfortable and energy-efficient driv-
ing strategy, offering the drivers not only psychological
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 3
comfort (energy efficiency) but also physical comfort (min-
imal actions required and maximal smoothness of
speed changes).
We will use an example to describe the calculation of the
proposed WMAE metrics in a highway end-of-queue scen-
ario, shown in Figure 1. This example assumes that the car
is driving at 60 km/h, and a soft-safety alert is triggered 80 s
before the incident (approaching the end of the traffic
queue). In an ideal speed profile, the driver will first keep a
steady car speed (0 acceleration) after receiving a well-
designed alert to know that the queue is ahead but still far
away at a certain distance. Changes in speed at this phase
indicate unnecessary distractions caused by the alert and are
not expected. Toward the ideal speed profile as OPD, drivers
only take one action to stop pushing the throttle during the
deceleration process. When getting closer to the stopped/
slowed traffic, the driver is expected to start coasting (con-
stant deceleration) after clearly identifying the incident and
eventually stop the car before a suitable distance with min-
imum active braking. The ideal speed profile is illustrated as
the blue line in Figure 1.
The point where the driver starts to coast in an ideal
speed profile is called the split point. Choosing an appropri-
ate split point is critical for the effectiveness of the metrics
and is scenario-dependent. In the end-of-queue situation
focused on in this study, we suggest that the split point is
considerably related to the speed and weather conditions. It
is obvious to slow down in a high-speed car earlier than in
a comparatively low-speed car. We could roughly estimate
the split point by the summation of Sand a suitable distance
before the queue. For example, suppose a regular driver is
driving a vehicle with a coasting deceleration rate a
c
, starting
at a constant speed V
0
on a straight two-lane highway. In
that case, we could use the following formula to estimate a
split point S.
S¼ðV0=ac
0
ðV0actÞdt (1)
3.1.2. Calculation of WMAE
It is challenging for people to achieve precisely the ideal
speed profiles in reality. We expect better profiles to be
more similar to the ideal profiles, indicating a more com-
fortable and preferred driving strategy. Hence, more signifi-
cant deviations will result in worse performance, represented
as higher scores in the proposed WMAE. For example, in
Figure 1, there are two other speed profiles where the
orange one is closer to the ideal profile than the grey one.
The proposed WMAE consists of two components:
WMAE
1
corresponds to the phase before the split point,
while WMAE
2
is after the split point.
WMAE1¼Ðt1,þ
k1aðtÞþdt þÐt1,
k2jaðtÞjdt
t1,þþt1,
(2)
WMAE2¼Ðt2,þ
k3ðaðtÞacÞþdt þÐt2,
k4jðaðtÞacÞjdt
t2,þþt2,
(3)
where k
1
,k
2
,k
3
, and k
4
are penalization parameters corre-
sponding to the acceleration before the split point (k
1
), the
deceleration before the split point (k
2
), the acceleration after
the split point (k
3
), and the deceleration after the split point
(k
4
), respectively. The variable a
c
represents the constant
deceleration rate to fully stop the car after passing the split
point in the ideal speed profile. Note that a
c
is equal to
0 m/s
2
after the vehicle stops. And aðtÞþand aðtÞis the
acceleration and deceleration at time t.
The acceleration and deceleration can be penalized differ-
ently as their effects on safety, traffic flow, and comfort dif-
fer. For example, early deceleration before seeing the end of
the queue might cause unnecessary traffic disturbances,
while continued acceleration when approaching the slow
traffic may cause crashes. These penalization parameters can
be adjusted individually to fit different situations and alerts.
In our example, we penalize the late breaks twice as much
as the early breaks, and thus the WMAE
2
score of the grey
profile is much higher than it of the orange one.
Figure 1. Blue line indicates the OPD-based ideal speed profile. The grey line and orange line are two instances of speed profile with additional driving actions.
4 Z. ZHANG ET AL.
In these metrics, WMAE
1
could capture drivers’actions
after the presence of a soft safety alert. For example, if the
delivery of the alert is inappropriate (e.g., too loud), the
driver may brake instantly in response. Note that a lower
score of WMAE
1
could represent a well-designed alert that
improves the driver’s situational awareness with minimum
distractions generated; and, thus, cause little safety concerns
at this phase. WMAE
2
aims to measure the smoothness of
the deceleration phase. Increased situational awareness will
allow the driver to efficiently plan and execute their maneu-
vers when approaching the incident and generate a lower
WMAE
2
score. On the contrary, an inconspicuous or absent
alert, in which the driver does not know any road informa-
tion ahead and keeps driving, will more likely result in sharp
brakes at the end. A good soft-safety alert system should
induce lower scores on WMAE
1
and WMAE
2
, both of which
are associated the driving safety and comfort.
3.2. Other metrics
Besides the WMAE, we also consider a widely used safety
indicator, minimum time to collision (mTTC), to examine
the effectiveness of the soft-safety alert. Unlike variables
associated with maximum, variance of, or average deceler-
ation, mTTC directly measures the minimum safety margin
(the minimum time gap during the whole encounter that
the car may crash if it keeps the instantaneous velocity). A
shorter mTTC indicates that there is a more dangerous
moment. Therefore, the soft-safety alert must maintain a
reasonably more considerable mTTC value.
In addition, the subjective measurements of perceived
comfort are collected through structured survey questions to
evaluate the proposed metric for measuring driving comfort.
Two questions for the subjects are answered after each trial
in the experiment. We applied a 7-points Likert scale to
both items. The items include:
Question 1: How do you feel about your braking timing?
Scales: very early, moderately early, slightly early, appro-
priate, slightly late, moderately late, very late.
Question 2: I feel comfortable about my driving experi-
ence of encountering the queue.
Scales: strongly agree, moderately agree, slightly agree,
undecided, slightly disagree, moderately disagree, strongly
disagree.
3.3. Design of the experiments
Driver simulator experiments were conducted to test the
effectiveness of soft-safety end-of-queue driver alerts using
the above-described metrics. We simulated the interstate
highway I-465, the ring road of Indianapolis, as the baseline
environment to test the end-of-queue alerting system. A
simulated I-465 was built on the DriveSafety DS600c high-
fidelity driving simulator in the Transportation &
Autonomous Systems Institute (TASI) at Indiana University
Purdue University Indianapolis (IUPUI), as shown in Figure
2. The total length of I-465 is about 52.79 miles (84.465 km).
The test conditions of both the external and in-vehicle views
of the instrumented driving simulator are depicted in Figure
3. Three different road segments were selected to reduce the
learning effects and predictability of the coming events, with
5, 6, and 7 miles, respectively, from the I-465 route. By the
end of each segment, a stopped traffic queue is added.
For each subject, different scenarios are randomly
assigned to the different road segments, as shown in Table
1. In combination, there are nine scenarios from two scen-
ario variables. To ensure the same viewing distance, we
maintain the weather conditions for all scenarios on a
sunny day.
The first scenario variable is the type of soft-safety alert.
Two different alert types will be studied: an in-cabin
auditory alert and a roadside traffic sign. Scenarios with
no presence of alert are considered as baseline. The in-
cabin auditory alert is issued by a smartphone pre-
installed with a”queue one mile ahead”female sound
when the driver is one mile from the designed end of
queues in the driving simulator. For more details about
Figure 2. The left figure is an overview of I-465 on the map. The figure on the right is the reconstructed route in the driving simulator.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 5
the design of the end-of-queue audio alert, please refer to
our prior work in Ruan et al. (2019).
The second scenario variable is the driver state. Three
different driver states are controlled to study their
impacts on driving performance, including normal, dis-
tracted, and drowsy.
3.4. Data collection
A total of 40 subjects were recruited to participate in the
study and assigned to two different sub-experiments. Thirty
subjects were in the first sub-experiment, where each experi-
enced all three driver states and one of the randomly
selected soft-safety alert types from the three levels. This
design avoids the difficulty of putting drivers into drowsy
states multiple times in the experiment. As a result, 10 sub-
jects experience each soft-safety alert type, including the
baseline, under all three different states. The distracted
driver state is achieved by asking the drivers to engage in
the 1-back secondary task (Kirchner, 1958), which requires
the subject to keep repeating a random number from the
pre-recorded sequence during the scenario. The drowsy state
is achieved by asking the drivers to drive for a long duration
in an endless dull session before the actual test scenario,
whose state is monitored with the Karolinska Sleepiness
Scale (KSS) (Kaida et al., 2006). The actual test scenario
starts after a 7 or higher KSS level has been achieved. Ten
subjects participated in the second sub-experiment, where
each subject experienced all three levels of soft-safety alerts
in normal conditions.
Driving speed data from all scenarios are recorded from
the soft-safety alert location until the queue position. After
each scenario, the driver comfort disagreement scale and the
braking timing scale are also collected from the subject. For
the quick survey, the subject remains in the driving simula-
tor and provides the answer orally to the data collector.
Data (n¼90) from all the trials, along with the scenario
variables, are used for the analysis in this study. For more
details on the experiment design, data collection, and data
preprocessing, please refer to our prior work (Zhang
et al., 2021).
4. Results
The results section is divided into two parts. Where the
first section aimed to validate the proposed WMAE-based
metrics based on our collected simulator data and
survey questionnaires, the second part examined the impact
of soft-safety alerts on driving comfort in the end-of-
queue situation.
For the calculation of WMAE, we set all ks to be equal to
1. This setting means the WMAE will penalize the sudden
acceleration (before and after the split point) equally as the
sudden deceleration (before and after the split point). The
constant deceleration a
c
we used in the study is an observed
average deceleration rate of 1.335 m=s2(Maurya & Bokare,
2012). Using the common driving speed of 100 km/h (speed
limit in the experiment), our estimation of the split point is
311 m before the end of the traffic queue using
Equation (3).
4.1. Metric validation
4.1.1. WMAE and driving comfort
The first objective is validating our proposed metric,
WMAE. As mentioned above, WMAE is designed to meas-
ure the comfort of the driving experience. Since we collected
subjective measurements of each end-of-queue trial via a
survey questionnaire, an intuitive way to validate WMAE is
to calculate the Pearson correlation regardless of the scen-
ario variables (type of alert and driver state). We follow
Cohen’s(2013) convention to interpret the effect size (.1–.3:
small; .3–.5: moderate; >.5: large). The main results (correla-
tions between proposed and subjective metrics) are organ-
ized in Table 2 and Figure 4, with some main findings:
We found that the Pearson correlation between WMAE
1
or WMAE
2
and comfort disagreement scale is 0.21
(p-value ¼0.048) and 0.43 (p-value <.001). Although the
Table 1. Table of experiment design.
Factor Levels
Driver state Normal Distracted Drowsy
Alert types No alert In-cabin auditory Roadside sign
Figure 3. Actual test conditions using the driving simulator.
6 Z. ZHANG ET AL.
values fall into small and moderate correlation categories,
both correlations are significant.
When we added WMAE
1
and WMAE
2
together, we
found that the correlation elevated close to large correl-
ation (correlation ¼.48, p-value <.001). Such correlation
showed that a high WMAE score is likely associated with
reduced comfort.
WMAE
1
focus on the deviation from the ideal speed pro-
file from the alert triggering to the split point, while
WMAE
2
measures the deviation after the split point. The
larger values of WMAE
2
than WMAE
1
might be because the
late experience of the braking counts more than the early
experience with respect to comfort. Based on the results, we
conclude that the WMAE
sum
is a convenient and valid met-
ric of comfort (hypothesis 1). Besides the validity of the
WMAE calculation, the result also provides evidence to sup-
port our ideal speed profile.
All the results are illustrated in the first row of Figure 4.
For WMAE
1
,WMAE
2
, and WMAE
sum
, we can see a positive
slope of the fitted line between each of them and the sub-
jective comfort disagreement scale. Please note that the sub-
jective comfort disagreement scale has level 1 for the most
comfortable feeling, and the driving comfort decreases as
the disagreement scale increases. Higher scores for the
WMAE measurements also indicate more significant devia-
tions from the ideal speed profile and worse driving com-
fort. The consistent positive relationships among the WMAE
measurements and comfort disagreement scales demonstrate
that the proposed metrics can surrogate the subjective meas-
urements for driving comfort. Furthermore, compared with
WMAE
1
and WMAE
2
, the WMAE
sum
has the smallest resid-
uals (less shadow in the figure), proving that the relationship
is stronger between the summed WMAE for the whole dur-
ation and the driving comfort feelings. Thus, the results also
confirm that WMAE
sum
is a good measurement for driv-
ing comfort.
4.1.2. WMAE and braking timing
In addition to asking about the comfort of the driving
experience directly, we also collected subjective measure-
ments for the braking timing as another indicator of driving
comfort. Note that the measurement of braking timing in
the survey is bi-directional, where the best braking timing is
represented by level 4, and the worst is either 1 or 7. We
assume that the early or late breaks equally affect driving
comfort and convert the scale ranges from 5 to 7 to the
ranges from 3 to 1.
The correlation between WMAE
1
,WMAE
2
and braking
timing is 0.26 (p-value ¼.014) and 0.21 (p-value ¼.041).
Both correlations are significant weak effects. The overall
Table 2. Correlations between WMAE and subjective metrics.
WMAE
1
WMAE
2
WMAE
sum
Perceived comfort 0.21 0.43 0.48
Braking timing 0.26 0.21 0.32
Figure 4. Scatter plots between proposed metrics of WMAE and subjective metrics of driving comfort and braking timing. For subjective comfort measurements, a
smaller disagreement score means higher comfort levels. For brake timing, level 4 is the most appropriate timing, and smaller levels indicate worse braking timing
(The blue lines indicate the best linear fit among the data, and the shadow shows residual ranges).
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 7
WMAE
sum
improved the correlation to moderate effect
(correlation ¼0.32, p-value ¼.002). All the correlations are
negative, indicating that the increased WMAE scores (worse
driving experiences) are associated with smaller levels of
braking timing scale (worse braking timing based on the
subjective responses). Therefore, we conclude that WMAE
could reflect the expectation of the braking timing well. We
calculated the correlation between braking timing and com-
fort and found a moderate correlation between them
(correlation ¼0.44, p-value <.001). Such correlation indi-
cated that low satisfaction with the braking timing could
decrease comfort, which also approves that the WMAE
measurements are significantly associated with driv-
ing comfort.
All the results are illustrated in the second row of Figure
4. For WMAE
1
,WMAE
2
, and WMAE
sum
, we can see a nega-
tive slope of the fitted line between each of them and the
subjective scales of braking timing. As described earlier,
lower levels of braking timing appropriateness indicate
worse driving performances and are associated with reduced
driving comfort. The consistent negative relationships indi-
cate that increased WMAE scores are always significantly
associated with reduced driving comfort, which confirms the
earlier findings with the driving comfort scale. Similarly, the
WMAE
sum
has the smallest residuals (less shadow in the fig-
ure) compared with WMAE
1
and WMAE
2
, proving it to be
a better measurement for braking timing and driv-
ing comfort.
4.1.3. WMAE and RMSE
As mentioned in Section 2, RMSE is another possible devi-
ation operator. We also calculated the RMSE scores accord-
ing to our conceptualized ideal speed profile and settings to
compare with WMAE. In Table 3, we showed that all the
correlation effect sizes of RMSEs are weaker than
the WMAEs. Moreover, the subjective measurements of the
appropriateness of the braking timing are barely correlated
to the RMSEs. Therefore, we conclude that WMAE is a bet-
ter deviation operator than RMSE in capturing the correl-
ation between the speed profile and subjective
measurements.
4.1.4. WMAE and mTTC
Regarding hypothesis 2, we calculated the correlations
between WMAE
1
,WMAE
2
, and the well-known safety indi-
cator, minimum time-to-collision (mTTC). The results are
0.45 (p-value <.001) and 0.17 (p-value ¼.10), respectively.
Though the WMAE
1
measures the deviation from the ideal
speed profile before the split point, it is positively related to
the mTTC (Hypothesis 2). Since WMAE
1
is a measurement
of comfort before the split point, we could infer that less
comfort before the split point might be associated with safer
brake timing. This finding may be explained for two pos-
sible reasons:
Less comfort is mainly induced by the early break, which
decelerates the vehicle more before approaching the end
of the queue and thus increases the minimum TTC.
Also, less comfort before the split point may be caused
by the stronger reactions to the soft-safety alert, which
indicates that the driver may be more cautious or pay
more attention to the on-coming risks so that safer driv-
ing is conducted.
Moreover, we found that there is no correlation between
mTTC and comfort (correlation ¼0.15, p-value ¼.172).
Though mTTC is a good safety indicator, it does not meas-
ure driving comfort very well. This result shows the unique
advantages of the proposed WMAE-based metrics.
Lastly, we computed the correlation between WMAE
1
and WMAE
2
. We found that there is no correlation between
them (correlation ¼0.01, p-value ¼.959). Therefore, the
values of WMAE
1
could not infer the value of WMAE
2
and
vice versa.
4.2. The effectiveness of soft-safety alert on
driving comfort
Our previous work (Zhang et al., 2021) has shown that soft-
safety alerts could increase safety when encountering an
end-of-queue situation. This study focuses on how soft-
safety alert affects driving comfort, evaluated with subjective
and proposed objective metrics.
4.2.1. Subjective measurements for driving comfort
We used a two-way Analysis of Variance (ANOVA) to study
the effects of the driver states and types of soft-safety alerts.
Before applying the ANOVA, we check the model assump-
tions of normality and homoscedasticity by examining their
normal quantile plot of the residuals and residuals vs. the
predicted value. No violation is found.
As illustrated in Figure 5, the average comfort disagree-
ment scale is smaller when the auditory alert is present than
in the other two scenarios. Note that lower comfort dis-
agreement scales indicate a higher perception of driving
comfort, which means that the in-vehicle auditory alert has
the highest average level of driving comfort. Also, roadside
alerts improve driving comfort compared to the baseline
scenarios when no alert is provided. However, the ANOVA
did not support the significance among different types of
alerts (F¼0.72, p-value ¼.49). Therefore, there is insuffi-
cient evidence to conclude that the implemented soft-safety
alerts in our experiment increase comfort when encounter-
ing the end-of-queue. A larger sample size may be necessary
to detect the statistical differences.
Also, the effect of driver states is not significant accord-
ing to the results of ANOVA (F¼0.45, p-value ¼.637).
Though there are no significant results from the model, we
observed a pattern in Figure 6. The average level of driving
Table 3. Correlations between RMSE and subjective metrics.
RMSE
1
RMSE
2
RMAE
sum
Perceived comfort 0.16 0.37 0.40
Braking timing 0.11 0.06 0.13
8 Z. ZHANG ET AL.
comfort disagreement for the drowsy state is higher than the
normal and distracted states, indicating lower driving com-
fort experiences.
4.2.2. WMAE-based driving comfort measurements
At last, we used the proposed metrics, WMAEs, to
re-evaluate the effectiveness of the implemented soft-safety
alert on driving comfort. According to the results of the
two-way ANOVA, we found that both types of alerts
(F¼1.56, p-value ¼.22) and driver states (F¼0.09, p-val-
ue ¼.92) are not significant factors regarding the WMAE
1
.
In other words, when the alert is triggered, we did not find
significantly different deviations from the ideal speed profile
among any combination of alert types and driver states
before the split point. Similarly, we found a near significant
difference in WMAE
2
with driver states (F¼2.63, p-val-
ue ¼.08), while types of alerts are not significant (F¼1.26,
p-value ¼.29). Moreover, we used the WMAE
sum
as the
independent variable because of the strong correlation with
the comfort. We did not find any significant results from
the ANOVA results.
From the perspective of perceived comfort, the ANOVA
results using WMAEs are consistent with the results using
subjective measurements, proving that the proposed object-
ive measurements may surrogate the subjective metrics for
driving comfort. However, both types of metrics did not
prove that the implemented soft-safety alert can improve
driving comfort (hypothesis 3).
5. Discussion and conclusion
This study focuses on an innovative measurement to assess
driving comfort based on driving behavior during the whole
cycle of soft-safety alert. Such an alert provides rich infor-
mation about upcoming road risks ahead of the events.
Moreover, it can supplement the current imminent driver
alerts by improving situational awareness over a more
extended period. However, measuring driving comfort dur-
ing such experiences with subjective measurements has
many limitations. Therefore, the proposed metrics try to
surrogate the traditional subjective surveys with computa-
tional behavior measurements relying on WMAE-based
deviations from the ideal speed profile. The calculation
details are described, and the validity of the proposed met-
rics for assessing driving comfort has been investigated by
driving simulator experiments in highway end-of-
queue scenarios.
We conducted driver simulator experiments along with
survey questionnaires to collect data for validating the pro-
posed metrics. According to the Pearson correlations
between WMAEs and survey scores on comfort disagree-
ment and braking timing, we found a significantly negative
correlation between the perception of driving comfort and
WMAE-based scores, proving the validity of the proposed
metrics to measure driving comfort. Linear models also con-
firm the findings. Furthermore, similar significant associa-
tions have been approved between the proposed metrics and
driving safety measured by mTTC, indicating the capability
of the proposed metrics to measure driving safety as well.
For both the subject measurements and the proposed
metrics, the average values show that the soft-safety end-
of-queue alerts tend to improve driving comfort. Still, the
statistical models do not confirm the above observations
based on the collected data. One explanation for such asym-
metric results is the deficiency of the experiment, like sam-
ple size limitations and the design of the prototype
alert systems.
The proposed metrics, WMAEs, could be used in any
driving scenario to assess comfort, with the kvalues
adjusted. For future work, we will fine-tune the kvalues in
the proposed WMAE and other hyper-parameters to further
improve the performance of the proposed measurements.
Moreover, a larger scale of the experiment is needed to yield
significant results regarding the effect of driver states
and types of alerts on driving comfort for the soft-safety
end-of-queue driver alert. Finally, more comparisons may be
conducted with physiological metrics to deepen the under-
standing and improve the estimation of driving comfort for
different automated car functions.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Figure 5. Average comfort disagreement scores for different types of alerts
(The error bars refer to the corresponding standard deviations).
Figure 6. Average comfort disagreements score for different driver states (The
error bars refer to the corresponding standard deviations).
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 9
ORCID
Renran Tian http://orcid.org/0000-0003-2028-3856
References
Abe, G., & Richardson, J. (2006). Alarm timing, trust and driver
expectation for forward collision warning systems. Applied
Ergonomics,37(5), 577–586. https://doi.org/10.1016/j.apergo.2005.11.
001
af Wåhlberg, A. (2006). Short-term effects of training in economical
driving: Passenger comfort and driver acceleration behavior.
International Journal of Industrial Ergonomics,36(2), 151–163.
https://doi.org/10.1016/j.ergon.2005.10.001
Browne, R., & Byrne, A. (2008). Highway 402 queue warning system
(wireless long haul application). In 2008 Annual Conference of the
Transportation Association of Canada Transportation Association of
Canada (TAC).
Butakov, V., & Ioannou, P. (2015). Driving autopilot with personaliza-
tion feature for improved safety and comfort. In 2015 IEEE 18th
International Conference on Intelligent Transportation Systems (pp.
387–393). IEEE.
Campbell, J. L., Richard, C. M., Brown, J. L., & McCallum, M. (2007).
Crash warning system interfaces: Human factors insights and lessons
learned. Report no. DOT HS 810 697. National Highway Traffic
Safety Administration.
Chan, C.-Y., Manasseh, C., & Rezaei, S. (2010). Experimental design
and evaluation of networked traveler field tests of safety applications
in California. In 2010 IEEE Intelligent Vehicles Symposium (pp.
39–44). IEEE.
Cohen, J. (2013). Statistical power analysis for the behavioral sciences.
Routledge.
Delle Site, P., Filippi, F., & Giustiniani, G. (2011). Users’preferences
towards innovative and conventional public transport. Procedia-
Social and Behavioral Sciences,20, 906–915. https://doi.org/10.1016/j.
sbspro.2011.08.099
Dillen, N., Ilievski, M., Law, E., Nacke, L. E., Czarnecki, K., &
Schneider, O. (2020). Keep calm and ride along: Passenger comfort
and anxiety as physiological responses to autonomous driving styles.
In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems (pp. 1–13). https://doi.org/10.1145/3313831.
3376247
Eriksson, A., & Stanton, N. A. (2017). Takeover time in highly auto-
mated vehicles: Noncritical transitions to and from manual control.
Human Factors,59(4), 689–705. https://doi.org/10.1177/001872081
6685832
Kaida, K., Takahashi, M., Åkerstedt, T., Nakata, A., Otsuka, Y.,
Haratani, T., & Fukasawa, K. (2006). Validation of the Karolinska
Sleepiness Scale against performance and EEG variables. Clinical
Neurophysiology,117(7), 1574–1581. https://doi.org/10.1016/j.clinph.
2006.03.011
Khan, A. M. (2007). Intelligent infrastructure-based queue-end warning
system for avoiding rear impacts. IET Intelligent Transport Systems,
1(2), 138–143. https://doi.org/10.1049/iet-its:20060086
Kim, H. J., & Yang, J. H. (2017). Takeover requests in simulated par-
tially autonomous vehicles considering human factors. IEEE
Transactions on Human-Machine Systems,47(5), 735–740. https://
doi.org/10.1109/THMS.2017.2674998
Kirchner, W. K. (1958). Age differences in short-term retention of rap-
idly changing information. Journal of Experimental Psychology,
55(4), 352–358. https://doi.org/10.1037/h0043688
Kozak, K., Pohl, J., Birk, W., Greenberg, J., Artz, B., Blommer, M.,
Cathey, L., & Curry, R. (2006). Evaluation of lane departure warn-
ings for drowsy drivers. In Proceedings of the Human Factors and
Ergonomics Society Annual Meeting (Vol. 50, pp. 2400–2404). Sage
Publications. https://doi.org/10.1177/154193120605002211
Lee, J. D., Hoffman, J. D., & Hayes, E. (2004). Collision warning design
to mitigate driver distraction. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 65–72).
Lee, J. D., McGehee, D. V., Brown, T. L., & Reyes, M. L. (2002).
Collision warning timing, driver distraction, and driver response to
imminent rear-end collisions in a high-fidelity driving simulator.
Human Factors,44(2), 314–334. https://doi.org/10.1518/
0018720024497844
Maurya, A. K., & Bokare, P. S. (2012). Study of deceleration behaviour
of different vehicle types. International Journal for Traffic &
Transport Engineering,2(3), 253–270. https://doi.org/10.7708/ijtte.
2012.2(3).07
Mekker, M. M., Remias, S. M., McNamara, M. L., & Bullock, D. M.
(2016). Characterizing interstate crash rates based on traffic conges-
tion using probe vehicle data. In Transportation Research Board
95th Annual Meeting.
Merat, N., Jamson, A. H., Lai, F. C., Daly, M., & Carsten, O. M.
(2014). Transition to manual: Driver behaviour when resuming con-
trol from a highly automated vehicle. Transportation Research Part
F: Traffic Psychology and Behaviour,27, 274–282. https://doi.org/10.
1016/j.trf.2014.09.005
Nowakowski, C., Vizzini, D., Gupta, S. D., & Sengupta, R. (2012).
Evaluation of real-time freeway end-of-queue alerting system to pro-
mote driver situational awareness. Transportation Research Record:
Journal of the Transportation Research Board,2324(1), 37–43.
https://doi.org/10.3141/2324-05
Ruan, K., Yarmand, Z., Tian, R., Li, L., Chen, Y., Li, F., & Sturdevant,
J. (2019). Highway end-of-queue alerting system based on probe
vehicle data. In International Conference on Human–Computer
Interaction (pp. 467–478). Springer.
Seppelt, B. D., & Victor, T. W. (2016). Potential solutions to human
factors challenges in road vehicle automation. In Road vehicle auto-
mation (Vol. 3, pp. 131–148). Springer.
Souders, D., & Charness, N. (2016). Challenges of older drivers’adop-
tion of advanced driver assistance systems and autonomous vehicles.
In International Conference on Human Aspects of IT for the Aged
Population (pp. 428–440). Springer.
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J.,
Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P.,
Guo, Y., Harris, D., Ioannou, A., Jeong, K., Konomi, S., Kr€
omker,
H., Kurosu, M., Lewis, J. R., Marcus, A., …Zhou, J. (2019). Seven
HCI grand challenges. International Journal of Human–Computer
Interaction,35(14), 1229–1269. https://doi.org/10.1080/10447318.
2019.1619259
Tamp
ere, C. M., Hoogendoorn, S. P., & Van Arem, B. (2009).
Continuous traffic flow modeling of driver support systems in mul-
ticlass traffic with intervehicle communication and drivers in the
loop. IEEE Transactions on Intelligent Transportation Systems,10(4),
649–657. https://doi.org/10.1109/TITS.2009.2026442
Ullman, G., Brewer, M. A., Bryden, J. E., Corkran, M. O., Hubbs,
C. W., Chandra, A. K., & Jeannotte, K. L. (2013). Enforcement
effectiveness for queue-end protection. In Traffic Law Enforcement
in Work Zones: Phase II Research. The National Academies Press.
https://doi.org/10.17226/22575
van Boekel, J., Besselink, I., & Nijmeijer, H. (2015). Design and realiza-
tion of a one-pedal-driving algorithm for the TU/e Lupo EL. World
Electric Vehicle Journal,7(2), 226–237. https://doi.org/10.3390/
wevj7020226
Work, D., Bayen, A., & Jacobson, Q. (2008). Automotive cyber physical
systems in the context of human mobility. In National Workshop on
High-Confidence Automotive Cyber-Physical Systems (pp. 3–4).
Yekhshatyan, L., & Lee, J. D. (2013). Changes in the correlation
between eye and steering movements indicate driver distraction.
IEEE Transactions on Intelligent Transportation Systems,14(1),
136–145. https://doi.org/10.1109/TITS.2012.2208223
Zhang, B., de Winter, J., Varotto, S., Happee, R., & Martens, M.
(2019). Determinants of take-over time from automated driving: A
meta-analysis of 129 studies. Transportation Research Part F: Traffic
Psychology and Behaviour,64, 285–307. https://doi.org/10.1016/j.trf.
2019.04.020
Zhang, Z., Shen, D., Tian, R., Li, L., Chen, Y., Sturdevant, J., & Cox, E.
(2021). Implementation and performance evaluation of in-vehicle
highway back-of-queue alerting system using the driving simulator.
10 Z. ZHANG ET AL.
In 2021 IEEE International Intelligent Transportation Systems
Conference (ITSC) (pp. 1753–1759). https://doi.org/10.1109/
ITSC48978.2021.9565067
Zhang, Z., Tian, R., & Duffy, V. G. (2023). Trust in automated vehicle:
A meta-analysis (pp. 221–234). Springer International Publishing.
About the authors
Zhengming Zhang is a PhD candidate at the School of Industrial
Engineering, Purdue University, West Lafayette. His research interests
are human-computer interaction, human factors, and deep learning for
intelligent transportation systems.
Renran Tian is an Assistant Professor at the Indiana University-Purdue
University Indianapolis (IUPUI). He received his Ph.D. from the School of
Industrial Engineering at Purdue University –West Lafayette in 2013. His
research interests include human-centered computing, human-AI teaming,
artificial intelligence, cognitive ergonomics, and autonomous driving.
Vincent G. Duffy is a Professor of Industrial Engineering and
Agricultural & Biological Engineering at Purdue University. He has
served as a faculty member at Purdue since 2005 and is a Fellow of the
UK Ergonomics Society (CIEHF) Chartered Institute of Ergonomics
and Human Factors in the United Kingdom.
Lingxi Li is a Professor of Electrical and Computer Engineering at
Indiana University-Purdue University Indianapolis. Dr. Li’s research
focuses on connected and automated vehicles, intelligent transportation
systems, and human-machine interaction. He has authored/co-authored
one book and over 130 research articles in refereed journals and
conferences.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 11
