Content uploaded by Feng Gao
Author content
All content in this area was uploaded by Feng Gao on Dec 08, 2020
Content may be subject to copyright.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
1
Automatic Virtual Test Technology for Intelligent
Driving Systems Considering both Coverage and
Efficiency
Feng Gao*, Jianli Duan, Zaidao Han and Yindong He
Abstract - The testing of the intelligent driving systems is
faced with the challenges of efficiency because real traffic
scenarios are infinite, uncontrollable and difficult to be precisely
defined. Based on the complexity index of scenario that designed
to measure the test effect indirectly, a new combinational testing
algorithm of test cases generation is proposed to make a balance
among multiple objects including test coverage, the number of
test cases and test effect. Then a joint simulation platform based
on Matlab, PreScan and Carsim is built up to realize the
construction of 3D test environment, execution of test scenarios
and evaluation of test results automatically and seamlessly. The
strategy proposed in this paper is validated by applying it to a
traffic jam pilot system. The result shows that the proposed
strategy can improve the overall complexity of the designed test
scenarios effectively, which can help us detect system faults
faster and easier. And the time required to conduct tests is
reduced obviously by means of automation.
Index terms – Autonomous vehicles, intelligent driving
systems, model-in-the-loop testing, automatic test and
evaluation, combinational testing
I. INTRODUCTION
Along with the development of basic theory and key
technology of artificial intelligence, vehicles have become
more and more intelligent [1][2]. Variety of intelligent
driving systems (IDS) have been put into market, e.g.,
intelligent cruise system and automatic parking system.
These IDS help reduce the occurrence of accidents and
enhance traffic safety efficiently. Compared with the
traditional onboard systems, almost infinite traffic scenes
make it challenging to achieve an efficient and complete test,
because the application condition cannot be defined precisely
and the influence factors with their possibilities are numerous
[3][4]. To ensure the functionality, performance and
reliability, manufactures have to take a large amount of
naturalistic field operational tests (NFOT), which cost huge
human and time resources [5]~[7].
Today global manufactures, technical companies,
researchers and so on devote themselves to accelerating the
evaluation process by introducing such virtual test
technologies as model-in-the-loop (MIL), hardware-in-the-
loop (HIL) and etc., which have already been successfully
applied in the development of traditional vehicle electronic
systems [4][8][9]. Some special tools have been developed to
promote the application of virtual test technologies to the
development of IDS including such simulation software as
PreScan, VTD, IPG Carmaker, and the radar signal and visual
simulators, etc. [9]~[11].
With the help of these tools, the collected data about
drivers, vehicles and traffic environment can be replayed to
IDS. Zhou et al. set up a HIL platform integrated with IPG
Carmaker to replay the data of GPS, camera and other sensors
[12]. To accelerate the test process, a parallel platform was
designed in [13]. Brannstrom et al. and Winkle et al. used the
accident database to evaluate a collision avoidance system
and a vision-related system respectively under extreme
conditions [14][15]. Such playback testing methods have the
following disadvantages: (1) The test process is open-loop
essentially because the record data cannot react to the
response of IDS dynamically; (2) The coverage and
effectiveness are determined directly by the collected data
whose completeness is still difficult to be evaluated
objectively.
To realize a closed-loop test, some random methods have
been adopted to generate test cases according to the
occurrence probability of the traffic factors. Based on the on-
road driving data, stochastic cases were constructed by the
Crude Monte Carlo method to test the collision avoidance
system [16] and lane departure correction system [17]. These
cases have a good consistency with the real traffic scenes in
This work was supported in part by the Natural Science Foundation of
Chongqing under grant cstc2019jcyj-zdxmX0018 and the Sichuan
Science and Technology Program under grant 2020YFSY0070. (Feng
Gao and Jianli Duan contributed equally to this work. Corresponding
author: Feng Gao). Copyright (c) 2015 IEEE. Personal use of this
material is permitted. However, permission to use this material for any
other purposes must be obtained from the IEEE by sending a request to
pubs-permissions@ieee.org.
F. Gao is with the School of Automotive Engineering, Chongqing
University, Chongqing, 400044, China and Shanghai Jiao Tong
University Sichuan Research Institute, Chengdu, 610200, China (email:
gaofeng1@cqu.edu.cn).
J. Duan is with the School of Vehicle and Mobility, Tsinghua
University, Beijing, 100000, China (email: duanjianli@cqu.edu.cn).
Zaidao Han is with the School of Automotive Engineering, Chongqing
University, Chongqing, 400044, China (email:
20193413006T@cqu.edu.cn).
Y. He is with the Mechanical Engineering, University of Michigan, MI
48109, USA (email: heyingd@umich.edu).
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
2
the statistical sense, and theoretically the required coverage
can be achieved with adequate sampling points. But the
efficiency of these methods is low because of the limited
exposure of critical conditions in real traffic. To overcome
this problem, Huang et al. proposed an accelerating testing
method which can generate more intense interactions among
different vehicles using the importance-sampling theory [18],
and used it to evaluate the function and capability of the
automatic lane change system [19]. However, the test
effectiveness of this method greatly depends on the
completeness of the database that can correctly reflect the
statistical characteristics of the real traffic. Moreover, when
to stop the random generation process has become a new
issue, because it is hard to measure the adequacy of testing.
Another method widely used in engineering fields is to
verify the systems with a test matrix (TM) that composed of
a set of test cases, in which the constituent factors can be
obtained from multiple sources [19]. A TM can be designed
in two ways: (1) Using the typical use cases such as test
standards etc. (2) Ensuring full combinations of all factors,
which is also referred to as the exhaustive testing (ET)
method. Because IDS is safe-related and required to be
validated adequately, besides its running environment is
uncontrolled and has almost infinite possibilities, therefore
verifying its performance with only the typical usage
conditions is nowhere near enough[20]. On the other hand,
the ET method will lead to an “exponential explosion” issue
in the number of test cases when the quantity of considered
factors is large [21].
To overcome these challenges, some researchers
introduced the combinatorial testing (CT) method to design
TM [22][23]. CT method can generate test cases that can
cover all the -wise combinations of considered factors. The
scale of test cases set is compact even when the number of
factors becomes huge. The fundamental of CT method is that
most of the system defects are caused by the interaction of
few factors [22][23]. This approach has been adopted in the
testing of medical devices [24], nuclear station software [22]
and etc. However, most of the studies focus on the algorithm
of test cases number reduction[22]~[25]. The test
effectiveness of the generated cases is hardly considered,
which cannot facilitate the increase of overall test efficiency.
Moreover, existing commercial simulation test tools for IDS
mainly provide the modelling function of 3D test scenarios.
However, the process of building these scenarios, conducting
test and evaluating results still needs to be realized manually.
In order to solve the above problems and improve the test
efficiency of IDS, this paper proposes a new virtual test
strategy including the design process of test scenarios and the
application process of automatic test and evaluation. Besides
the requirement of test coverage and number of test cases, a
new CT algorithm for test cases generation is further
proposed to improve the test effect based on the complexity
index of test scenario. To make a balance between the test
cases number and overall test effect, the Bayesian method is
adopted to realize the black-box optimization using a few
samples. With the designed scenarios, a joint simulation
platform based on Matlab, PreScan and Carsim is set up to
reduce the test executation time by automation. It can
construct the 3D test environment from the generated
scenarios, conduct tests, evaluate the simulation results and
generate test reports automatically and seamlessly.
The paper is organized as follows: Section II introduces the
overall framework of the automated virtual testing strategy.
A new combinational test cases generation algorithm is
introduced in Section III. In Section IV, the efficiency of the
proposed strategy is validated by appling it to a traffic jam
pilot (TJP) system. And finally Section V concludes the paper.
II. FRAMEWORK OF AUTOMATIC VIRTUAL TESTING
In order to realize the automatic test of IDS for better
efficiency, the complete process of the proposed virtual
testing strategy is shown in Fig. 1
Weather
Curvature
Roadside
facilities
Time
Lane line
clarity
Lane
line
color
Location
of HV
High
efficiency
High
coverage
Desigh
process
MATLAB
Influence
factors analysis Test cases
generation Clustering into
scenarios
Automation testing
script
Scenario mod el
Dynamic
model
Sensor
model
SIMULINK &
Intelligent
driving logic
Driver model
Automation evaluation
script
.m
script
.m
script
Apply
process
Fig. 1. Automatic virtual test strategy for intelligent driving system
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
3
There are two key steps contained in Fig. 1:
(a) Design process (Upper part in Fig. 1). It is an offline
process that can generate test scenarios considering multiple
aspects including test coverage, test cost and test effect
automatically. This procedure consists of the following three
main stages, namely, influence factors analysis, test cases
generation, and clustering test cases into dynamical test
scenarios.
(b) Apply process (Lower part in Fig. 1). In this process,
the test scenarios designed offline are applied into a joint
simulation platform, which can construct 3D test
environment, perform test scenarios execution and evaluate
test results automatically by integrating ‘Prescan’, ‘Carsim’
and ‘Maltab/Simulink’ together with the help of ‘Matlab’ m-
script.
A. Offline design of test scenarios
The driving performance of IDS is highly related to the
complexity of its application environment. If the designed
test scenario is simple and easy to be handled, the fault
detection rate of it will become comparatively low, even
though all the considered factors are fully covered. Therefore,
it is necessary to design more effective test scenarios while
taking the test coverage and test cost into account. However,
it is almost impossible to evaluate the effectiveness of
scenario without conducting the test, because the algorithms
of IDS are too complicated to establish an analytical
relationship between the test scenario and the system
response.
To overcome this challenge and realize the evaluation of
the test effect indirectly, a tree model is built first to analyze
the influencing factors that are used to construct a test case.
All these factors should affect the functionality and
performance of the tested IDS, as shown in Fig. 2.
Fig. 2. Tree structure model of the influencing factors in the test cases
The key influencing factors of IDS in Fig. 2 can be
obtained from some existing databases, such as the 6 hours of
traffic scenarios collected by a variety of sensor modalities in
KITTI [36], the Ford campus vision and LiDAR dataset
which includes 2 months of the real road scenes collected
from the research campus and downtown Dearborn [37], etc.
These datasets collected from real life can be used as one of
the most important sources of the factors. Besides, factors
analyzed from such sources as technical specifications,
accident scenarios database, test standards, etc., should also
be taken into consideration as a supplement. These resources
are helpful to the determination of the key influence factors.
For the continuous or unbounded ones, theoretically it is
impossible to test the system under all possible conditions.
From a testing point of view, there exist some engineering
ways to discretize these types of factor into limited and
representative points, e.g., equivalence class division,
boundary value analysis and etc. [4][9] To acquire the
relationships among these factors simultaneously, the tree
model can be denoted as:
(1)
where is the -th factor located in the -th layer of the tree
model, is consisted of all subscripts of the sub-factors
that subordinated to , is the number of layer, and is
the number of factors in the -th layer. Among all the factor
nodes, the ones that cannot be further divided into more
factors are called the end nodes factors. A test case can be
constructed by combining all these end nodes factors together
with one of their corresponding values. Because all the sub-
factors that are subordinate to a certain factor can finally exist
in one test case, while the different values of the same end
node factor cannot. Therefore, we need another formulation
to represent the relationship between the factors and their
values, as shown below:
(2)
where is the -th value of the factor , and is
consisted of all the subscripts of all the end nodes factors.
Then a TM, that is the set of test cases can be expressed as:
(3)
where denotes the end node factor, is the number of test
cases, is the number of end node factors, and .
Here the symbol represents the number of elements.
denotes the value of the -th factor in the -th case. The details
of the generation method of more effective test cases by
taking their complexity into account is studied in section III.
Since a test case only represents a specific
working condition and is different from the real traffic, which
is dynamical and continuous. Therefore, similar test cases are
then clustered into continuous test scenarios for better
temporal continuity. In order to consider the practical
restrictions of environments and the complexity and number
of test scenarios in the same time, the following weighted
Euclidean distances is used here to measure the similarity
between the test cases, and a similarity matrix can be
formed as:
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
4
(4)
where represents the distance between test case and ,
is the number of clusters, is the number of remaining
scenarios, which decreases with the increasing of . is the
continuity index of the -th factor and is the importance
index of value . is the importance index, which is used
to quantify the ability of the factors or values to degrade
system performance, and is also used as the input parameter
of the test cases generation algorithm (See section III). The
index is used to measure the change capacity of the
corresponding factor in time domain. For such factors as
weather and etc. that the values of which are not easy to
change, is set to a relatively large value. The hierarchical
clustering algorithm [26] is selected to cluster the scenarios
because of its advantages of easy-using and suitable for large
sample data.
Finally, a specific application instance is shown below to
better illustrate the implementation process of the test
scenarios design method. For example, according to the
analysis method of influencing factors shown in Fig. 2, we
can finally acquire “weather” as an end node factor, and
“sunny,” “cloudy”, “rainy”, “snowy” as the value nodes that
used to indicate the different states of “weather” in the test
cases. Then, by combining all these end node factors and their
corresponding values together, a complete test case can be
obtained. Taking “sunny” as the value of “weather” as an
example, the -th test case can be recorded as
weather:sunny, as shown in (3). Then, after calculating
the importance indices of different values with the
complexity index calculation method proposed in section III,
the test matrix containing all test cases is obtained by
utilizing the proposed test cases generation algorithm CTBC
in the same section. Afterwards, the similarity degree
between any two test cases in is calculated by (4). At last,
the most similar test cases are clustered into one test scenario,
so as to improve the test efficiency.
B. Automatic test and evaluation process
IDS interacts with traffic environments closely. Some
commercial tools, such as Prescan, IPG Carmaker and etc.,
have been developed to simulate the traffic environments and
sensors, such as radar, LiDAR, vision and etc. But the
processes of building the 3D test environment, conducting the
test and evaluating the test results are still performed
manually, which is time-consuming and can hardly meet the
critical requirement of vehicle development cycle.
To increase the test efficiency through automation, a joint
simulation platform is developed by combing Prescan,
Matlab and Carsim together as shown in the lower part of Fig.
1. Prescan provides the 3D modelling environment for traffic
and physical models of sensor. The vehicle dynamics is
simulated by Carsim. And the tested algorithm runs in
Matlab/Simulink. With these commercial tools, the following
programs can be developed to realize the full automation of
test with the help of Matlab m-script, as shown in Fig. 3.
(1) Automatic test program. It reads the data of the scenario
generated by the offline design process, constructs the three
dimensional test environment in Prescan through its model
API, integrates all the simulation models together in
Matlab/Simulink, sets the parameters of models, controls the
simulation process and stores the test results from Matlab
workspace to disk;
(2) Automatic evaluation program. It reads the test results
from disk, evaluates the data according to the embedded
evaluation criteria and generates the test report in Microsoft
Word format by the COM add-ins of Word.
Automation program
Scenario data
Vehicle mo del Traffic envir onment
Sensor model
IDS algor ithm
Disk
m-script
Report
COM
Interface Interface
Test data
Fig. 3. Integrated automatic simulation test platform
In the development process of the automation testing
program, there are several technical details that need to be
further illustrated:
(1) The initial state of the tested IDS and the vehicle where
it is mounted on is different from the required one of the
designed test scenario. This might cause unreasonable
responses of the IDS, which will lead to wrong judgement.
To overcome this problem, and realize a fast and smooth
transition from one scenario to another, a driver model is used
to control the vehicle to reach the initial condition of the test
before IDS takes over the control authority.
(2) The values of the influencing factors should be
discretized first when constructing the tree structure model.
This will cause unreasonable sudden values changes of such
factors as speed and etc. between two successive cases, which
is not consistent with the real vehicle dynamics and
kinematics characteristics. Besides, the generated test
scenario does not contain the time characteristic. Therefore,
a duration of 20 seconds for each case is set to fully test each
factor’s specific value contained in the cases, and avoid the
impact of external noise disturbance on the evaluation results.
Besides, in order to enhance the time continuity of the test
scenario and avoid the inconsistency with the actual traffic
environment caused by the sudden change of values of
specific factors such as the subject vehicle’s longitudinal
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
5
speed and etc., this paper utilizes the linear interpolation to
make the discrete values of factors of adjacent test cases in a
test scenario to change continuously, so as to make the
evaluation results more reasonable.
(3) To reduce the testing cycle, the multi-threaded parallel
technology based on Matlab is adopted. The test results of the
previous scenario are evaluated by the evaluation program,
while in the meantime a new test scenario starts to run in
parallel without stopping the test program.
(4) When there exist conflicts or errors, for example, the
value “3.75m” is incorrectly assigned to the factor “Self-
Vehicle Speed” (See Table 4), the scenario will be skipped to
ensure that the automation process will not be stopped. The
error is recorded in the report to facilitate the tester’s further
decision on how to deal with it.
III. TEST CASES GENERATION ALGORITHM ENSURING
COVERAGE AND EFFICIENCY
One key problem of the above-mentioned automatic virtual
test strategy is the generation algorithm of the test cases that
can ensure both coverage and efficiency requirements of IDS
test. Because of the complexity of IDS, it is hard to establish
an analytic relationship between the test effectiveness of the
test scenarios and the degree of performance degradation of
IDS. And it also brings great challenges on the optimization
of the test cases further considering the test effectiveness
besides the number of cases. An indirect evaluation index
called complexity is designed in section A to measure the test
effectiveness with the motivation that IDS is more sensitive
to its important influence factors.
A. Complexity index of test case
To fully use the experiences of engineers, the analytic
hierarchy process (AHP) method [9] is adopted to measure
the relative importance of the factors according to the tree
model of influence factors (See Fig. 2), and the subjectivity
can be eliminated as far as possible by utilizing the Delphi
method [27]. Then, the relative importance indices of the
factors or values can be derived as:
(5)
where
is composed of all the relative importance indices
of influence factors or its corresponding values that
belong to the factor node . Then, all relative importance
indices are placed in the same reference frame and
normalized according to the tree structure (See Fig. 2):
(6)
where is the importance index of , and
is
composed of the subscript of the nodes in the route from
value to the root node factor in the tree structure
model. The complexity index of the i-th test case is
expressed as the accumulation of all the importance indices
of corresponding values:
(7)
For simplicity, the matrix of importance indices in
accordance with is defined as:
(8)
B. Test case generation algorithm considering complexity
To facilitate the introduction of the proposed test cases
generation algorithm, some definitions and fundamental
concepts of CT method are given first.
Definition 1 [23]: Coverage of -wise combinations. For
any influence factors, if all the possible combinations of
their corresponding values are covered by at least one test
case in , then it is said that can fulfill the complete
coverage of -wise combinations. Here is called as the
strength of combination coverage.
The fundamental of CT is illustrated in Fig. 4 by realizing
2-wise (also referred to as pair-wise) coverage of three factors
as an example [23].
(a) Influencing factors and their corresponding values
(b) TM that can fulfill the coverage of 2-wise combinations
Fig. 4. Fundamental diagram of CT
In Fig. 4 (a), each horizontal level represents a factor, each
node in the layer represents a value. Each edge between two
nodes represents a combination that need to be covered,
which can be expressed as:
(9)
where denotes the set of remaining uncovered -wise
combinations, is the -th -wise combination and
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
6
denotes the number of combinations contained in . When
a test case is generated, only one node will and must be
selected in each level, so each case is a subgraph that
composed of edges connecting 3 nodes, which is shown in
Fig. 4(b). Then the subgraphs (a) to (f) form the smallest set
of subgraphs that can cover all the edges in Fig. 4(a) at least
once, that is, one of the smallest scale of TM that fulfills
the coverage of 2-wise combinations. Since minimizing the
set scale meanwhile meeting the coverage requirement is a
challenging task and proven to be NP-complete [22], most
researches focus on the reduction algorithms of the number
of test cases [23]~[25]. Therefore, it becomes even more
challenging to further consider the complexity requirement of
the generated test cases. For example, the importance index
of in Fig. 4 is assumed to be and then the complexity
indexes of the test cases, i.e., subgraphs (a) to (f) in Fig. 4(b),
are
, , ,
, and ,
respectively. Besides the number of subgraphs required to
cover the required combination strength, the overall
complexity of generated test cases should also be considered.
In order to solve the above problems, a new test cases
generation algorithm called “Combinatorial Testing Based on
Complexity” (CTBC) is designed. Its fundamental principal
is that if more combinations with larger sum of importance
indices can be covered by a test case early, then the cases
generated in this stage will have high overall complexity. The
sum of importance indices can be described as:
(10)
where is the set of subscript of the values contained in ,
represents the sum of importance indices of the -th
combination, and is the set of all . However, as the
generation process proceeds, the sum of importance indices
of the remained combinations will be too small to be used to
construct high complexity cases. Therefore, a threshold value
is set up to tackle this problem. By setting , the
combinations can be preliminarily screened to determine
which of the following strategies is implemented:
(1) If , the algorithm will give priority to improving
the complexity of the generated test cases, that means when
a new case is generated, the values with the maximum
importance indices will be assigned to the unassigned factors;
(2) Otherwise, the algorithm will give priority to
decreasing the number of the generated test cases. When this
happen, the new case tends to covering the largest number of
combinations in , which means the coverage
requirement is preferred.
To describe the above-mentioned decision-making process
more clearly, a schematic diagram of the implementation
process is shown in Fig. 5.
Fig. 5. Schematic diagram of CTBC algorithm
As shown in the figure, the generation process of each new
test case mainly includes the following key steps: (1)
Generate the set of combinations that remain uncovered; (2)
Select the -wise combination with the largest sum of
importance indices; (3) Compare the sum of the chosen
combination’s importance indices with the threshold value;
(4) According to the comparison results, the unassigned
factors in the new test cases are assigned by choosing a
balance point between the above two generation strategies of
“improving the complexity of cases first” and “reducing the
number of cases first”. The generation process generates one
case at a time and lasts until all the combinations in
are covered. In this way, we can control the number of test
cases in a more reasonable range under the premise of
ensuring coverage, and effectively improve the overall
complexity of the generated cases at the same time.
To facilitate the practical application, a new parameter that
called the complexity improvement index is
introduced to help select the proper by normalizing the
improvement of complexity:
(11)
where and are the minimum and maximum
accessible complexity index respectively. The optimization
of this parameter is studied in the next section. The
pseudocode of CTBC algorithm is shown in Algorithm 1.
Algorithm 1: Pseudocode for CTBC algorithm.
1: Input: , , , , ,
2: Output:
3: Obtain , , , , .
4: while do
5: Pick with the max() in .
6: for all in do
7: Assign factors and values from to .
8: Remove from .
9: end for
10: if > then
11: while ((length of ) < ) do
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
7
12: Pick with the max() in .
13: Compare values of the same factors between and .
14: if there exist conflicting values then
15: Continue to select the next in .
16: end if
17: Assign factors and values from to .
18: Remove from .
19: end while
20: else
21: Assign factor in that are not contained in with the
value with .
22: end if
23: Add to .
24:end while
When programming the executable code, it should be
noted that:
(1) All the combinations contained in are stored in a
red-black tree in descending order of the sum of importance
indices for the convenience of target searching ;
(2) Considering the repeatability and deterministic
requirement, the lexicographical order is used in the
algorithm to achieve the pseudo-random effect.
C. Parameter optimization of CTBC
The overall performance of CTBC is determined by its
parameter . To make a balance between the number and
effectiveness of the test cases, the overall complexity of is
needed to be defined first:
(12)
where and are the overall complexity and the
complexity of the -th test case respectively, is the
number of the test cases. With this definition, the optimal
can be found by the following optimization problem:
(13)
where represents the test effect, and are
normalized by using the inverse tangent function to obtain
and respectively. This facilitates the selection
of , which makes a balance between the effectiveness
improvement and the cost reduction.
Because and cannot be derived directly from
, that makes a black box function of . Therefore,
such traditional optimization methods as gradient descent and
etc. become inapplicable in this case. In this paper, the
Bayesian optimization method which can quickly find the
optimal solution by using a small number of samples is
utilized to solve this problem [28]. The optimization process
includes two parts: Gauss process regression (GPR) and
acquisition function (AF).
GPR assumes that satisfies the multivariate Gauss
distribution. sample points are selected randomly as the
input of the algorithm first, namely , , …, , where
is a relative small positive integer. Then, combined with the
output , , …, of CTBC, the training
samples can be obtained as , , …,
. And the prior distribution of can be
expressed as:
(14)
where , represents obeys Gaussian
distribution, , . Here represents the
mean value of the Gaussian distribution. In order to simplify
the calculation, we set . is the covariance matrix,
which can be expressed as:
(15)
where is calculated by the Matern 5/2 kernel [29]:
(16)
where here represents the 2-norm of vector. is called
as the characteristic length scale, which briefly defines how
far apart the input values can be for the output values to
become uncorrelated. Then, for a new point , is
predicted by:
(17)
where , . And the covariance matrix
and can be formed as:
(18)
The prior distribution of the predicted value of the
new sample points is shown as follows:
(19)
where and represent the mean and variance
respectively. And there is:
(20)
With this posterior probability distribution of the predicted
value, AF can be used to choose the next sampling point so
that GPR can more accurately approximate the actual
distribution of the black box function more accurately with a
relatively few samples. Among the AF methods, the most
commonly used ones are the probability of improvement (PI)
[34], the expected improvement (EI) [35], the upper
confidence bound (UCB) [29] and etc. Taking UCB as an
example, the calculation process is shown as follows:
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
8
(21)
where is the confidence interval parameter and is the
profit of CTBC when taking as an input. Then the
maximum and optimal can be obtained by
interactively calling GPR and AF.
IV. APPLICATION RESULTS AND ANALYSIS
The proposed strategy is applied to a TJP system to
validate its effectiveness. TJP is a L3 ADS, which is used to
help the subject vehicle to follow the target vehicle in traffic
jams, where the vehicles speed of surrounding traffic flow is
(0,55] km/h. Its influence factors and their corresponding
values are acquired through the analysis process shown in Fig.
2. Then the importance indexes of different values are
calculated by (3)-(6). Finally, all these factors, values and
indexes will be used as the input of CTBC algorithm, as
shown in Table 4. Here the influence factor “Weather” is
taken as an example to illustrate the calculation process of
importance index. The factor “Weather” is denoted by
and its values are “Sunny, Cloudy, Rainy, Foggy” denoted by
. Then, a judgement matrix
can be constructed to describe their relative contribution by
the AHP method [9]:
(22)
According to the AHP method [9], the relative importance
indices can be represented by the following eigenvector
corresponding to the maximum eigenvalue of :
(23)
Being similar to the aforementioned process, the relative
importance indices of influence factors “Weather”,
“Lightning environment” and “Environment” are 0.5396,
0.1634 and 0.3333, respectively. Then the importance indices
to the root of values “Sunny, Cloudy, Rainy, Foggy” are
obtained by (6) as
.
A. Parameter optimization process
First of all, we need to find the optimal complexity
improvement index of the proposed CTBC algorithm, so as
to generate the set of test cases with the best test effect. The
parameters of the automatic test scenarios generation process
are set to be , , and , and
the optimization process is shown in Fig. 6.
(a) GPR model constructed with 9 sampling points
(b) Values calculated by different AF with 9 sampling points
Fig. 6. Bayesian optimization of test effect
In Fig. 6(a), the Bayesian optimization algorithm predicts
the test effect and its variance, which are marked by “Means”
and “Variances” respectively. In order to verify its
effectiveness, the actual values of with different is
also calculated. And 0.5540 is found when
= 0.04 . The Bayesian optimization algorithm only
needs 9 sampling points to find the optimal solution. From
Fig. 6(b), it is found that when the number of sampling points
is 9, the maximum value calculated by three different AF all
appear at . That means the same optimization
results can be obtained by using either method in the specific
application object of this paper and UCB is selected as the
AF in this paper.
We can further get that the number of the test cases is
590 and the overall complexity is 0.4137.
Then, the number of clusters can be roughly determined by
an engineering method [26]:
(24)
where indicates the rounding operator and the correlation
coefficient is set to be 1/25. And finally, the 590 test
cases are clustered into 24 test scenarios.
B. Test effectiveness vs. Complexity index
The proposed strategy is based on the assumption that the
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
9
IDS are more prone to failure in more complex scenarios.
Therefore, to verify this hypothesis, the test results under
different scenarios are further evaluated by the test
effectiveness, which is measured by the number of the failed
functional logic indices. The functional logic index is the
expected response of TJP and reflected by the signals listed
in Table 1.
Table 1. Signals used to evaluate the functional logic indices of TJP
Signals
Unit
Longitudinal speed
km/h
Lateral speed
km/h
Longitudinal acceleration
m/s2
Longitudinal deceleration
m/s2
Deviation from the lane center
m
Engine torque
N·m
Target vehicle recognition
-
Front of the front vehicle recognition
-
Left vehicle recognition
-
Right vehicle recognition
-
Left lane line recognition
-
Right lane line recognition
-
Following distance
m
Relative speed
km/h
Time gap
s
Acceleration request
m/s2
Deceleration request
m/s2
Angular velocity of steering wheel
deg/s
Status display of HMI
-
State machine state of TJP system
-
Enable signal state of TJP system
-
Unavailable signal state of TJP system
-
There are totally 16 function logic indices, and for the sake
of brevity and readability, the index for distance control logic
is shown as an example:
(25)
where and are the following distance and relative
vehicle speed respectively, and is the time gap. The
proposed function logic indices are used to measure the
system behavior of TJP, which are independent of the
specific vehicle to be equipped with TJP. Moreover, the
sensor model and vehicle dynamical model in the simulation
platform shown by Fig. 3 have been calibrated according to
the real developed vehicle to ensure consistency of behavior
between the simulation and the real one.
The generated 590 test cases are arranged in order of the
complexity from small to large and divided into 10 parts
uniformly according to complexity. The relationship between
the average complexity of the test cases and the average
number of detected faults is shown in Fig. 7. The complexity
index increases from 0.1071 to 0.4484 uniformly. In general,
the average number of detected faults also raises from 0 to 9,
which has a positive correlation with the complexity of cases.
This result shows that TJP tends to degrade under more
complex cases. It is beneficial to increase the fault detection
rate by improving the complexity of designed scenarios.
Fig. 7. Relationship between complexity and effectiveness
C. Comparative analysis of scenario complexity
As shown in the previous section, the test effectiveness can
be measured by the complexity index indirectly. In this
section, the improvement effect of CTBC algorithm on the
complexity of test scenarios is further analyzed by comparing
it with TM and CT methods. For TM, two widely used
methods are selected, that is, the ISO standards and ET
method. Because TJP is still in the development stage and
there is no standard, the ISO standard of the low speed
following system [30] is used as an alternative. Since it only
contains 5 test scenarios and such key factors as the rapid
change of light etc. are not taken into account, the importance
indices of factors included in Table 4 but not considered by
ISO are also added when calculating the complexity index of
the test scenario. As for CT, because to the best of our
knowledge, there is no other algorithm that considers the
number of test cases and the overall complexity index at the
same time, therefore, three common-used algorithms are
selected here for a more detailed comparison because of their
universality, open source and free characteristics, namely the
PICT [31] developed by Microsoft Corporation, the AETG
[32] developed by IDA Center for Computing Sciences, and
the AllPairs [33] developed by Satisfice, Inc. The strength of
combinatorial coverage for all these CT algorithms including
CTBC is set to . The quartile map of complexity is
shown in Fig. 8 comparatively.
Fig. 8. Complexity distribution of different methods
As shown in Fig. 8, the scenarios designed in ISO standard
are quite simple and can only validate the basic functions. As
for ET, its complexity distributes uniformly between 0.0717
and 0.4484. Although the overall complexity is much higher
than that in ISO, the test efficiency is still low because the
cases with different complexity are given the same attention.
Moreover, compared with ET, the complexity distributions of
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
10
the other CT algorithms have not been promoted obviously:
only the minimum and lower quartile values are improved,
the mean and median are basically unchanged, and the upper
quartile and maximum values are even decreased. Therefore,
the results show that there is no significant improvement in
complexity by using traditional CT methods directly. On the
contrary, obvious overall complexity improvements have
been achieved by utilizing CTBC algorithm comparing with
other TM and CT methods. For example, compared with ET,
the minimum value (regardless of outliers), the lower quartile,
the median, the upper quartile and the mean value are
increased by 5.7015, 2.7070, 1.7298, 1.2817 and 1.6221
times respectively. What else needs to be explained is that the
reason why there are so many outliers in the figure is that
when using CTBC, the ones with relatively small complexity
indices account for a quite small proportion, and there is a big
gap between these smaller indices and those of the rest of all
the cases.
D. Test results of TJP
To show the reduction of time by automation, 5 scenarios
are selected randomly to be conducted with the same
commercial tools manually by an experienced engineer. The
average memory/CPU consumption and the time cost in the
host computer are compared to observe to differences
between the proposed method and the conventional one, as
shown in Table 2.
Table 2. Comparison of resource consumption between the conventional
testing method and the automation testing method
Average resource
consumption
Conventional
testing method
Automation
testing method
memory occupation
25%
61%
CPU occupation
29%
73%
Test consumption time
(single test scenario) (min)
234
6
Test consumption time
(complete test process) (day)
63
2.5
As shown in the table, comparing with the conventional
manual method, the average occupancy ratio of memory and
CPU in the process of test and evaluation by the automation
testing method is much higher, which enable the computer to
more fully utilize its performance. In addition, the average
consumption time for a single test scenario is about 234
minutes. Most of the time is spent on designing and building
the 3D test scenarios manually, and processing the test results
artificially. The proposed method can reduce the average
execution time of each scenario to about 6 minutes by
automating all the above processes. In a complete typical test
evaluation process, the number of required test scenarios is
about 390 mainly including the typical conditions such as the
scenarios defined in the test standards, the invalidation
scenarios found in the road test and etc. The total time
consumption in a typical round will be 63 days, while the
time consumed by using the automation testing method is
only 2.5 days. Therefore, we can see that the proposed
method can significantly increase the test efficiency through
more efficient resource calling method and scenario
generation method, and finally achieve cost control in labor
and time. It can help to meet the requirement of the vehicle
development cycle and support the algorithm design of TJP
effectively.
Finally, some typical faults of TJP during the first round of
testing are shown in Table 3 to give an example of the causes
of the system failures. Although it is analyzed by the
engineers after getting the test report with evaluation results
manually, and cannot be regarded as a part of the “automatic”
evaluation method, it is an important step of a complete and
closed-loop test process.
Table 3. Some typical faults of TJP
No.
1
Object vehicle can’t be identified in the heavy fog.
2
When objet vehicle cuts in from left lane with
deceleration -2m/s2, there’s a collision.
3
Object vehicle cannot be identified when entering tunnel
in sunny days.
4
Under condition that subject vehicle runs at 15km/h and
object vehicle cut out to adjacent right lane, subject
vehicle can’t follow the new object vehicle.
5
Object vehicle on road with radius 125 m can’t be
identified.
…
…
V. CONCLUSION
This paper proposes an automatic virtual test strategy for
IDS to improve test efficiency. A new combinational testing
algorithm of test cases generation is designed to take both the
number of test cases and the test effectiveness into
consideration by introducing the complexity index of test
scenario. And then, based on the commercial tools, a joint
simulation platform is established to realize the automation
of test and evaluation process for better efficiency. The
application results show that:
1) In general, the larger the complexity of test scenario is,
the easier it is to find out the malfunctions of IDS. The
proposed complexity index can be used to measure the ability
of the test scenario to find system faults.
2) The new algorithm can generate test cases with higher
complexity comparing with other CT and TM methods, while
the coverage performance remains unchanged.
3) The developed joint simulation platform can realize the
automatic test and evaluation of IDS, which can reduce the
time consumption of the complete test process greatly.
APPENDIX
Table 4. Influence factors and their corresponding values with importance indices of TJP
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
11
Influence factor
Values
Importance index
TJP
system
Environ-
ment
Lightning
environment
Weather
Sunny
Cloudy
Rainy
Foggy
0.0012
0.0074
0.0074
0.0134
Time
Day (8:00-17:00)
Dusk/Dawn (17:00-19:30/5:30-8:00)
Night (19:30-5:30)
0.0011
0.0053
0.0098
Rapid changes
in light
Pass through tunnel
Pass under footbridge
No change
0.0030
0.0051
0.0007
Lane lines
parameters
Lane line clarity
Few fade and holes
Intermediate fade and holes
Much fade and holes
No fade and holes
0.0028
0.0108
0.0319
0.0011
Lane line
integrity
Lane line on one side
Lane line on both sides
0.0029
0.0204
Lane line
number
Single
double
0.0044
0.0015
Lane line color
White
Yellow
0.0020
0.0096
Lane line type
Dashed
Solid
0.0105
0.0011
Road
parameters
Curvature
Straight road (0)
Bend road (1/750)
Bend road (1/305)
Bend road (1/125)
0.0061
0.0130
0.0290
0.0626
Slope
Uphill (Slope of +5%)
Downhill (Slope of -5%)
No slope
0.0292
0.0186
0.0048
Roadside
facilities
One facility
Combination of two facilities
Combination of three facilities
Combination of four facilities
Combination of five facilities
No facility
0.0009
0.0015
0.0025
0.0039
0.0074
0.0005
Driving
task
Driving
capacity
Location of
HV*
Left lane
Right lane
Middle lane
0.0064
0.0191
0.0064
Longitudinal
speed of HV***
5 km/h
10 km/h
15 km/h
30 km/h
35 km/h
50 km/h
55 km/h
0.0079
0.0221
0.0221
0.0329
0.0427
0.0548
0.0079
Object detection
Left lane RV*
lateral behavior
Cuts in/out to adjacent right lane
Takes no action
0.0202
0.0040
Left lane RV
longitudinal
behavior
VLRV*=VHV*, aLRV*=0m/s2
VLRV=VHV, aLRV=2m/s2
VLRV=VHV, aLRV=-2m/s2
VLRV>VHV, aLRV=0m/s2
VLRV>VHV, aLRV=2m/s2
VLRV>VHV, aLRV=-2m/s2
VLRV<VHV, aLRV=0m/s2
VLRV<VHV, aLRV=2m/s2
VLRV<VHV, aLRV=-2m/s2
0.0019
0.0061
0.0061
0.0031
0.0061
0.0061
0.0031
0.0061
0.0061
Middle lane RV
lateral behavior
Cuts in/out to adjacent left lane
Cuts in/out to adjacent right lane
Takes no action
0.0348
0.0348
0.0070
Middle lane RV
longitudinal
behavior
VMRV*=VHV, aMRV*=0m/s2
VMRV=VHV, aMRV=2m/s2
VMRV=VHV, aMRV=-2m/s2
VMRV>VHV, aMRV=0m/s2
VMRV>VHV, aMRV=2m/s2
VMRV>VHV, aMRV=-2m/s2
VMRV<VHV, aMRV=0m/s2
VMRV<VHV, aMRV=2m/s2
VMRV<VHV, aMRV=-2m/s2
0.0033
0.0105
0.0105
0.0054
0.0105
0.0105
0.0054
0.0105
0.0105
Right lane RV
lateral behavior
Cuts in/out to adjacent left lane
Takes no action
0.0202
0.0040
Right lane RV
longitudinal
behavior
VRRV*=VHV, aRRV*=0m/s2
VRRV=VHV, aRRV=2m/s2
VRRV=VHV, aRRV=-2m/s2
VRRV>VHV, aRRV=0m/s2
VRRV>VHV, aRRV=2m/s2
0.0020
0.0063
0.0063
0.0033
0.0063
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
12
VRRV>VHV, aRRV=-2m/s2
VRRV<VHV, aRRV=0m/s2
VRRV<VHV, aRRV=2m/s2
VRRV<VHV, aRRV=-2m/s2
0.0063
0.0033
0.0063
0.0063
Distance
between HV
and target
RV***
0.5 desired distance**
1 desired distance
2 desired distance
5 desired distance
0.0721
0.0435
0.0233
0.0122
* HV: Host Vehicle; RV: Remote Vehicle; VHV: Speed of HV; VLRV, VMRV, VRRV: Speed of left RV, middle RV and right RV; aLRV, aMRV, aRRV: Acceleration
of left RV, middle RV and right RV;
** “Desired distance” is the safety car-following distance in the technical manual, which is nonlinear positive relation with the longitudinal speed of HV.
*** “Longitudinal speed of HV” and “Distance between HV and target RV” represent the initial speed and distance of HV at the starting time of the evaluation
process respectively.
REFERENCES
[1]. K. Li, S. Li, F. Gao and et al., “Robust distributed consensus control
of uncertain multi-agents interacted by eigenvalue-bounded
topologies,” IEEE Internet of Things Journal. vol. 7, no. 5, pp. 3790-
3798, 2020
[2]. S. E. Li, F. Gao, D. Cao and et al., “Multiple-model switching control
of vehicle longitudinal dynamics for platoon level automation,” IEEE
Transactions on Vehicular Technology. vol. 65, no. 6, pp. 4480-4492,
2016
[3]. C. Lv, D. Cao, Y. Zhao and et al., “Analysis of autopilot
disengagements occurring during autonomous vehicle testing,”
IEEE/CAA Journal of Automatica Sinica. vol. 5, no. 1, pp. 58-68, 2018
[4]. J. Duan, F. Gao and Y. He, “Test scenario generation and optimization
technology for intelligent driving systems,” IEEE Intelligent
Transportation Systems Magazine, available online, doi:
10.1109/MITS.2019.2926269, 2020
[5]. L. Fridman, D. E. Brwon, M. Glazer and et al., “MIT advanced vehicle
technology study: large-scale naturalistic driving study of driver
behavior and interaction with automation,” IEEE Access. vol. 7, pp.
102021–102038, 2019
[6]. P. Wu, F. Gao and K. Li, “A vehicle type dependent car-following
model based on naturalistic driving study,” Electronics, vol. 8, no. 4,
pp. 453-468, 2019
[7]. L. Zhu, J. Gonder, E. Bjarkvik and et al., “An automated vehicle fuel
economy benefits evaluation framework using real-word travel and
traffic data,” IEEE Intelligent Transportation Systems Magazine, vol.
11, no. 3, pp. 29-41, 2019
[8]. S. Masuda, H. Nakamura, K. Kajitani, “Rule-based searching for
collision test cases of autonomous vehicle simulation,” IET Intelligent
Transport Systems, vol. 12, no. 9, pp. 1088-1095, 2019
[9]. Q. Xia, J. Duan, F. Gao and et al., “Test scenario design for intelligent
driving system ensuring coverage and effectiveness,” International
Journal of Automotive Technology, vol. 19, no. 4, pp. 751-758, 2018
[10]. U. Chipengo, “Full physics simulation study of guardrail radar-returns
for 77GHz automomtive radar systems,” IEEE Access, vol. 6, pp.
70053-70060, 2018
[11]. M. Zulkefli, P. Mukherjee, Z. Sun and et al., “Hardware-in-the-loop
testbed for evaluating connected vehicle applications,” Transportaion
Research Part C: Emerging Technologies, vol. 78, pp. 50–62, 2017
[12]. J. Zhou, R. Schmied, A. Sandalek and et al., “A framework for virtual
testing of ADAS,” SAE International Journal of Passenger Cars-
Electronic and Electrical Systems. vol. 9, no. 1, pp. 66–73, 2016
[13]. L. Li, W. Huang, Y. Liu and et al., “Intelligence testing for
autonomous vehicles: a new approach,” IEEE Transactions on
Intelligent Vehicles, vol. 1, no. 2, pp. 158–166, 2016
[14]. M. Brannstrom, E. Coelingh and J. Sjoberg, “Model-based threat
assessment for avoiding arbitrary vehicle collisions,” IEEE
Transactions on Intelligent Transportation Systems, vol. 11, no. 3, pp.
658-669, 2010
[15]. T. Winkle, C. Erbsmehl and K. Bengler, “Area-wide real-world test
scenarios of poor visibility for safe development of automated
vehicles,” European Transport Research Review, vol. 10, no. 2, pp.
32, 2018
[16]. H. Yang and H. Peng, “Development and evaluation of collision
warning/collision avoidance algorithms using an errable driver model,”
Vehicle System Dynamics, vol. 48, sup1., pp. 525–535, 2010
[17]. W. Wang and D. Zhao, “Evaluation of lane departure correction
systems using a regenerative stochastic driver model,” IEEE
Transactions on Intelligent Vehicles, vol. 2, no. 3, pp. 221-232, 2017
[18]. Z. Huang, D. Zhao, H. Lam and et al., “Accelerated evaluation of
automated vehicles using piecewise mixture models,” arXiv:
1610.09450, 2017
[19]. D. Zhao, H. Lam, H. Peng and et al., “Accelerated evaluation of
automated vehicles safety in lane-change scenarios based on
importance sampling techniques,” IEEE Transactions on Intelligent
Transportation Systems, vol. 18, no. 3, pp. 595-607, 2016
[20]. ISO Standard 17361, “Intelligent transport systems – lane departure
warning systems – performance requirements and test procedures,”
ISO, 2017
[21]. D. I. Katzourakis, N. Lazic, C. Olsson and et al., “Driver steering
override for lane-keeping aid using computer-aided engineering,”
IEEE/ASME Transactions on Mechatronics, vol. 20, no. 4, pp. 1543-
1552, 2015
[22]. N. Pande, S. Kumar, L. R. Everson and et al., “Understanding the key
parameter dependences influencing the soft-error susceptibility of
standard combinational logic,” IEEE Transactions on Nuclear
Science, vol. 67, no. 1, pp. 116-125, 2020
[23]. S. Sangeeta and A. Manuj, “A novel approach for deriving
interactions for combinatorial testing,” Engineering Science and
Technology, vol. 20, no. 1, pp. 59–71, 2017
[24]. D. Wallace and D. Kuhn, “Failure modes in medical device software:
an analysis of 15 years of recall data,” International Journal of
Reliability Quality and Safety Engineering, vol. 81, no. 4, pp. 351–
371, 2002
[25]. H. Wu, C. Nie, F. Kuo, and et al., “A discrete particle swarm
optimization for covering array generation,” IEEE Transactions on
Evolution Computation, vol. 19, no. 4, pp. 575–591, 2015
[26]. S. Zhou, Z. Xu and F. Liu, “Method for determining the optimal
number of clusters based on agglomerative hierarchical clustering,”
IEEE Transactions on Neural Networks and Learning Systems, vol.
28, no. 12, pp. 3007-3017, 2017
[27]. R. Schofield, A. Chircop, C. Baker and et al., “Entry-to-practice
public health nursing competencies: a delphi method and knowledge
translation strategy,” Nurse Education Today, vol. 65, pp. 102−107,
2018
[28]. R. Tamura and K. Hukushima, “Bayesian optimization for
computationally extensive probability distributions,” Plos One, vol.
13, no. 3, e0193785, 2018
[29]. E. Schulz, M. Speekenbrink and A. Krause, “A tutorial on Gaussian
process regression: modelling, exploring, and exploiting functions,”
Journal of Mathematical Psychology, vol. 85, pp. 1−16, 2018
[30]. ISO Standard 22178, “Intelligent transport systems – Low speed
following (LSF) systems – performance requirements and test
procedures,” ISO, 2009
[31]. J. Czerwonka, “Pairwise testing in real world,” In Proceedings of the
24th Pacific Northwest Software Quality Conference, Portland, pp.
419–430, 2006
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
0018-9545 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TVT.2020.3033565, IEEE
Transactions on Vehicular Technology
13
[32]. D. Cohen, S. Dalal, M. Fredman and et al., “The AETG system: an
approach to testing based on combinatorial design,” IEEE
Transactions on Software Engineering, vol. 23, no. 7, pp. 437–444,
1997
[33]. Satisfice, Inc., “Epistemology for the rest of us,” Available:
http://www.satisfice.com/tools.shtml
[34]. R. Calandra, A. Seyfarth, J. Peters and et al., “Bayesian optimization
for learning gaits under uncertainty,” Annals of Mathematics &
Artificial Intelligence, vol. 76, no. 1, pp. 5–23, 2016
[35]. B. Shahriari , K. Swersky , Z. Wang and et al., “Taking the human out
of the loop: a review of Bayesian optimization,” Proceedings of the
IEEE, vol. 104, no. 1, pp. 148–175, 2015
[36]. A. Geiger, P. Lenz and C. Stiller, “Vision meets robotics: The KITTI
dataset,” International Journal of Robotics Research, vol. 32, no. 11,
pp. 1231–1237, 2013
[37]. G. Pandey, J. R. Mcbride, R. M. Eustice, “Ford Campus vision and
lidar data set,” International Journal of Robotics Research, vol. 30,
no. 13, pp. 1543–1552, 2014
Feng Gao received M.S. and Ph.D. in Tsinghua
University in 2003 and 2007, respectively. From
2007 to 2013, he worked as a senior engineer in
Changan Auto Global R&D Centre, where he has
led several projects involving electromagnetic
compatibility, durability test of electronic module,
ADAS and engine control. He is now a professor in
School of Automotive Engineering, Chongqing
University. His current research interests include
robust control and optimization approach with
application to automatic driving systems. He is the author of more than 100
peer-reviewed journal and conference papers, and co-inventor of over 20
patents in China. Prof. Gao was the recipient of Best Award of Automatic
Driving Technology of International Intelligent Industry Expo. (2018),
Technical Progress Award of Automotive Industry (2017, 2018, 2020) and
Technical Progress Award of Chongqing (2019).
Jianli Duan received the B.S. degree in electrical
engineering and automation from North China
Electric Power University in 2013, and the Ph.D.
degree in electrical engineering at Chongqing
University in 2020. He is currently doing
postdoctoral research in Tsinghua University. His
research interest includes the design of testing
scenarios of intelligent traffic systems, automated
testing method research and implementation,
hardware-in-the-loop and software-in-the-loop
testing technology.
ZaiDao Han received the B.S. degree of Automation
from Chongqing University in 2019, and he is
currently pursuing the Master degree in the School of
Automotive Engineering, Chongqing University.
His research interests include swarm intelligence and
application to test and evaluation of automatic driving
systems.
Yingdong He received the B.S. degree in Mechanical
Engineering from Beijing Institute of Technology in
2016, and is currently pursuing the Master degree in
the Mechanical Engineering Department, University
of Michigan.
His research interests include driving decision and
vehicle dynamics control. He is the author of more
than 6 peer-reviewed journal and conference papers
Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on October 28,2020 at 00:42:03 UTC from IEEE Xplore. Restrictions apply.
