VANETs’ research over the past decade: Overview, credibility, and trends

Abstract

Since its inception, Vehicular Ad hoc Networks (VANETs) have been attracting much attention from both academia and industry. As for other wireless networking areas, scientific advancements are mainly due to the employment of simulation tools and mathematical models. After surveying 283 papers published in the last decade on vehicular networking, we pinpoint the main studied topics as well as the most employed tools, pointing out the changes in research subject preference over the years. As a key contribution, we also evaluate to what extent the research community has evolved concerning the principles of credibility in simulation-based studies, such as repeatability and replicability, comparing our results with previous studies.

Full text

VANETs’ research over the past decade: overview, credibility,
and trends
Elmano Ramalho Cavalcanti
Federal Institute of Pernambuco,
Brazil
elmano.cavalcanti@garanhuns.ifpe.
edu.br
José Anderson Rodrigues de
Souza
Federal University of Campina
Grande, Brazil
joseanderson@copin.ufcg.edu.br
Marco Aurélio Spohn
Federal University of Fronteira Sul,
Brazil
marco.spohn@us.edu.br
Reinaldo Cézar de Morais
Gomes
Federal University of Campina
Grande, Brazil
reinaldo@computacao.ufcg.edu.br
Anderson Fabiano Batista
Ferreira da Costa
Federal Institute of Paraiba, Brazil
anderson@ifpb.edu.br
This article is an editorial note submitted to CCR. It has NOT been peer reviewed.
The authors take full responsibility for this article’s technical content. Comments can be posted through CCR Online.
ABSTRACT
Since its inception, Vehicular Ad hoc Networks (VANETs) have
been attracting much attention from both academia and industry.
As for other wireless networking areas, scientic advancements are
mainly due to the employment of simulation tools and mathemati-
cal models. After surveying 283 papers published in the last decade
on vehicular networking, we pinpoint the main studied topics as
well as the most employed tools, pointing out the changes in re-
search subject preference over the years. As a key contribution, we
also evaluate to what extent the research community has evolved
concerning the principles of credibility in simulation-based studies,
such as repeatability and replicability, comparing our results with
previous studies.
CCS CONCEPTS
•General and reference →Surveys and overviews
;
•Networks
→Mobile ad hoc networks;
KEYWORDS
Survey, Vehicular networks, Simulation, Reproducibility
1 INTRODUCTION
Vehicular Ad hoc Networks (VANETs) are one of the most promi-
nent specialization of Mobile Ad hoc Networks (MANETs). In
VANETs, vehicles communicate with each other (vehicle-to-vehicle,
V2V) or with the infrastructure (vehicle-to-infrastructure, V2I). Cur-
rently, the term vehicle-to-everything (V2X) has also been widely
adopted.
Since its inception, vehicular networking has been attracting
much attention from both academia and industry. Several interna-
tional industrial and governmental consortia such as the European
C2C-CC
1
and the American VII
2
have shown the high relevance
and feasibility of V2V and V2I communication for our society.
Figure 1 depicts the total number of publications related to
VANETs and MANETs during the last years. We have not plotted
2017 stats since the number of papers that appeared in conferences
and journals in the last quarter of the year is not fully available from
indexing databases. For VANETs, there is a noticeable increase in
the number of published papers, while a reduction is observed for
MANETs’ publications. Particularly, from 2012 to 2016 the former
increased by 50% while the latter decreased by 18%.
Figure 1: Number of MANET and VANET papers indexed in
Google Scholar from 2006 to 2016.
To support the growth of wireless ad hoc networks, researchers
have designed a variety of protocols, spanning the main layers of
the protocol stack. When it comes to evaluating such protocols,
1CAR 2 CAR Communication Consortium, https://www.car-2-car.org
2Vehicle Infrastructure Integration, http://www.vehicle-infrastructure.org
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

analytic modeling, experimentation, and simulation are the three
main approaches available to researchers. The rst approach usually
lacks generalization, by not taking into account the intrinsic high
complexity level. The second one provides realistic achievements
such as proof of concept, but for large scale scenarios it is nancially
unfeasible. Therefore, due to its better cost benet, simulation is by
far the leading approach employed by researchers [1].
However, one of the major issues concerning simulation results
from its frequently low credibility level. There are likely errors
due to the simulation model or from improper data analysis [
3
].
Unfortunately, there is a widespread bad practice by not describing
or omitting information, source code, and datasets, which are crucial
for guaranteeing replicability and reproducibility of any published
research. Moreover, it is quite common to present statistical results
with no information regarding their accuracy in terms of condence
interval, at any given condence level, including relative statistical
errors [27].
In this paper, we analyze the current state of VANETs’ research by
taking into account hundreds of papers published from 2007 to 2016.
The goals of this survey are two-fold: (1) to check to what extent
the research community has evolved concerning the principles of
credibility in simulation-based studies [
17
]; (2) to gure out the
main research topics investigated as well as the most employed
tools (e.g., simulators), models (e.g., mobility, trac, propagation),
and protocols, pointing out several research trends.
This paper is organized as follows: Section 2 describes the related
work, Section 3 shows the results of the survey, followed by a critical
analysis of the VANET literature (Section 4), and Section 5 concludes
the paper.
2 RELATED WORK
The rst eort on the evaluation of simulation-based research in
computer networks was carried out by Pawlikowski et al. [
25
]. The
authors conducted a survey of over 2
,
200 publications related to
telecommunications from IEEE INFOCOM (1992 to 1998), and three
other computer networking journals, from 1996 to 1998. The survey
took into account only two aspects of simulation credibility: the use
of appropriate pseudo-random number generators (PRNGs), and the
proper analysis of simulation output data. The results revealed that
almost 50 percent of all papers did not state the type of simulation
employed (i.e., terminating or steady-state), while more than 70
percent did not mention which PRNG was used.
Kotz et al. [16] compared experimentation with simulation of a
MANET in order to measure how the available radio propagation
models dier from real measurements. The results showed that
the usual assumptions considered by the research community (e.g.,
a radio’s transmission area is circular), and adopted by simpler
models (e.g., free-space and two-ray ground), are unrealistic. The
authors also surveyed a set of MobiCom and MobiHoc articles from
1995 through 2002, showing that about 80% of all papers adopted
unrealistic radio models.
In what concerns credibility in MANET simulation-based studies,
Kurkowski et al. [
17
] surveyed 151 papers presented at the Interna-
tional Symposium on Mobile Ad Hoc Networking and Computing
(MobiHoc) from 2000 to 2005. They focused on measuring how
credible the simulation-based papers are, considering four areas of
credibility in research: repeatability, rigorous scenario modeling,
unbiased results, and proper statistical analysis. General results
pointed out that less than 15% of the published papers were found
repeatable, only 7.0% addressed initialization bias, and 87.5% did
not include condence intervals in the plots. Later on, we compare
their results to ours.
Andel and Yasinsac [
3
] also discuss the issue of credibility in
MANETs’ simulation, listing the most common improper simulation
practices such as the absence of repeatable simulation requirements,
and questionable statistical validity. The former is mainly due to
the lack of documentation, whereas the latter is mainly related
to inaccurate data collection techniques, or insucient statistical
analysis. In addition to that, another crucial aspect is regarding
to the level of details and the model validation employed. As the
authors claim, “Omitting detail or oversimplifying the model can
lead to ambiguous or erroneous outcomes”.
A more recent study [
27
] presents a comprehensive survey of
over 8,300 IEEE publications on telecommunication networks. The
authors concluded that, even after ten years of the Pawlikowski et
al. study [
25
], “there is no signicant change with respect to quality
and credibility of the simulation studies revised and the deep crisis
of credibility still remains”. Even though a large number of papers
were surveyed, only papers published from 2007 to 2009 were taken
into account.
Dierently of the previous works, in this paper we consider a
longer period, from 2007 to 2016, and focus not only on the credi-
bility of simulation-based studies but also on the overall VANETs’
research eld. To the best of our knowledge, this is the rst work
to present such kind of review and contribution.
3 SURVEY RESULTS
To conduct the survey we have considered papers containing the
words “VANET” or “Vehicular ad hoc Network” in the title, abstract,
or index terms (e.g., keywords) on publications from the following
conferences and journals for the period from 2007 to 2017:
•
The ACM International Symposium on Mobile Ad Hoc Net-
working and Computing (MobiHoc);
•
The Annual International Conference on Mobile Computing
and Networking (Mobicom) main event and workshops;
•Vehicular Technology Conference (VTC);
•IEEE Transactions on Mobile Computing (TMC).
Mobihoc and Mobicom are well-known highly selective premier
conferences on computer networking, whereas TMC is one of the
leading journals on mobile computing. In addition to that, VTC was
also chosen due to its relevance and the high number of available
papers covering overall VANETs’ topics (indeed, the conference
takes place twice a year).
A total of 283 papers matched this particular selection, of which
147 were published during the rst ve years (2007-2011), and the
other 136 were published during the second period (2012-2017).
We manually inspected the 283 papers, extracting all valuable
information required to characterize the research, and to highlight
how the authors performed their research. The questionnaire em-
ployed for gathering all required information, as well as the data
collected, are available at zenodo 3.
3https://doi.org/10.5281/zenodo.1205633
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

Table 1: Top-level research categories in VANETs.
CATEGORIES
Title Description Topic examples
MAC-PHY MAC and Physical layers issues
Media Access Control Techniques, schemes protocols,
or algorithms; Channel modeling, modulating and coding;
Adaptive transmit power control
PERF Performance comparison analysis Protocol performance/comparison analysis and simulation;
Protocol design, testing, and verication analysis
APP Application layer protocols and services
Safety (e.g., crash prevention); Eciency (e.g., road congestion
avoidance); Entertainment (e.g., multimedia streaming);
Environmental (e.g., pollution detection)
DATA Data management Data collection and message dissemination methods
(e.g., broadcasting)
SERV Complementary services
Location Tracking, Location Estimate Correction, Integration
with Infrastructure Networks, Quality-of-Service Issues,
Security issues and countermeasures
ROUT Routing protocol Proposal of a new routing procotol
MOB Mobility issues Mobility / Connectivity analysis, modelling, management,
Clustering Algorithm
TOOL Tools, testbeds
Tool / Platform / Architecture / Framework proposal
Deployment and eld testing, Experimental and
Prototype Results
3.1 Overview of VANET research in the last
decade
Starting from the list of the most common research topics in VANETs
(extracted and merged from several ‘call for papers’ of computer
networking conferences and journals), we dened a group of eight
research areas (top-level categories), covering a wide range of top-
ics related to vehicular communications (Table 1). For example, if a
paper focuses on any aspect related to the link or physical layers,
such as a MAC algorithm, channel modeling, network coding, or
adaptive transmit power control, the paper is labeled as MAC-PHY.
At the other end there is the APP class, containing all papers ad-
dressing user applications (e.g., intersection collision avoidance,
road congestion notication, and multimedia streaming).
As in other computer networking areas, there are many papers
concerning performance evaluation of protocols. A paper is taken
as belonging to the PERF class if and only if its main contribution
is on the performance evaluation itself. Therefore, if a new MAC
or routing protocol was proposed, followed by its performance
evaluation, then the paper was labeled as MAC-PHY, or ROUT,
instead of PERF.
Papers describing a new tool, platform, framework, or architec-
ture were labeled as TOOL. For the sake of simplicity, all experi-
mental studies, usually concerning deployment and eld testing,
were also included into that class.
The ROUT class stands for papers proposing a new routing
protocol whereas the MOB class relates to mobility issues, such as
mobility modeling and clustering algorithms.
We end our classication with the DATA and SERV labels. The
former was used for works focusing on data collection/dissemination
schemes, while the latter is for what we call ‘complementary ser-
vices’, such as Quality of Service (QoS), security, and localization.
Figure 2 depicts the content and temporal distribution for the
published papers. Almost a quarter of them was on MAC-PHY is-
sues (23%). During the last decade, both academia and industry have
worked in order to enable eective and ecient wireless communi-
cation in vehicular environments. The IEEE 802.11p amendment as
well as the IEEE 1609 Wireless Access in Vehicular Environments
(WAVE) standards result from such an eort. Moreover, our results
suggest a slight increase on such issues. The applicability of net-
work coding schemes in VANETs may be considered as one major
reason for this trend.4
At the second place, we got the SERV class, which includes pa-
pers with regard to complementary services, whether it be Quality
of Service (QoS), security approaches, or location-based services. In
the second period, we noticed more studies associated to the inte-
gration between vehicular networks and traditional infrastructure
networks (e.g., cellular). There is an expectation for VANETs to be
integrated with 5G mobile technology by 2020 [23].
Routing (ROUT label) is still an important topic of research.
Firstly, traditional MANET routing protocols were found not suit-
able for the vast requirements, and unique characteristics of VANETs’
scenarios and applications. Tens of routing protocols have been
proposed during the last decade, including various taxonomies for
classifying them [
6
]. Rather than the traditional MANETs’ routing
protocols, which are topology-based (proactive, reactive, or hy-
brid), the most promising VANETs’ alternatives are the geographic
and the delay-tolerant approaches. The former is more suitable for
4
A search for the expression (VANET “NETWORK CODING”) in Google scholar results
in 478 papers in 2007-2011, and 1570 in 2012-2016.
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

Figure 2: The main research topics in VANETs.
vehicular communication whereas the latter involves the carry-
and-forward strategy which is employed to overcome intermittent
connectivity, a typical situation in vehicular scenarios.
We noticed a signicant increase on the number of works related
to dissemination strategies and broadcasting algorithms (DATA
label). In fact, it was the one subject with the highest growth among
all (38%). As noted by Lee et al. [
18
], most application protocols rely
on variants of the epidemic data dissemination approach so that
the appropriate information propagates to nodes in an area where
the information is originated. The main challenge in this area is on
addressing redundancy and eciency all together.
Meanwhile, publications in the PERF class dropped from 20 to
only 8 papers between the two periods. This result suggests that
less eort has been put just on protocol performance comparison
and analysis. As the research in VANETs evolves, more attention
is given to novel protocols, services, and applications instead of
understanding how existing technologies would behave in new en-
vironments. Thus, one could take this result somehow predictable.
Table 2: VANET simulation freeware / open source software.
Category Tool Release1
Integrated framework
TraNS [26] 2007
MobiREAL [19] 2006
Veins [28] 2006
Network Simulator
NCTUns [31] 2007
ns-3 [14] 2008
OMNET++ [30] 2006
GrooveNet [20] 2006
Mobility Generator
SUMO [4] 2006
VanetMobiSim [13] 2006
MOVE [15] 2007
CityMob [21] 2008
Scenario Generator VERGILIUS [10] 2010
1Or when it rst supported vehicular network simulation.
The class TOOLS was also less addressed, with publications
dropping from 22 in the rst period to only ten papers in 2012-2017.
This might well be due to the fact that, during the rst years of
research, there is a huge need for tools and frameworks for laying
down the groundworks for eective research. For instance, Table
2 shows a list of well-known simulation tools for VANETs, and
it is noticeable that all of them were developed during the rst
period or even earlier. Nevertheless, with the adoption of vehicular
communication technology by automobile manufacturers in the
following years, it is possible that many new tools and frameworks
are going to be introduced.
Lastly, two research categories had shown a slight decrease over
the last decade: mobility modeling and analysis, and application
layer protocols and services. With regards to the former, as there are
already realistic two-dimensional trac-based vehicular mobility
models [
4
,
13
], we foresee future research to be more specic, cover-
ing topics like geo-social mobility modeling, and three-dimensional
connectivity analysis. On the other hand, concerning the latter topic
(i.e., APP class), we were somehow surprised by the decrease on
studies concerning the application layer, however, it’s understand-
able that, after an initial period modeling the behavior of existing
applications in the new VANETs scenarios, community has moved
to more impactful research topics.
3.2 Has the MANET/VANET research
community evolved in conducting
simulation-based studies?
First of all, the results corroborate the fact that simulation is still
the leading approach for validating and evaluating solutions in
MANETs. More outstanding is that the percentage of articles that
employed simulation remains basically the same as in 2000-2005:
three out of four papers include simulation as the main tool (see
Table 4).
Other research methods, namely experimentation (i.e., real-world
measurements), and formal mathematical analysis reached 12.7%
and 24.7%, respectively. The sum of the percentages is higher than
100% since 38 works used more than one evaluation technique as
presented in Figure 3a. Only six publications employed no method
at all, while 277 papers employed at least one out of the three
approaches.
Figure 3b depicts the variation on the number of publications by
research method over the last two ve-year periods. One possible
reason for the increase of simulation based works and the reduction
of formal analysis studies is the development/availability of more
specic and featured simulators for vehicular networking, including
tools for realistic road trac generation. A summarized overview
of freeware and open source tools that either have emerged or have
added support for VANETs in the last decade is provided in Table 2.
Such increase in the number of simulation tools, openly available
for the community [
22
], most likely contributed to the reduction of
publications (27.3% to 7.62%, Table 4) based on self-developed or
custom simulators.
One astonishing result is that only a single publication made it
clear whether the code, or dataset, is publicly available. As code, we
mean any piece of software, including scripts, required for repeating
the simulations/experiments. Without this resource, no researcher
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

(a) Number of publications according to the evaluation method em-
ployed: Simulation (SI), Experimentation (EX), and Formal Analysis
(FA).
(b) Variation over the periods.
Figure 3: On the evaluation method employed.
can reproduce, and eventually validate, one other’s research. Indeed,
reproducibility is not yet a common approach of neither VANETs’
research nor any other computer networks’ area [
29
]. As a matter
of fact, the lack of replicability and reproducibility may be a major
issue in all sub-areas of computer science [7].
Even though a minority of papers actually include references
(i.e, links) to author’s web pages for some of the artifacts (e.g., code,
dataset), the links are rarely permanent, and often become unavail-
able after a few months or years. For supporting open science, as
well as the need for the 3Rs (repeatability, replicability, and repro-
ducibility), cloud-based platforms have recently emerged aiming at
providing a way for researchers to share all kind of data underlying
their publications. A comprehensive list of hundreds of repositories
is accessible at [
24
]. We briey describe nine open research data
repositories in Table 3, highlighting the number of datasets and
code repositories that are considered as an artifact of at least one
publication related to VANETs.
It is noticeable that only a tiny part of research projects get their
artifacts actually published. The platforms with more VANETs’
registries are Github and CRAWDAD. The focus of the former
is on code whereas the latter is on archiving data sets from real
networking experiments. In more recent platforms, any type of
data may be uploaded, including code, datasets, documents, images,
and videos. In most cases, there is DOI support making it possible
to cite the resource, also with versioning feature.
Table 3: Research data repositories.
Platform Launch Size1Data2Code3
codeocean.com 2017 ∼100 0 0
data.mendeley.com 2015 >200 0 0
gshare.com 2011 >10K 5 0
github.com 2008 >10M 0 22
zenodo.org 2013 >8K 1 0
dataverse.harvard.edu 2007 >70K 0 0
Wolfram Repository 2017 ∼600 0 0
crawdad.org 2005 121 14 0
datadryad.org 2008 18K 0 0
1Number of research data records.
2Number of VANETs’ dataset records.
3Number of VANETs’ code records.
We have also noticed that no paper had included information
on the pseudo-random number generator (PRNG) employed along
the simulations. The remaining results regarding simulation and
environment issues are somewhat similar to MobiHoc’s survey [
17
].
With regard to basic input parameters (Table 4), our ndings
endorse the claim that there is a low credibility in simulation-based
studies. More than one-third of the papers made no statement
regarding node transmission range or the size of the simulation area,
while only 15.71% cited the trac pattern (e.g., CBR). In addition
to that, nearly one-third of the publications did not mention the
number of simulation runs (i.e., trials).
As a direct result of that, only 34.76% of all publications presented
the condence intervals along the plots. Nevertheless, this rate is
almost three times higher than the one reported in [
17
] (12.5%),
which suggests some improvement on the credibility of VANETs’
simulation-based studies.
3.3 Tools, models, and protocols preferences in
VANETs’ research
Even though there are several network simulators for a great va-
riety of networks, our focus is on simulators suitable for vehicu-
lar networks. In the last years, there have been changes in tools’
preferences. Network simulator 2 (NS-2) was the most adopted,
reaching 37.9% of the 198 articles citing the application of some sort
of network simulator. NS-2 was followed by OMNET++ with 8.6%,
MATLAB with 8.1%, and 7.1% for NS-3. Flexible, well-documented,
and popular open source frameworks such as Veins [
28
] enhanced
the adoption of OMNET++, mainly for the simulation of V2X net-
works. It is noticeable that while NS-2 usage dropped 19.5% (41
to 33) since the rst ve years, OMNET++ adoption has increased
more than double (5 to 12).
Figure 5 presents the main vehicle movement pattern generators
employed by researchers. The most used mobility simulators are
SUMO (31.2%) and VanetMobiSim (11.8%). A high percentage (42.4%)
of all simulation-based papers do not report the type of mobility
tool employed for the simulations.
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

Table 4: Survey results comparison for papers published in ACM’s MobiHoc conference (2000-2005) and in IEEE’s VTC, ACM’s
MobiCom, and ACM’s MobiHoc conferences, and IEEE’s TMC journal (2007-2017).
Simulator and Environment
Our survey’s results1MobiHoc’s survey [17]2
Totals Percent Description Totals Percent
210 of 283 74.20% Used simulation in the research. 114 of 151 75.5 %
1 of 210 0.47% Stated the code was available to others. 0 of 114 0%
180 of 210 85.71% Stated which simulator was used. 80 of 114 70.2%
16 of 210 7.62% Used self-developed or custom simulators 22 of 80 27.3%
43 of 210 20.48% Stated which version of the public simulator was used. 7 of 58 12.1%
17 of 210 8.09% Stated which operating system was used. 3 of 114 2.6%
20 of 210 9.52% Addressed initialization bias. 8 of 114 7%
52 of 210 24.76% Addressed the type of simulation. 48 of 114 42.1%
6 of 210 2.86% Addressed the PRNG used. 0 of 114 0%
Simulation Input Parameters
Our survey’s results1MobiHoc’s survey [17]2
Totals Percent Description Totals Percent
122 of 210 58.10% Stated the size of the simulation area. 62 of 109 56.9%
137 of 210 65.23% Stated the transmission range. 62 of 109 56.9%
88 of 210 41.90% Stated the simulation duration. 49 of 109 45%
83 of 210 39.52% Stated the trac send rate. 41 of 109 37.5%
33 of 210 15.71% Stated the trac type (e.g., CBR, VBR, etc.) 31 of 109 28.4%
73 of 210 34.76% Stated the number of simulation runs (iterations). 39 of 109 35.8%
Plots / Graphs
Our survey’s results1MobiHoc’s survey [17]2
Totals Percent Description Totals Percent
206 of 210 98.09% Used plots to illustrate the simulation results. 112 of 114 98.2%
73 of 210 34.76% Used condence intervals on the plots. 14 of 112 12.5%
88 of 210 41.90% Missed labels or units on the data. 28 of 112 25%
1A total of 283 papers published in IEEE’s VTC Conference, IEEE’s TMC, ACM’s MobiCom,
and ACM’s MobiHoc, from 2007 to 2017.
2A total of 151 papers published in ACM’s MobiHoc conference from 2000 to 2005 [17].
Figure 4: Network simulator preference.
When taking into account the papers related to the application
layer, we have identied a particular trend: fewer publications
regard safety and eciency issues, while more works deal with
entertainment and urban sensing. The former issues are among the
Figure 5: Mobility simulation environment.
original motivations for VANETs’ studies and their integration with
Intelligent Transport System (ITS). Nevertheless, for either end
users or industry players, entertainment integrated services such as
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

multimedia streaming [
8
] (e.g., geo-located touristic video guide),
and urban sensing for a greener environment [
2
] (e.g., reduction of
CO2emission), are subjects of current and future research.
It is possible that these four applications’ categories are going
to be targeted by distinct stakeholders. Vehicles’ manufacturers
may focus on providing safety and urban sensing applications,
whereas public agencies (e.g., the department of transportation)
work on developing trac ecient solutions. Lastly, entertainment
applications might well be developed by major software companies.
Figure 6: Research trends for VANETs’ applications.
While somehow unexpected, the results presented in Figure 7
show that the application of real maps reduced when compared
to synthetic maps (i.e., user-dened maps, usually in a Manhat-
tan geographic style). We think that a proper scenario should be
based on a realistic mobility model and a real map, which could be
easily accomplished with freely available tools such as SUMO and
OpenStreeMap [12].
Figure 7: Map type/style.
In addition to the overall analysis of the material and methods
presented in the papers (as stated in Table 4), we have also extracted
more specic information related to three key components for any
wireless network simulation: mobility model, propagation model,
and routing protocol. For each one of these components, we present
the most chosen options.
The majority of the surveyed papers employed some mobility
model: 221 out of 283 (78.10%); however, only 43.44% of them stated
which model was used, resulting on a considerable omission rate of
56.56%. As for the propagation model, we observed an even worse
result (61.22%).
Propagation models are usually integrated into network simu-
lators, whereas external tools (e.g., SUMO [
4
]) are frequently cho-
sen for generating realistic vehicle movement patterns. Very often
(35.42%) authors cite the adopted mobility tool but they do not men-
tion which model was then selected. Mobility model tools, such as
VanetMobiSim [
13
], have several models available so that it is para-
mount to mention which specic model was chosen (e.g., Intelligent
Driver Model with Lane Changes, IDM-LC).
With regards to propagation models, the Nakagami was the most
employed model, reaching a preference rate of 40.96%, followed
by the Two-Ray Ground model with 31.32%. Note that these per-
centages are not exclusive, since more than one model is usually
employed in the same study.
Most network simulators have both deterministic and proba-
bilistic models, including all those described in Table 5. However,
several solid studies (such as [
5
]) have shown that a realistic model
should consider vehicles as obstacles, because they impact on the
LOS obstruction, received signal power, and the packet reception
rate.
About one-third of all papers used some routing protocol (97 out
of 283, 34.28%), being two-third of them designed by the authors
themselves. The geographic GPRS protocol, and the reactive AODV
protocol were the most employed ones, with 18.75% and 14.06%
respectively. Among the link state protocols, OLSR was the most
adopted one, representing 9.38% among all protocols. In several
cases (13 out of 64), the authors did not mention which protocol
was chosen, but they only cited the publication where the protocol
was rst introduced.
4 DISCUSSION
Some general remarks can be highlighted from survey results to
indicate the trends of researches during the last years such as the
increase in the number of published papers related to the physical
layer, link layer and routing solutions, and also new services and
data management studies. Such increase is directly related to the
new demands for solutions from users and manufacturers. The
former asked not only for new applications, but mainly for new
mechanisms to improve applications in terms of QoS, security, and
geographical correlation, in order to enhance users’ experience.
The latter need new infrastructure technologies to improve data
transmission, while supporting new users’ services.
On the other hand, investigations regarding users’ application
behaviour and performance evaluation comparing existing solu-
tions designed for dierent environments (e.g., MANETs’ routing
protocols applied in VANETs’ scenarios) received less attention
from the community. These two topics were extremely important
in the initial development of VANETs since they allowed a better
understanding of the eld and how/when existing solutions could
be employed. After that moment, it was expected that new solutions
were proposed considering the specic requirements of VANETs,
moving eorts to the design of new technologies and protocols.
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

Table 5: Survey results regarding researchers’ preferences on
mobility and propagation models, and routing protocols in
simulation-based studies.
Mobility Model (MM)
Totals Percent Description
221 of 283 78.10% Used MM in the research.
96 of 221 43.44% Stated which model/tool was used.
34 of 96 35.42%
Only stated the mobility trace
generator tool (e.g., VanetMobisim),
not specifying the MM.
16 of 96 16.67% Used Manhattan model.
14 of 96 14.58% Used own (or proposed) model.
13 of 96 13.54% Used IDM model.
7 of 96 7.29% Used Freeway model.
7 of 96 7.29% Used other models.
4 of 96 4.17% Used only real traces.
3 of 96 3.13% Used random waypoint model.
Propagation Model (PM)
Totals Percent Description
214 of 283 75.62% Used PM in the research.
83 of 214 38.78% Stated which model was used.
34 of 83 40.96% Used Nakagami model.
26 of 83 31.32% Used Two-Ray Ground model.
8 of 83 9.64% Used Free Space (Friis) model.
8 of 83 9.64% Used Rayleigh model.
7 of 83 8.43% Other models (e.g., CORNER [11]).
6 of 83 7.23% Used own model.
Routing Protocol (RP)
Totals Percent Description
97 of 283 34.28% Used RP in the research.
64 of 97 65.98% Stated which protocol was used.
33 of 64 51.56% Used own (or proposed) protocol.
12 of 64 18.75% Used the GPSR protocol.
9 of 64 14.06% Used the AODV protocol.
6 of 64 9.38% Used the OLSR protocol.
4 of 64 6.25% Other protocols.
3 of 64 4.69% Used the DSR protocol.
3 of 64 4.69% Used the DYMO protocol.
Another interesting point is the fact that more works are relying
on simulation platforms to conduct their experiments. This happens
because more sophisticated simulation platforms, and auxiliary
tools (e.g., enhanced models for mobility, radio propagation, and
trac), were developed focusing on VANETs’ specic scenarios. The
evolution on supporting tools such as SUMO and VanetMobiSim
allowed researchers to employ general purpose network simulators
(mainly NS2 and OMNet++) instead of developing new simulation
tools from scratch.
Nevertheless, reproducibility remains a huge hurdle since au-
thors rarely make available any material (e.g., source code, con-
guration les, data sets, and mobility traces) to be used by other
researchers. In addition to that, a considerable number of papers
presented only basic information for understanding the experi-
ments (e.g., the lack of number of runs/trials, radio transmission
range, size of the simulation area, trac pattern, mobility model,
propagation model, and routing protocol).
Considering those specic points, when one compares the gen-
eral results presented by our work to the ones presented by the
papers discussed in Section 2, it is possible to see that some research
topics switched priorities, with simulation gaining more notewor-
thiness due to more accurate and reliable tools. However, even
though there was some noticeable evolution, most of the problems
previously identied remain present in current publications.
5 CONCLUSION
As vehicular networks are expected to be integrated with the 5G mo-
bile technology by 2020, we noted that less research has been done
regarding protocol performance evaluation. The focus has moved
toward new solutions for services such as location tracking and
estimate correction, QoS, cross-layer protocols, and maximization
of network resources based on network coding. Moreover, consid-
ering the four categories of VANETs’ applications as described by
Gerla and Kleinrock [
9
], our results indicate that fewer publications
concern safety and eciency issues, while a growing number are
related to entertainment and environmental urban sensing.
The results of our survey show that VANETs’ simulation-based
studies still lack credibility due to issues similar to those reported
in previous studies published in 2002 [
25
], 2005 [
17
], and 2014 [
27
].
Besides omitting technical information, such as the implied models
(e.g., mobility, propagation), and input/conguration parameters,
we could not identify a single repository for code or dataset re-
sulting from any of the surveyed papers. Nevertheless, with the
advent of open science principles and the recent development of
cloud-based platforms for research data sharing (Table 3), we expect
this reality to change in the next years.
REFERENCES
[1]
Nabeel Akhtar, Sinem Coleri Ergen, and Oznur Ozkasap. 2015. Vehicle mobility
and communication channel models for realistic and ecient highway VANET
simulation. IEEE Transactions on Vehicular Technology 64, 1 (2015), 248–262.
https://doi.org/10.1109/JPROC.2010.2070470
[2]
Maazen Alsabaan, Waleed Alasmary, Abdurhman Albasir, and Kshirasagar Naik.
2013. Vehicular Networks for a Greener Environment: A Survey. IEEE Communi-
cations Surveys and Tutorials 15, 3 (2013), 1372–1388. https://doi.org/10.1109/
SURV.2012.101912.00184
[3]
Todd R Andel and Alec Yasinsac. 2006. On the credibility of manet simulations.
Computer 39, 7 (2006), 48–54. https://doi.org/10.1109/MC.2006.242
[4]
Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. 2011.
SUMO–simulation of urban mobility: an overview. In Proceedings of SIMUL 2011,
The Third International Conference on Advances in System Simulation. ThinkMind.
[5]
Mate Boban, Tiago TV Vinhoza, Michel Ferreira, Joao Barros, and Ozan K Tonguz.
2011. Impact of vehicles as obstacles in vehicular ad hoc networks. IEEE journal
on selected areas in communications 29, 1 (2011), 15–28. https://doi.org/10.1109/
JSAC.2011.110103
[6]
JiuJun Cheng, JunLu Cheng, MengChu Zhou, FuQiang Liu, ShangCe Gao, and
Cong Liu. 2015. Routing in internet of vehicles: A review. IEEE Transactions
on Intelligent Transportation Systems 16, 5 (2015), 2339–2352. https://doi.org/10.
1109/TITS.2015.2423667
[7]
Christian Collberg and Todd A. Proebsting. 2016. Repeatability in Computer
Systems Research. Commun. ACM 59, 3 (Feb. 2016), 62–69. https://doi.org/10.
1145/2812803
[8]
Mario De Felice, Eduardo Cerqueira, Adalberto Melo, Mario Gerla, Francesca
Cuomo, and Andrea Baiocchi. 2015. A distributed beaconless routing protocol for
real-time video dissemination in multimedia VANETs. Computer Communications
58 (2015), 40–52. https://doi.org/10.1016/j.comcom.2014.08.009
[9]
Mario Gerla and Leonard Kleinrock. 2011. Vehicular networks and the future of
the mobile internet. Computer Networks 55, 2 (2011), 457–469. https://doi.org/10.
1016/j.comnet.2010.10.015
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018

[10]
Eugenio Giordano, Enzo De Sena, Giovanni Pau, and Mario Gerla. 2010. Vergilius:
A scenario generator for vanet. In Vehicular Technology Conference (VTC 2010-
Spring), 2010 IEEE 71st. IEEE, 1–5. https://doi.org/10.1109/VETECS.2010.5494208
[11]
Eugenio Giordano, Raphael Frank, Giovanni Pau, and Mario Gerla. 2010. CORNER:
a realistic urban propagation model for VANET. In Wireless On-demand Network
Systems and Services (WONS), 2010 Seventh International Conference on. IEEE,
57–60.
[12]
Mordechai Haklay and Patrick Weber. 2008. Openstreetmap: User-generated
street maps. IEEE Pervasive Computing 7, 4 (2008), 12–18. https://doi.org/10.
1109/MPRV.2008.80
[13]
Jérôme Härri, Fethi Filali, Christian Bonnet, and Marco Fiore. 2006. VanetMobiSim:
generating realistic mobility patterns for VANETs. In Proceedings of the 3rd
international workshop on Vehicular ad hoc networks. ACM, 96–97. https://doi.
org/10.1145/1161064.1161084
[14]
Thomas R Henderson, Mathieu Lacage, George F Riley, C Dowell, and J Kopena.
2008. Network simulations with the ns-3 simulator. SIGCOMM demonstration 14,
14 (2008), 527.
[15]
Feliz Kristianto Karnadi, Zhi Hai Mo, and Kun-chan Lan. 2007. Rapid generation of
realistic mobility models for VANET. In Wireless communications and networking
conference, 2007. WCNC 2007. IEEE. IEEE, 2506–2511.
[16]
David Kotz, Calvin Newport, Robert S Gray, Jason Liu, Yougu Yuan, and Chip
Elliott. 2004. Experimental evaluation of wireless simulation assumptions. In
Proceedings of the 7th ACM international symposium on Modeling, analysis and
simulation of wireless and mobile systems. ACM, 78–82.
[17]
Stuart Kurkowski, Tracy Camp, and Michael Colagrosso. 2005. MANET simula-
tion studies: the incredibles. ACM SIGMOBILE Mobile Computing and Communi-
cations Review 9, 4 (2005), 50–61. https://doi.org/10.1145/1096166.1096174
[18]
Uichin Lee, Ryan Cheung, and Mario Gerla. 2009. Emerging vehicular applications.
In Vehicular Networks: From Theory to Practice, Stephan Olariu and Michele C.
Weigle (Eds.). Chapman and Hall/CRC", Chapter 6, 1–26.
[19]
Kumiko Maeda, Takaaki Umedu, Hirozumi Yamaguchi, Keiichi Yasumoto, and
Teruo Higashino Higashino. 2006. MobiREAL: scenario generation and toolset
for MANET simulation with realistic node mobility. In Mobile Data Management,
2006. MDM 2006. 7th International Conference on. IEEE, 55–55. https://doi.org/10.
1109/MDM.2006.106
[20]
Rahul Mangharam, Daniel Weller, Raj Rajkumar, Priyantha Mudalige, and Fan Bai.
2006. Groovenet: A hybrid simulator for vehicle-to-vehicle networks. In Mobile
and Ubiquitous Systems: Networking & Services, 2006 Third Annual International
Conference on. IEEE, 1–8. https://doi.org/10.1109/MOBIQ.2006.340441
[21]
Francisco J Martinez, J-C Cano, Carlos T Calafate, and Pietro Manzoni. 2008.
Citymob: a mobility model pattern generator for VANETs. In Communications
Workshops, 2008. ICC Workshops’ 08. IEEE International Conference on. IEEE, 370–
374. https://doi.org/10.1109/ICCW.2008.76
[22]
Francisco J Martinez, Chai Keong Toh, Juan-Carlos Cano, Carlos T Calafate, and
Pietro Manzoni. 2011. A survey and comparative study of simulators for vehicular
ad hoc networks (VANETs). Wireless Communications and Mobile Computing 11,
7 (2011), 813–828. https://doi.org/10.1002/wcm.859
[23]
Rupendra Nath Mitra and Dharma P Agrawal. 2015. 5G mobile technology: A
survey. ICT Express 1, 3 (2015), 132–137. https://doi.org/10.1016/j.icte.2016.01.003
[24]
Heinz Pampel, Paul Vierkant, Frank Scholze, Roland Bertelmann, Maxi Kindling,
Jens Klump, Hans-Jürgen Goebelbecker, Jens Gundlach, Peter Schirmbacher, and
Uwe Dierolf. 2013. Making research data repositories visible: The re3data. org
registry. PloS one 8, 11 (2013), e78080. https://doi.org/10.1371/journal.pone.
0078080
[25]
Krzysztof Pawlikowski, H-DJ Jeong, and J-SR Lee. 2002. On credibility of simula-
tion studies of telecommunication networks. IEEE Communications magazine 40,
1 (2002), 132–139. https://doi.org/10.1109/35.978060
[26]
Michal Piorkowski, Maxim Raya, A Lezama Lugo, Panagiotis Papadimitratos,
Matthias Grossglauser, and J-P Hubaux. 2008. TraNS: realistic joint trac and
network simulator for VANETs. ACM SIGMOBILE mobile computing and commu-
nications review 12, 1 (2008), 31–33. https://doi.org/10.1145/1374512.1374522
[27]
Nurul I Sarkar and Jairo A Gutiérrez. 2014. Revisiting the issue of the credibility
of simulation studies in telecommunication networks: highlighting the results of
a comprehensive survey of IEEE publications. IEEE Communications Magazine
52, 5 (2014), 218–224. https://doi.org/10.1109/MCOM.2014.6815915
[28]
Christoph Sommer, Reinhard German, and Falko Dressler. 2011. Bidirectionally
Coupled Network and Road Trac Simulation for Improved IVC Analysis. IEEE
Transactions on Mobile Computing 10, 1 (January 2011), 3–15. https://doi.org/10.
1109/TMC.2010.133
[29]
Adelinde M Uhrmacher, Sally Brailsford, Jason Liu, Markus Rabe, and Andreas
Tolk. 2016. Panel - Reproducible research in discrete event simulation - A must or
rather a maybe?. In Winter Simulation Conference (WSC), 2016. IEEE, 1301–1315.
[30]
András Varga and Rudolf Hornig. 2008. An overview of the OMNeT++simulation
environment. In Proceedings of the 1st international conference on Simulation tools
and techniques for communications, networks and systems & workshops. ICST
(Institute for Computer Sciences, Social-Informatics and Telecommunications
Engineering), 60.
[31]
SY Wang, CL Chou, YH Chiu, YS Tzeng, MS Hsu, YW Cheng, WL Liu, and TW
Ho. 2007. NCT Uns 4.0: An integrated simulation platform for vehicular trac,
communication, and network researches. In Vehicular Technology Conference,
2007. VTC-2007 Fall. 2007 IEEE 66th. IEEE, 2081–2085. https://doi.org/10.1109/
VETECF.2007.437
ACM SIGCOMM Computer Communication Review Volume 48 Issue 2, April 2018