Content uploaded by Thien Duc Hua
Author content
All content in this area was uploaded by Thien Duc Hua on Sep 27, 2023
Content may be subject to copyright.
A Review on Recent Approaches in mmWave
UAV-aided Communication Networks and Open
Issues
Quang Tuan Do, Demeke Shumeye Lakew, Anh Tien Tran, Duc Thien Hua, and Sungrae Cho
School of Computer Science and Engineering, Chung-Ang University, Seoul, South Korea
Email: {dqtuan, demeke, attran, thien}@uclab.re.kr, srcho@cau.ac.kr
Abstract—Recently, the use of unmanned aerial vehicles
(UAVs) is spreading to many fields, especially for wireless
communication-related tasks. However, such communication is
facing many challenges, as the sub-6 GHz frequency band is
now heavily occupied. As a result, millimeter-wave (mmWave)
frequency band communication is now a promising technology
to tackle that issue. By equipping multiple antennas to the UAV
to perform 3D beamforming, as well as making good use of
the flexible mobility of the UAV, we can greatly improve the
communication link in mmWave communication systems. On the
other hand, the trend of using intelligent-based learning methods,
specifically reinforcement learning greatly increases recently, due
to their ability to capture the dynamic of complex systems.
In this study, we review recent approaches in mmWave UAV-
aided communication networks. We first introduce the main
characteristics of mmWave UAV networks, then we provide some
insight into the recent trend of applying intelligent learning-based
methods for solving this type of system. After that, some open
issues and potential research directions for the mmWave UAV
communication systems are provided.
Index Terms—UAV, mmWave, reinforcement learning, wireless
communication
I. INTRODUCTION
Recently, due to the fast growth speed in the number
of Internet of Things (IoT) devices, the demand for high-
quality wireless communications services is now growing at
an unprecedented rate. The uses of aerial platforms, such as
unmanned aerial vehicles (UAV) as aerial base stations (BSs)
or relay nodes for communication, now play an important role
in providing ubiquitous wireless services [1]. UAVs as aerial
BSs can greatly improve the coverage ability and provide a
flexible communication service under various scenarios, due
to their fully configurable mobility. Additionally, UAVs can
establish more reliable line-of-sight (LoS) links with ground
users, allowing UAVs as aerial BSs to have better communica-
tion channels than ground BSs. For example, recently, multiple
UAVs can be used as aerial BSs to provide services to ground
terminals with a good LoS channel, like in [2], or combining
it with rate-split multiple access (RSMA) like in [3]. In [4]
Lakew et al. proposed a placement and trajectory design for
a single UAV-assisted wireless network. Additionally, high-
altitude platforms (HAPs) and low-altitude platforms (LAPs)
can be combined, enabling the significant advantages of both
types of UAVs to be effectively used [5], [6], [7], [8]. Hua
et al. have also studied the integration system of the high-
altitude platforms (HAP) and low-altitude platforms (LAP) to
improve the communication fairness coverage [9]. However,
such communication is facing many challenges, as the sub-
6 GHz frequency band is now heavily occupied. As a result,
millimeter-wave (mmWave) frequency band communication is
now a promising technology to tackle that issue. By equipping
multiple antennas to the UAV to perform 3D beamforming,
as well as making good use of the flexible mobility of the
UAV, we can greatly improve the communication link in
mmWave communication systems. Additionally, due to the
high complexity of dynamic communication networks with
a large number of complicated constraints that need to be
satisfied, recent learning-based methods are emerging to deal
with those issues. In this paper, we provide an overview of
mmWave UAV-aided communication networks.
The remaining of this paper is structured as follows: Section
II provides some main characteristics related to mmWave
UAV communication networks, we also provide a generalized
system model in this section. In Section III, we introduce about
recent reinforcement learning approaches in mmWave UAV
communication systems. Section IV discussed some open is-
sues and research directions in mmWave UAV communication
networks. And lastly, we conclude this paper in Section V.
II. CHARACTERISTICS OF MMWAV E UAV NETWORK
In this study case, we consider a cellular network with either
a single UAV or a total of MUAV to provide a communication
link for a total of Kusers on the ground. Each UAV may
have a battery budget depending on applications [10]. In this
type of study, we usually consider a limited target area, where
all ground users’ positions are randomly assigned within the
ground level of the target area. Meanwhile, each UAV location
may be depending on the specific application, for example,
there are some systems where we consider optimizing the 3D
deployment location [11], [12], and some other systems may
have one or more charging stations serving as starting and
ending point of the UAV flight path [13], otherwise we can
consider the UAV’s location to be also randomly assigned in
the target area like that of ground users, but with a predefined
altitude, which can be either fixed throughout the entire flight
or can be dynamically adjusted in a range of possible altitudes
[14].
728978-1-6654-6268-6/23/$31.00 ©2023 IEEE ICOIN 2023
2023 International Conference on Information Networking (ICOIN) | 978-1-6654-6268-6/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICOIN56518.2023.10049043
Authorized licensed use limited to: Queens University Belfast. Downloaded on September 27,2023 at 13:24:24 UTC from IEEE Xplore. Restrictions apply.
For antenna array structure, each UAV is equipped with
Nantenna elements in different geometries. The main 3
geometries that are usually used the most are uniform linear
array (ULA), uniform rectangular array (URA), and uniform
circular (UCA) [15]. In the ULA structure, all Nantenna
elements are lying in the same line and are separated by
an equal distance from each other. With the URA structure,
Nantenna elements are evenly distributed in a plane in a
rectangular shape, with Nxnumber of antenna elements in
the x-direction and Nynumber of antenna elements in the
y-direction of the Cartesian coordinate. Meanwhile, the UCA
structure consists of Nantenna elements that are also evenly
distributed in a plane like URA, but in this structure, the
elements formed a circular shape instead [15]. Each antenna
array geometry has a corresponding formula for calculating
the steering vector, which represents the set of phase delays
a plane wave experiences, with respect to an arbitrary origin.
The steering vector formulation for each specific antenna array
geometry is calculated using the azimuth angle and elevation
angle of the communication link between the UAV and ground
user, which is given in [15]. On the other hand, the azimuth
angle and elevation angle can be derived using basic geometry,
and it will be depending on whether the current application is
for downlink transmission or uplink transmission [16].
The phases of each antenna array element can be controlled
by changing the steering vector, which will form different
wave radiation patterns. By intelligently changing the steering
vector, we can form a narrow beam that points to a specific
location, we can greatly improve the signal power received by
the target receiver as well as reduces the interference signal
transmitted to other undesired receivers. The above technique
is called beamforming [17].
The hardware structure to control the antenna array steering
vector is divided into three main categories. The first category
is the full digital beamforming structure. In this structure, each
antenna array element is connected to a single radio-frequency
(RF) chain that is dedicated to serving that only element, and
the steering vector is controlled entirely using digital signal
processing for a more flexible beamforming process. Each
RF chain can be used for a different transmission line, so
the digital structure can have a high number of simultaneous
communication links. However, its advantages come with a
drawback, which is higher hardware complexity and higher
energy consumption. Analog beamforming structures, on the
other hand, have very little energy consumption and a much
simpler hardware structure compared to digital ones, as they
now only have one single RF chain to control all the phase
shifts of all the antenna array elements. As a result, analog
structures are less flexible in beamforming. The last category
is the hybrid structure between digital and analog. In this
structure, a total of NRF <N number of RF chains are used
to control the beamforming of the antenna array, which can
be either connected to all the array elements or only partially
connected to some number of array elements. We can also use
the intelligent reflective surface (IRS) for beamforming instead
of using the above antenna array structures, which are basically
reflecting elements on a rectangular surface, where each of
those reflecting elements can independently act as an antenna
element by changing its own angle to reflect the incident signal
to a different desired location.
For channel modeling, we can use the ray-tracing method
for deterministic channel models. This method models the
channel between the transmitter and the receiver by trying the
simulate the actual electromagnetic wave propagation around
the surrounding environment [18], [19], [20]. By combining it
with the digital map data, we can calculate the exact channel,
like in the map-based method [21], [22]. However, those meth-
ods required an immense amount of computational resources,
are really hard to implement, and also lack of real-world
digital map database for an accurate evaluation. As a result,
stochastic channel modeling methods are preferred by most
existing works. This method is much simpler compared to the
deterministic ones, as it uses different statistical parameters
related to different environments to model the channel charac-
teristics, thus, resulting in far lower computational complexity
[23], [24]. For this type of channel modeling, the channel
between transmitter and receiver can be calculated using their
corresponding steering vectors that we already derived above
[15].
III. RECENT REINFORCEMENT LEARNING APPROACHES
Most of the existing works in the UAV communication
systems are used optimal control theory to control the UAV
in the system. The main advantage of using mathematic
optimization methods is that, if you are able to model the exact
dynamics inside your system, it can guarantee you a stable
system under any predefined constraints, as it is already a
well-understood field with strong mathematic tools to prove its
convergence ability. However, nowadays problems usually re-
quired interaction between multiple controllers, or specifically
in this study’s scope, multiple UAVs, with the endless stream
of data coming from the high complexity and dynamic of the
environment, which makes it really overwhelming to solve
using conventional optimization tools. Because of this reason,
recently, the trend of applying intelligent learning methods to
solve more complex and dynamic systems is emerging, with
most of them being reinforcement-learning-based methods, as
the Markov Decision Process (MDP) can be easily derived
from the mathematic optimization formulations due to using
the same principles.
In order to use RL methods, we first have to derive the
MDP from the optimization formulas that we use to describe
our system. The MDP is usually defined as a tuple consisting
of the state space, the action space, the reward function, and
the state transition probability function.
1) The state space usually carries information that each
agent can observe during each time step. Note that the
time dimension in the RL system is usually discrete into
equal-size time steps. Depending on the applications,
the state space may consist of information related to the
ground users’ location, information about other UAVs in
the system, the remaining battery if we consider that our
729
Authorized licensed use limited to: Queens University Belfast. Downloaded on September 27,2023 at 13:24:24 UTC from IEEE Xplore. Restrictions apply.
UAV can have awareness about its energy consumption,
and some other environment-related information.
2) The action space is a set of all possible actions that each
of our agents has to derive after each observation step.
After selecting an appropriate action from the learned
policy, each UAV performs that action and makes some
impact on the environment. Because of that, the environ-
ment will return some immediate reward that feedback
to each UAV about its impact on the environment, and
the state of the environment will change to a new state,
which now each of our UAVs has to observe to form a
closed feedback loop cycle.
3) The reward usually is the parameters that we are trying
to optimize in the system, with some penalty terms to
punish the agent if they violate some of our predefined
constraints.
4) Most of the time, for simplicity, the state information
will be deterministic, so we won’t need to use the
state transition probability. However, in some cases, we
can consider a stochastic state transition, in which the
next state is sampled from a state transition probability
function.
After deriving the MDP tuple, we can start the training process
for each agent in the system, using some deep RL approaches
such as deep deterministic policy gradient (DDPG) [25], deep
Q-network (DQN) [26], proximal policy optimization (PPO)
[27], etc. We can also use a replay memory buffer Dto
store the previous experience tuple containing the state, action,
next state, and reward. After that, during the training task,
we can randomly sample a batch of data from this buffer to
update the RL network’s parameters. By using this method,
we can handle the data correlation issue. We can also use
the ϵ-greedy strategy during our training task to help the
agent expands its environment exploration. The ϵhere is the
probability of performing action randomly in the current step,
which first will start out as a high value to help the UAV
explore its surrounding, then decrease over time, gradually
changing to use the best action output from the learned policy.
Despite their advantages, RL-based methods usually suffer
from a high amount of hyper-parameter combinations, which
require careful twists and evaluations to ensure that they can
successfully converge. Additionally, RL-based methods also
require an immense amount of interactions between RL agents
and the environment before they can successfully learn the
optimal policy, leading to high training time.
I V. O PEN ISSUES AND RESEARCH DIRECTIONS
Because applying mmWave to the UAV-aid communication
network is still in its early stage, the standards channel model
is still lacking. Most of the current works on the mmWave
UAV network are solving the system using a simplified version
of the channel modeling, and are more focused on evaluating
the performance of their proposed methods on their specific
scenario. As a result, we can’t verify those works in real-
world scenarios. Thus, a transmission parameters measurement
campaign for the mmWave UAV communication system is
urgently needed right now in order to build a robust channel
model for different environment settings.
Even though the high mobility of UAVs is an advantage, it
also comes with a drawback, as their communication channels
are now changing rapidly, compared to terrestrial ones. In
addition, their fast movement speed also makes the Doppler
shift effect worse for the high-frequency band of the mmWave.
Hence, it requires more studies on Doppler effect compensa-
tion for the mmWave UAV system.
In UAV communication systems, the scenario is extended to
3D space, which provides new domains of freedom (DOFs)
for optimization tasks, as the UAV can flexibly change its
own horizontal position and altitude to improve the channel
link. On the other hand, mmWave communication usually
requires 3D beamforming to mitigate the high propagation
loss characteristic of the mmWave link. As a result, joint
optimization of 3D position and beamforming are essential
in mmWave UAV communication systems.
V. C ONCLUSION
This paper provided a generalized system model to study
the characteristics of mmWave UAV communication systems,
which overviewed about the antenna array geometry, the
antenna array controller hardware structure, and the channel
modeling for this type of system. After that, we reviewed
the state-of-the-art learning-based methods that were recently
used to solve the high complexity nature of the mmWave
UAV communication network, particularly about reinforce-
ment learning-based solutions. Additionally, open issues re-
lated to the mmWave UAV system are discussed, which can
provide some potential research directions for the mmWave
UAV communication systems.
ACKNOWLEDGEMENT
This work was supported by the National Research Foun-
dation of Korea(NRF) grant funded by the Korea govern-
ment(MSIT)(No. 2022R1A4A5034130)
REFERENCES
[1] Y. Zeng, R. Zhang, and T.J. Lim, “Wireless communications with
unmanned aerial vehicles: Opportunities and challenges,” IEEE Com-
munications magazine, vol. 54, no. 5, pp. 36–42, 2016.
[2] T.H. Nguyen and L. Park, “Multi-agent drl-based task offloading in
hierarchical hap-lap networks,” in 2022 International Conference on In-
formation and Communication Technology Convergence (ICTC). IEEE,
2022.
[3] D.T. Hua, Q.T. Do, T.V. Nguyen, C. Ho, and S. Cho, “Trajectory design
in multi-uav-assisted rsma downlink communication,” in 2022 Inter-
national Conference on Information and Communication Technology
Convergence (ICTC). IEEE, 2022.
[4] D.S. Lakew, A. Masood, and S. Cho, “3d uav placement and trajectory
optimization in uav assisted wireless networks,” in 2020 International
Conference on Information Networking (ICOIN). IEEE, 2020, pp. 80–
82.
[5] D.S. Lakew, U. Sa’ad, N.N. Dao, W. Na, and S. Cho, “Routing in flying
ad hoc networks: A comprehensive survey,” IEEE Communications
Surveys & Tutorials, vol. 22, no. 2, pp. 1071–1120, 2020.
[6] D.S. Lakew, A.T. Tran, N.N. Dao, and S. Cho, “Intelligent offloading
and resource allocation in heterogeneous aerial access iot networks,”
IEEE Internet of Things Journal, pp. 1–1, 2022.
730
Authorized licensed use limited to: Queens University Belfast. Downloaded on September 27,2023 at 13:24:24 UTC from IEEE Xplore. Restrictions apply.
[7] D.S. Lakew, A.T. Tran, N.N. Dao, and S. Cho, “Intelligent offloading
and resource allocation in hap-assisted mec networks,” in 2021 Inter-
national Conference on Information and Communication Technology
Convergence (ICTC), 2021, pp. 1582–1587.
[8] A. Masood, T.V. Nguyen, T.P. Truong, and S. Cho, “Content caching in
hap-assisted multi-uav networks using hierarchical federated learning,”
in 2021 International Conference on Information and Communication
Technology Convergence (ICTC), 2021, pp. 1160–1162.
[9] D. Hua, D. Lakew, and S. Cho, “Drl-based energy
efficient communication coverage control in hierarchical hap-
lap network,” in 2022 International Conference on Information
Networking (ICOIN). Los Alamitos, CA, USA: IEEE
Computer Society, jan 2022, pp. 359–362. [Online]. Available:
https://doi.ieeecomputersociety.org/10.1109/ICOIN53446.2022.9687180
[10] Q. Zhang, W. Saad, and M. Bennis, “Distributional reinforcement learn-
ing for mmwave communications with intelligent reflectors on a uav,”
in GLOBECOM 2020-2020 IEEE Global Communications Conference.
IEEE, 2020, pp. 1–6.
[11] F. Li, C. He, X. Li, J. Peng, and K. Yang, “Geometric analysis-based
3d anti-block uav deployment for mmwave communications,” IEEE
Communications Letters, 2022.
[12] J. Sabzehali, V.K. Shah, H.S. Dhillon, and J.H. Reed, “3d placement and
orientation of mmwave-based uavs for guaranteed los coverage,” IEEE
Wireless Communications Letters, vol. 10, no. 8, pp. 1662–1666, 2021.
[13] W. Chen, D.K. Chang, and Y.J. Chen, “Trajectory control in self-
sustainable uav-aided mmwave networks: A constrained multi-agent re-
inforcement learning approach,” in 2022 IEEE International Conference
on Communications Workshops (ICC Workshops). IEEE, 2022, pp.
1017–1022.
[14] Z. Chen, N. Zhao, D.K.C. So, J. Tang, X.Y. Zhang, and K.K. Wong,
“Joint altitude and hybrid beamspace precoding optimization for uav-
enabled multiuser mmwave mimo system,” IEEE Transactions on Ve-
hicular Technology, vol. 71, no. 2, pp. 1713–1725, 2021.
[15] Z. Xiao, L. Zhu, Y. Liu, P. Yi, R. Zhang, X.G. Xia, and R. Schober, “A
survey on millimeter-wave beamforming enabled uav communications
and networking,” IEEE Communications Surveys & Tutorials, vol. 24,
no. 1, pp. 557–610, 2021.
[16] W. Yi, Y. Liu, Y. Deng, and A. Nallanathan, “Clustered uav networks
with millimeter wave communications: A stochastic geometry view,”
IEEE Transactions on Communications, vol. 68, no. 7, pp. 4342–4357,
2020.
[17] B.D. Van Veen and K.M. Buckley, “Beamforming: A versatile approach
to spatial filtering,” IEEE assp magazine, vol. 5, no. 2, pp. 4–24, 1988.
[18] W. Khawaja, O. Ozdemir, and I. Guvenc, “Uav air-to-ground channel
characterization for mmwave systems,” in 2017 IEEE 86th Vehicular
Technology Conference (VTC-Fall). IEEE, 2017, pp. 1–5.
[19] A. Perez, A. Fouda, and A.S. Ibrahim, “Ray tracing analysis for uav-
assisted integrated access and backhaul millimeter wave networks,” in
2019 IEEE 20th International Symposium on” A World of Wireless,
Mobile and Multimedia Networks”(WoWMoM). IEEE, 2019, pp. 1–5.
[20] L. Cheng, Q. Zhu, C.X. Wang, W. Zhong, B. Hua, and S. Jiang,
“Modeling and simulation for uav air-to-ground mmwave channels,”
in 2020 14th European Conference on Antennas and Propagation
(EuCAP). IEEE, 2020, pp. 1–5.
[21] S. Jiang, Q. Zhu, C.X. Wang, K. Mao, W. Xie, W. Zhong, and M. Yao,
“Map-based uav mmwave channel model and characteristics analysis,” in
2020 IEEE/CIC International Conference on Communications in China
(ICCC Workshops). IEEE, 2020, pp. 23–28.
[22] Q. Zhu, S. Jiang, C.X. Wang, B. Hua, K. Mao, X. Chen, and W. Zhong,
“Effects of digital map on the rt-based channel model for uav mmwave
communications,” in 2020 International Wireless Communications and
Mobile Computing (IWCMC). IEEE, 2020, pp. 1648–1653.
[23] A.A. Saleh and R. Valenzuela, “A statistical model for indoor multipath
propagation,” IEEE Journal on selected areas in communications, vol. 5,
no. 2, pp. 128–137, 1987.
[24] E.T. Michailidis, N. Nomikos, P. Trakadas, and A.G. Kanatas, “Three-
dimensional modeling of mmwave doubly massive mimo aerial fading
channels,” IEEE Transactions on Vehicular Technology, vol. 69, no. 2,
pp. 1190–1202, 2019.
[25] J. Xiang, Q. Li, X. Dong, and Z. Ren, “Continuous control with deep
reinforcement learning for mobile robot navigation,” in 2019 Chinese
Automation Congress (CAC). IEEE, 2019, pp. 1501–1506.
[26] T.H. Nguyen and L. Park, “A survey on deep reinforcement learning-
driven task offloading in aerial access networks.”
[27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox-
imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347,
2017.
731
Authorized licensed use limited to: Queens University Belfast. Downloaded on September 27,2023 at 13:24:24 UTC from IEEE Xplore. Restrictions apply.
