Smart Jamming Attacks in Wireless Networks During a Transmission Cycle: Stackelberg Game with Hierarchical Learning Solution

Abstract

Due to the broadcast nature of the shared medium,
wireless communications become more vulnerable to malicious
attacks. In this paper, we tackle the problem of jamming in
wireless network when the transmission of the jammer and
the transmitter occur with a non-zero cost. We focus on a
jammer who keeps track of the re-transmission attempts of
the packet until it is dropped. Firstly, we consider a power
control problem following a Nash Game model, where all players
take action simultaneously. Secondly, we consider a Stackelberg
Game model, in which the transmitter is the leader and the
jammer is the follower. As the jammer has the ability to sense
the transmission power, the transmitter adjusts its transmission
power accordingly, knowing that the jammer will do so. We
provide the closed-form expressions of the equilibrium strategies
where both the transmitter and the jammer have a complete
information. Thereafter, we consider a worst case scenario where
the transmitter has an incomplete information while the jammer
has a complete information. We introduce a Reinforcement
Learning method, thus, the transmitter can act autonomously in
a dynamic environment without knowing the above Game model.
It turns out that despite the jammer ability of sensing the active
channel, the transmitter can enhance its efficiency by predicting
the jammer reaction according to its own strategy

Full text

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
Smart Jamming Attacks in Wireless Networks
During a Transmission Cycle: Stackelberg Game
with Hierarchical Learning Solution
Moulay Abdellatif LMATER∗, Majed Haddad†, Abdelillah Karouit†and Abdelkrim Haqiq∗
∗Computer, Networks, Mobility and Modeling laboratory
FST, Hassan 1st University, Settat, Morocco
†LIA/CERI University of Avignon Agroparc,
BP 1228, 84911, Avignon, France
Abstract—Due to the broadcast nature of the shared medium,
wireless communications become more vulnerable to malicious
attacks. In this paper, we tackle the problem of jamming in
wireless network when the transmission of the jammer and
the transmitter occur with a non-zero cost. We focus on a
jammer who keeps track of the re-transmission attempts of
the packet until it is dropped. Firstly, we consider a power
control problem following a Nash Game model, where all players
take action simultaneously. Secondly, we consider a Stackelberg
Game model, in which the transmitter is the leader and the
jammer is the follower. As the jammer has the ability to sense
the transmission power, the transmitter adjusts its transmission
power accordingly, knowing that the jammer will do so. We
provide the closed-form expressions of the equilibrium strategies
where both the transmitter and the jammer have a complete
information. Thereafter, we consider a worst case scenario where
the transmitter has an incomplete information while the jammer
has a complete information. We introduce a Reinforcement
Learning method, thus, the transmitter can act autonomously in
a dynamic environment without knowing the above Game model.
It turns out that despite the jammer ability of sensing the active
channel, the transmitter can enhance its efficiency by predicting
the jammer reaction according to its own strategy.
Keywords—Wireless networks; jamming attacks; game theory;
reinforcement learning
I. INTRODUCTION
Technology and system requirements in the telecommuni-
cations domain are changing very rapidly. Over the previous
years, since the transition from analogue to digital communica-
tions, and from wired to wireless networks, different standards
and solutions have been adopted, implemented and modified,
often to deal with new and different business requirements.
However, in the development of the wireless Next Generation
Networks (NGNs) in which the layered architecture is adopted
the common challenge of how further improve the resource uti-
lization efficiency and provide better quality-of-service (QoS)
is conditioned by the capacity of systems to accommodate
changes quickly and with minimum impact on the services
already implemented. Furthermore, the flexible topology and
the low cost in term of use and setup have motivated the
exploration of the wireless NGNs with increasingly higher data
rates to meet the rapidly growing demand for wireless access.
Distributed protocols would be required to improve the
radio resource utilization and provide high performance for
wireless NGNs. In particular, an integrated design of Medium
Access Control (MAC) based on Wireless Random Access
(WRA) mechanism may lead to an efficient solution. This is
why it is important to design distributed algorithms which can
be used by the mobiles to compute the equilibrium strategy
and simultaneously achieve the optimal operation points. On
the other hand, the basic underlying assumption in legacy
WRA protocols is that any concurrent transmission of two
or more users causes all transmitted packets to be lost [2].
However, this model does not reflect the actual situation in
many practical wireless networks where some information
can be received correctly from a simultaneous transmission
of several packets. This result is due to the fact that the
packet arriving with the highest power has a good chance to
be detected accurately, even when other packets are present.
The effect of capture on Aloha [9], [10], [11], [18] and
on IEEE 802.11 protocol (Carrier Sense Multiple Access-
Collision Avoidance (CSMA/CA)) [19], [20], [21] has been
studied extensively in the literature and new MAC protocols
for channels with capture have been proposed. Furthermore,
the full system utilization requires coordination among users
which may be impractical given the distributed nature and
arbitrary topology changes of wireless collision channels.
Fig. 1. Layered architecture for wireless networks.
www.ijacsa.thesai.org 358 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
However, while seeking ways to increase the performance
of wireless network, there are increasing number of critical
security issues that need to be addressed in order to make
these wireless NGNs safer [7], [24], [25] (e.g., time-critical
services, military operations, etc.). Note that wireless networks
are vulnerable to security threats such as distributed denial of
service attacks (DoS), spoofing attacks, Sybil attacks, faked
sensing attacks and smart jamming attacks [7]. Thus the study
of jamming problem in the context of wireless networks is an
important challenge since it’s easy to destroy communications
due to the fact that the jammer can create dynamic and
intelligent jamming attacks [23], [5].
The Game theory provides a convenient framework for
approaching the power control in wireless based distributed
MAC protocols. In fact, given the broadcast nature of the
wireless MAC, the users are considered as selfish transmitters
[2], and each transmitter seeks to maximize its payoff, while
a malicious user tries to degrade the performance of the
whole system. In this paper, we consider the IEEE 802.11
MAC CSMA/CA mechanism which is used by a large number
of wireless systems, therefore, the problem of jamming can
occurs during the transmission duty. In addition, the adversary
or the jammer has to expend a significant amount of energy to
jam the selected frequency bands, also the continuous presence
of unusual high interference levels makes these attacks easy
to detect. Thus, the main challenge in this paper is to derive
the optimal strategy defense against the DoS attacks [16], [3],
[4], [8], knowing the fact that the behavior of a malicious user
may jam the network by sending abnormal packets to another
user to block the channel from doing any things useful (Fig.
1).
It is well known that the Game theoretical approach is an
appropriate concept to dealing with the competitive situation.
Compared to the approaches used in previous works [12], [13],
[14], [15], [17], etc. we are interested here in the impact of a
smart jammer on the transmitter power levels during the period
that starts at the first attempt of a packet transmission until the
next packet transmission first attempt, due to the fact that when
re-transmissions are used, the jammers cause the effective
network activity factor (and hence the interference among
the Receiver Sides (RSs) to be doubled [24]. In particular,
we consider a scenario where a single transmitter (player 1)
and a single jammer (player 2) coexist. The case of several
transmitters/jammers is a subject of future research. Namely,
the strategies of both the jammer and the transmitter are their
transmission power levels during the packet transmission cycle.
Since each packet transmission attempt incurs a cost in term
of power, we consider that the Game objective utilities of both
players are functions of the Signal to Interference plus Noise
Ratio (SINR) value and the transmission cost. Under this anti-
jamming Game based on power control problem, we propose
two Game formulations, Nash Game where all players act
simultaneously and Stackelberg Game where the transmitter
is considered as leader (i.e. first to determine its transmit
power) while jammer is considered as follower. At first, we
derive the Nash Equilibrium (NE) expression, thereafter, we
prove the existence of the Stackelberg Equilibrium (SE) and
by using the Simulated Annealing Algorithm we sort out the
SE measurement. From the comparison of the two schemes, we
deduce that the transmitter can efficiently enhance the system
performance. The main limitation with regard to the proposed
power control-based anti-jamming problem is that there may
be information loss for unknown jamming patterns. Thus,
we consider a worst case scenario where the transmitter has
an incomplete information while the jammer has a complete
information. We introduce a Reinforcement Learning method,
thus, the transmitter can act autonomously in a dynamic
environment without knowing neither the estimating jamming
patterns and parameters nor the above Game model.
The rest of this paper is outlined as follows. We briefly
describe the related work in Section II. Then, we introduce
the system model and the Game formulation in Section III.
In Sections IV and V, we analyze the system in the presence
of a regular and a smart jammer. In Section VI we propose a
hierarchical learning solution. Simulation results are provided
in Section VII. Finally, we conclude the paper and give some
perspectives for future research.
II. RE LATE D WOR K
Designing mechanisms that can be able to detect wireless
network jamming as well as avoid it has been widely studied
under several works. In [26], authors investigate the anti-
jamming problem with discrete power strategies, they formu-
late a Stackelberg Game to model the competitive interactions
between the user and jammer. Then, they analyzed the asymp-
totic convergence by proposing a hierarchical power control
algorithm (HPCA). In [27], a smart jammer can quickly learn
the transmission strategies of the legitimate transmitters, and
then he would adjust his strategy to damage the legitimate
transmission. Meanwhile, the transmitters are aware of the
existence of the smart jammer. The difference from [28] is
that they consider relay nodes which help the source counteract
a smart jammer. Furthermore, in [29] reinforcement learning
can be applied to determine transmission powers against a
jammer in a dynamic environment without knowing the un-
derlying Game model. In [1], authors propose an anti-jamming
Bayesian Stackelberg Game with incomplete information. In
all the previous works on anti-jamming, the authors consider
the problem transmitter-jammer during only one transmission
attempt.
In this paper, we study the power control problem during
a packet transmission cycle in the presence of a smart jam-
mer, which has energy-efficiency and keeps track of the re-
transmission attempts of the packet until that it is dropped. We
suppose that the power level set is continuous and we consider
a non-zero Game by introducing a transmission power cost.
III. SYS TE M MOD EL
Let a mobile use IEEE 802.11 CSMA/CA standard which
is the most widely known standard in wireless networks. We
assume that a transmission fails with probability that depends
on the SINR. If a transmission fails then it is attempted again
after some back-off time. After a certain number of attempts
Kthe packet is dropped. Let’s assume that the power is
controlled. Hence, the power of the mobile user used at the ith
transmission attempt can be denoted by Ti∈[0,¯
T]. Assume
that:
Ti=p0xi−1(1)
where p0≥0is the initial transmission power and x > 1is
the power multiplier factor for each re-transmission attempt. In
www.ijacsa.thesai.org 359 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
this paper, we examine a scenario with one transmitter, which
has its own traffic to send, and one jammer, which does not
have its own traffic and simply wants to jam the transmitter
attempts. As the mobile user spreads its signal over a common
frequency band and treats interference as noise, thus, the signal
to interference plus noise ratio at the ith transmission attempt
SI N Riat the receiver side is given by
SI N Ri=αTi
N+βJi
(2)
where Nis the background noise level on the channel,
Ji∈[0, J max]is the jammer power at the ith transmission
attempt, α > 0and β > 0denote the fading channel gain of
the mobile user and the jammer, respectively.
Since a jammer chooses which transmission or re-
transmission to jam, we assume that it jams all packets that
are in the back-off stage k≥K2, where K2is an integer, that
means that the competition starts from the back-off stage K2.
Since the quick detection of the start of a packet is becoming
very harder for the jammer and this is due to the large
bandwidths and the widely spread signals, we assume the worst
situation in which the jammer can jam the communication from
the first transmission attempt despite the fact that it arrives at
a completely unpredictable time and frequency.
On the other hand, Let’s define a cycle as the period that
starts from the first attempt of a packet transmission to the
first attempt of the next packet transmission. During a cycle,
we consider a Game in which the two mobiles are players.
Moreover, we consider that each transmission occurs a
certain cost and let C > 0and D > 0be the transmission costs
per unit power of the mobile user and jammer respectively. We
assume that players have perfect knowledge of the environment
state and costs constraint at the beginning of each cycle.
Let St={(p0, x)|0< p0≤pmax
0; 1 ≤x≤xmax}the
feasible set of the power multiplier and the initial transmission
power of the mobile user and Sj={(J1, J2, ...JK)|Ji≥
0; Ji≤Jmax}the feasible set of the jammer power vector.
We consider the following power control problem where (T, J )
is to be determined, where J= (J1, J2, ...JK)and T= (p0, x)
.
The mobile user objective is to achieve the maximum
PK
i=1 SI N Riwith the minimum cost. Intuitively, from (1)
and (2), the utility function of the mobile user during a cycle
denoted as U(T, J)is given by:
U(T, J) =
K
X
i=1
(αp0xi−1
N+βJi
−Cp0xi−1)(3)
The jammer objective is to achieve the minimum
PK
i=1 SI N Riwith the minimum cost. From (1) and (2),
the utility function of the jammer during a cycle denoted as
V(T, J)is given by:
V(T, J) =
K
X
i=1
(−αp0xi−1
N+βJi
−DJi)(4)
IV. NASH GAME
In this section, we assume the presence of a regular
jammer, and we consider a Game Gn= ({Transmitter, Regular
jammer}, {T, J}, {U, V}). Since the regular jammer does
not have the capability to sense the ongoing transmission
power, all players take actions simultaneously. We focus on
finding a Nash equilibrium in which neither the transmitter
nor the jammer can increase its utility function by unilaterally
changing its strategy. we define the Nash Equilibrium by the
following formulation:
TNE =Arg maxT∈StU(T, JNE )
JNE =Arg maxJ∈SjV(TN E , J)(5)
Theorem 1: Let a jammer without the intelligence of learn-
ing the transmitter strategy. There exists a NE (TN E , JNE )in
the Game, in addition,
C > α/N TN E = (0,1)
JNE = 0 (6)
C < α/(N+β Jmax)








TNE = (pmax
0, xmax)
JNE = (M in(Jmax,
Max(0,1
β(qpmax
0αβ
Dxmaxi−1
−N))))i∈[1,K]
(7)
ow (TNE = ( αD
βC 2,1)
JNE = ( 1
β(α
C−N))i∈[1,K]
(8)
Proof: By (3) we have:
∂U (T , J)
∂x =p0
K
X
i=2
(i−1)( α
N+βJi
−C)xi−2(9)
∂U (T , J)
∂p0
=
K
X
i=1
(α
N+βJi
−C)xi−1(10)
The first order partial derivative of V(T, J)with respect
to Jifor i∈[1, K], is
∂V (T , J)
∂Ji
=αβp0xi−1
(N+βJi)2−D(11)
The second order partial derivatives of the jammer
objective function are:
∂2V(T, J)
∂Ji∂Jj
=(−2αβ2p0xi−1
(N+βJi)3i=j
0ow (12)
Therefore, the Hessian matrix of V(T, J)with respect to the
vector Jis negative and V(T, J )is strictly concave in J. Thus
we consider the following cases:
•C > α/N :
As ∂U
∂T <0∀T∈St, thus xN E = 1 and pN E
0= 0
yielding TNE = (0,1) . By using the concavity of
V in J and setting ∂V (T ,J)
∂Jito zero, we have J0i=
1
β(qp0αβ
Dxi−1−N). Since 0≤Ji≤Jmax, let J0=
www.ijacsa.thesai.org 360 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
(Min(Jmax, M ax(0, J 0i)))i∈[1,K]. According to Fig.
1, we have ∀J∈Sj:V(TNE , J )≤V(TNE , J 0).
Thus JNE =J0= 0.
•C < α/(N+β Jmax):
As ∂U
∂T >0∀T∈St, then, pN E
0=pmax
0and
xNE =xmax . By using the concavity of V in
J and setting ∂V (T,J )
∂Jito zero, we have J0i=
1
β(qpmax
0αβ
Dxmaxi−1−N). Since 0≤Ji≤Jmax , let
J0= (Min(Jmax, M ax(0, J 0i)))i∈[1,K]. According
to Fig. 1, we have ∀J∈Sj:V(TNE , J )≤
V(TNE , J 0). Thus JNE =J0.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
−14
−12
−10
−8
−6
−4
−2
0
2
4
Jmax
Jmax
Ji
Ji
0
Jmax
0
0
0
Fig. 2. The assumption of Vi(TNE , Ji)with respect to Ji.
•α/(N+βJ max )≤C≤α/N:
let J0= ( 1
β(α
C−N))i∈[1,K], then U(T , J0)=0
∀T∈St. By using the concavity of V in J and setting
∂V (T ,J)
∂Jito zero, we have J0
i=1
β(qp0αβ
Dxi−1−N).
In order to have J0
i=J0
ifor i∈[1, K]we must
have x= 1 and p0=αD
βC 2, without loss of generality
we assume that αD
βC 2≤pmax
0. As result, we get,
TNE = ( αD
βC 2,1) and JNE = ( 1
β(α
C−N))i∈[1,K],
That means:
∀T∈St:U(T, JNE ) = U(TN E , J NE )=0.
∀J∈Sj:V(TNE , J )≤V(TN E , JNE ).
V. STAC KE LB ER G GAME
We assume the presence of a smart jammer. Since this kind
of jammer has the capability to sense the ongoing transmission
power, we model this problem as a Stackelberg Game denoted
as: Gs= ({Transmitter, Regular jammer}, {T, J}, {U, V}),
where the leader is the transmitter and the follower is the
jammer. Thus, the leader fixes its optimal strategy based on
the reaction of the follower, then the follower optimizes its
own utility according to the leader strategy, namely, we define
the Stackelberg Equilibrium by the following formulation:
TSE =Arg maxT∈StU(T , Arg maxJ∈SjV(T , J))
JSE =Arg maxJ∈SjV(TS E , J )(13)
A. Jammer’s Optimal Reaction
Assume that the two players have a complete information
about the environment.
Lemma 1: Let T be a given strategy of the transmit-
ter. There exists a unique J∗(T)such that J∗(T) =
Arg maxjV(T , j). In addition, the optimal jammer reaction
is given by:
Ji∈[1,K]
∗(T) = 




0E1
Jmax E2
1
β(qp0αβ
Dxi−1−N)ow
(14)
The conditions are given by:
•E1:xi−1<N2D
p0αβ
•E2:xi−1>(N+βJ max)2D
p0αβ
Proof: According to (4), V(T, .)is a continuous function
on the compact set Sjand it can achieve its maximum value at
some point J∈Sj. Since the first order partial derivative of
the jammer objective function with respect to Ji,∀i∈[1, K]
is: ∂V (T , J)
∂Ji
=αβp0xi−1
(N+βJi)2−D(15)
and the second order partial derivatives of the jammer
objective function are:
∂2V(T, J)
∂Ji∂Jj
=(−2αβ2p0xi−1
(N+βJi)3i=j
0ow (16)
Therefore, the Hessian matrix of V(T, J)with respect to
the vector Jis negative and V(T, J )is strictly concave in
J, [30]. Thus there exists a unique solution J∗(T)such that
J∗(T) = Arg maxJ∈SjV(T , J).
On the other hand, by resolving the following equation
∂V (T ,J)
∂Ji= 0, we have J0i=1
β(qp0αβ
Dxi−1−N). Since
0≤Ji≤Jmax. 1) If J0i> J max , yielding E2, then
V(T, J)increases in Sjand thus Ji∈[1,K]
∗(T) = Jmax.
2) If J0i<0, yielding E1, then V(T, J)decreases in Sj
yielding Ji∈[1,K]
∗(T)=0. 3) If 0≤J0i≤Jmax, yielding
E3, therefore, Ji∈[1,K]
∗(T) = J0i. Thus, we deduce the
property of the optimal jammer strategy given the strategy of
the transmitter given in lemma 1.
B. Stackelberg Equilibrium
Let’s now focus on analyzing the transmitter objective
function given the reaction of the jammer.
Theorem 2: There exists TSE ∈Stsuch that
(TSE , J ∗(TSE )) is a Stackelberg Equilibrium of the
Game.
Proof: To do so, we begin by proving the continuity of J∗
on St. It’s obvious that J∗is continuous in St\
K
S
i=1
{Sai, S bi},
where for each i∈[1, K],S aiis the set of couple (p0, x)∈St
such that p0xi−1=N2D
αβ , and Sbiis the set of couple
www.ijacsa.thesai.org 361 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
(p0, x)∈Stsuch that p0xi−1=(N+βJ max)2D
αβ . Let now
tai∈Saiand tbi∈Sbi, it’s clear from lemma 2 that
limT→t−
aiJ∗
i(T) = limT→t+
aiJ∗
i(T) = 0 ;limT→t−
bi
J∗
i(T) =
limT→t+
bi
J∗
i(T) = Jmax . The continuity of J∗on all Stis
proved. Since U(T, J )is a continuous function on St×Sj,
thus U(T, J∗(T)) is continuous in T.
Since the set Stis compact, U(T, J∗(T)) achieves its
maximum at some point TSE ∈St. This prove the existence
of TSE ∈Stsuch that (TS E , J∗(TSE )) is a Stackelberg
Equilibrium of the Game.
U(T,J∗(T)) is not a concave function:: Despite we
proved the existence of a SE (TSE , J∗(TSE )), calculating the
SE is a challenging due to the non-concavity of the function
U(T, J∗(T)). We use an example to show that there exists
(T1, T2, t)∈S2
t×]0,1[ where: U(t.T1+ (1 −t)T2, J ∗(t.T1+
(1 −t)T2)) <t.U(T1, J ∗(T1)) + (1 −t).U(T2, J ∗(T2))
Let N= 0.2; E= 1; C= 0.1; p01=p02= 2; k= 10; α=
β= 0.5; x1= 1.05, x2= 1.1, t = 0.63.
In this example we have, U(t.x1+ (1 −t)T2, J∗(t.T1+
(1 −t)T2)) = 13.7463,U(T1, J ∗(T1)) = 13.3062,
U(T2, J ∗(T2)) = 14.5018.
Hence, U(t.T1+ (1 −t)T2, J ∗(t.T1+ (1 −t)T2)) −
t.U(T1, J ∗(T1)) + (1 −t).U(T2, J ∗(T2)) = −0.0023. Thus
U(T, J∗(T)) is not a concave function on the set St. This
results proves the complexity of finding a closed form of the
global optimum, that’s why we propose a simulated annealing
technique as shown in Algorithm 2 in order to approximate
the global optimum of our given function U(T, J ∗(T)).
Algorithm 1 Calculate TSE =Arg maxT∈StU(T, J ∗(T))
Require: T∈St
Initialize the system parameters.
Initialize G with a large value.
T0=[0,1];
while (G6= 0)do
while (Accepted states number is below a threshold level)
do
Pick a random neighbor, T new ←neighbour(T)
δT =neighbour(T)−T new
δU =U(T new, J ∗(T new)) −U(T , J∗(T))
if δU > 0then
T←T new
else
T←T new +δ T. exp (δU/G)
end if
end while
G←G−1
end while
VI. AN TI -JAMMING WITH RE IN FORCEMENT LEARNING
Reinforcement Learning (RL) is considered as a method
in which the player takes action in a current time step and
receives the corresponding reward in the next time step to
evaluate its previous action [6]. RL is capable of solving
more complex problems, specially, as the player does not
require knowledge about the environment reaction and the
reward function. However, the player learns just from previous
experiences by interacting with the environment.
Through the above Game model, where both the transmitter
and the jammer have a complete information of each other
(i.e., channel gain and transmission cost), the SE strategies
are derived. However, in view of the fact that Neither the
jammer physical location nor its transmission cost is known by
the transmitter due to the assumption that firstly, the jammer
can change its physical location in a completely unforeseen
time; secondly, the value of the jammer’s transmission cost
is not shared over the channel. Consequently, we introduce
a reinforcement learning technique, especially the Q-learning
method, so that the transmitter can act autonomously in a dy-
namic environment without knowing the above Game model.
We assume that the transmitter can choose its power
multiplier and its initial transmission power from Mand
Nlevels respectively. Let Pand AMN denote the power
action taken by the transmitter and the set of power action
respectively. Meanwhile, the state observed by the transmitter
is denoted by stn. In each transmission cycle, the transmitter
and the jammer take actions sequentially, we denote by Jthe
jammer power action. At the beginning of the n-th transmission
cycle, the transmitter first takes action and the decision making
of its power action Pnis based on the transmission state in the
previous transmission cycle, i.e., stn= (Jn−1). sequentially,
based on the observed state sjn= (Pn), the jammer chooses
its optimal power Jngiven by (14). The received utility value
of the transmitter is denoted by un. Let now describe the anti-
jamming power control strategy based on Q-learning. Let αt
and βtdenote the learning rate and the discount factor of the
transmitter. The Q-function with the power action Pin the
state stis denoted by Q(st, P ). The maximum Q value in the
state stis denoted by V(st). We define the update rule of the
Q-function in the n-th transmission cycle as follows:
Q(stn, Pn)←Q(stn, Pn) + αt(un+βtVn+1 −Q(stn, Pn))
(17)
V(stn)←max
∀P∈AMN Q(stn, P )(18)
As a well-known reinforcement learning method, Q-
learning should try to balance between exploration and ex-
ploitation according to -greedy policy where the transmitter
chooses with a high probability 1-the power action that
maximizes the Q value in the state stwhile other power actions
are taken with an equal low probability 
MN −1. Thus, the
probability of power action xtaken by the transmitter is given
by the following formulation:
P r(P=x) = (1− x = argmax
∀P∈AMN
Q(stn, P )

MN −1ow (19)
Anti-Jamming Strategy of the transmitter with Q-learning
is shown in detail as Algorithm 2.
VII. SIMULATION RES ULTS
In this section, numerical results are performed to evaluate
the performance of the proposed power control problem during
www.ijacsa.thesai.org 362 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
Algorithm 2 Anti-Jamming Strategy of the transmitter with
Q-learning
Require: P∈St
Set the system parameters: βt,αt,, episode
Set st,AMN
Initialize Q(st, P ),V(st)as zero ∀st, P ∈AM N
while (episode 6= 0)do
Set the starting state s1
t
For n= 1,2,3, ..
Observe the current state sn
t
Pick a random power Pnfrom sn
taccording to Equation
(19).
if Pn= argmax
∀P∈AMN
Q(sn
t, P )then
Decrease 
end if
Observe the next state sn+1
tand un
Update Q(sn
t, Pn)by Equation (17).
Update V(sn
t)by Equation (18).
Break if convergence: small deviation on Q
episode ←episode −1
end while
a cycle in both scenarios: 1) Transmitter against smart jammer
(in which the jammer has the intelligence to quickly learn
the transmission power of the transmitter and adjust its own
transmission power). 2) Transmitter against regular jammer (in
which both players play the Game simultaneously in a non-
cooperative manner).
Among all the system variables, only fading channel gains
of the transmitter and the jammer, may vary significantly due
to the fact that the players can change their physical locations.
Thus, we investigate the relations of the utilities of all players
in equilibrium with respect to αand β. Let N= 1,D= 0.2,
C= 0.2and K= 10.
Fig. 3 shows the impact on the Utility function with
respect to αat NE and SE. We observe that, as αincreases,
transmitter’s SE utility increases while jammers’ SE utility
decreases; this phenomenon is due to the fact that the larger α
became, the closer the transmitter became from the receiver.
In addition, we depict in Fig. 4 the Utility function at NE
and SE of both players with respect to β. As we can remark,
the transmitter’s utility at the SE decreases with β, while the
jammer’s utility increases with it; this is due to the fact that
the larger βbecame, the closer the jammer became from the
receiver. Moreover, in both Fig. 3 and 4, the transmitter at
the NE has a lower utility than that at the SE, because at
the latter the transmitter knows the existence of a jammer and
utilizes its transmit power more efficiently. Similarly, a jammer
obtains a higher utility at the SE than that at the NE, due to
its ability to learn and adjust its own power according to the
ongoing transmission power. This results proves that despite
the jammer ability of sensing the active channel, the transmitter
can enhance its efficiency by predicting the jammer reaction
according to its own strategy.
Let now consider a scenario with power control strategy
based on Q-learning. In this simulation, we set M=N= 20
and we set the maximum episode numbers in the learning to
120 in order to ensure the transmitter can learn an optimal
action. The learning rate αt= 0.8which indicates how far the
current estimate value of Q is adjusted toward the update target
value of Q. The discount factor of the source βt= 0.8that
indicates the increasing uncertainty about rewards that will be
received in the future. We assume a transmitter that does not
have a complete knowledge about the dynamic environment,
while the jammer has these knowledge. The initialization of
the value for greedy algorithm is starting from 0.5 to ensure
that the transmitter can try all actions in all states repeatedly.
The utility of the transmitter received by the receiver according
to the learning episodes are shown in Fig. 5. We can remark
that the utility of the transmitter converges towards the solution
proved in the above model. This result validate the proposed
power control model. Note that, as the transmitter is gradually
aware of the dynamic environment with the learning episodes
increasing, which indicates a well anti-jamming performance.
This is due to the fact that the transmitter chooses a more
proper power action after has a well knowledge about the
environment.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
−40
−35
−30
−25
−20
−15
−10
−5
0
5
α
USE
VSE
UNE
VNE
Fig. 3. The impact of αon the utility function of Jammer/transmitter at NE
and SE. β=0.5.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
−80
−70
−60
−50
−40
−30
−20
−10
0
10
20
β
USE
VSE
UNE
VNE
Fig. 4. The impact of βon the utility function of Jammer/Transmitter at NE
and SE. α=0.5.
VIII. CONCLUSION
In this paper we studied denial of service vulnerability in
wireless networks in the presence of jamming attacks. We
choose a Game theoretical approach which is an abstract
concept that indicates how the final outcome of a competitive
www.ijacsa.thesai.org 363 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
0 20 40 60 80 100 120
−450
−400
−350
−300
−250
−200
−150
−100
−50
0
50
X: 88
Y: 2.5
Episode
Utility
Qlerning
SE
Fig. 5. Utility function of the transmitter, where the transmitter action is
chosen based on Q-learning.
situation is dictated by interactions among the players. We
considered a jamming during a transmission cycle. We studied
the case where all players take action simultaneously and the
case where the transmitter is the leader and the jammer is
the follower. We proposed a Nash Game in the simultaneous
Game and a Stackelberg Game in the hierarchical Game. A
closed form of Nash Equilibrium is derived, then, we proved
the existence of Stackelberg equilibrium. We sorted out the
Stackelberg problem by using a simulated annealing algorithm.
Moreover, we studied the relations of the utilities of all players
in Nash and Stackelberg equilibrium. In order to validate our
Stackelberg model, Q-learning method can is considered to
be used by the transmitter to determine their transmission
power actions in the presence of a smart jammer in a dynamic
environment without knowing the underlying Game model.
Simulation results have verified that despite the jammer ability
of sensing the active channel, the transmitter can enhance
its efficiency by predicting the jammer reaction according to
its own strategy. Finally, this work can be extended to the
case of several jammers that operate on a single sub-carrier
during a single time slot in order to investigate the interaction
among jammers who have interest to damage the source node
transmission.
REFERENCES
[1] L. Jia, F. Yao, Y. Sun, Y. Niu and Y. Zhu "Bayesian Stackelberg Game
for Anti-jamming Transmission with Incomplete Information." IEEE
Communications Letters, vol. 20, no. 10, p. 1991-1994, 2016.
[2] Lmater Moulay Abdellatif, Abdelillah Karouit, and Abdelkrim Haqiq.
"An efficient pricing mechanism of random access in wireless network
with self-interested mobile users." Wireless Networks and Mobile Com-
munications (WINCOM), 2015 International Conference on. IEEE, 2015.
[3] Guan, Yanpeng, and Xiaohua Ge. "Distributed secure estimation over
wireless sensor networks against random multichannel jamming attacks."
IEEE Access, vol. 5, p. 10858 - 10870, June 2017.
[4] Koh, Jing Yang, and Pengfei Zhang. "Localizing Wireless Jamming
Attacks with Minimal Network Resources." International Conference on
Security, Privacy and Anonymity in Computation, Communication and
Storage. Springer, vol. 10658, p. 322-334, 2017.
[5] Jaitly, Sunakshi, Harshit Malhotra, and Bharat Bhushan. "Security vul-
nerabilities and countermeasures against jamming attacks in Wireless
Sensor Networks: A survey." Computer, Communications and Electronics
(Comptelix), 2017 International Conference on. IEEE, 2017.
[6] Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine
learning 8.3-4 (1992): vol. 8, no. 3-4, p. 279â ˘
A¸S292, May 1992.
[7] Pelechrinis, Konstantinos, Marios Iliofotou, and Srikanth V. Krishna-
murthy. "Denial of service attacks in wireless networks: The case of
jammers." IEEE Communications surveys and tutorials 13.2, p. 245-257,
2011.
[8] Dhuria, Shivam, and Monika Sachdeva. "Detection and Prevention of
DDoS Attacks in Wireless Sensor Networks." Networking Communica-
tion and Data Knowledge Engineering. Springer, Singapore, vol. 3, p.
3-13, 2017.
[9] Arnbak, J. E. N. S. C., and Wim Van Blitterswijk. "Capacity of slotted
ALOHA in Rayleigh-fading channels." IEEE Journal on Selected Areas
in Communications 5.2, p. 261-269, 1987.
[10] Nguyen, Gam D., Anthony Ephremides, and Jeffrey E. Wieselthier.
"Comments on â ˘
AIJcapture and retransmission control in mobile ra-
dioâ ˘
A˙
I." IEEE Journal on Selected Areas in Communications 24.12, p.
2340-2341 ,2006.
[11] LaMaire, Richard O., Arvind Krishna, and Michele Zorzi. "On the
randomization of transmitter power levels to increase throughput in
multiple access radio systems." Wireless Networks 4.3, p. 263-277, 1998.
[12] Guan, Yanpeng, and Xiaohua Ge, "Distributed Attack Detection and
Secure Estimation of Networked Cyber-Physical Systems Against False
Data Injection Attacks and Jamming Attacks". IEEE Transactions on
Signal and Information Processing over Networks , vol. 4, no. 1, p. 48-
59, March 2018.
[13] Tayebi, Arash, Stevan Berber, and Akshya Swain. "Performance analy-
sis of chaotic DSSS-CDMA synchronization under jamming attack." Cir-
cuits, Systems, and Signal Processing , vol. 35, no. 12, p. 4350â ˘
A¸S4371,
December 2016.
[14] Kim, Yongchul, Young-Hyun Oh, and Jungho Kang. "Asynchronous
Channel-Hopping Scheme under Jamming Attacks." Mobile Information
Systems, vol. 98, no. 4, p. 3583â ˘
A¸S3610, February 2018.
[15] Wei, Xianglin, et al. "Collaborative mobile jammer tracking in multi-
hop wireless network." Future Generation Computer Systems, vol. 78,
no. 3, p. 1027-1039, January 2018.
[16] Li, Li, et al. "Security estimation under Denial-of-Service attack with
energy constraint." Neurocomputing , vol. 292, no. 31, p. 111-120, May
2018.
[17] Peng, Lianghong, et al. "Energy efficient jamming attack schedule
against remote state estimation in wireless cyber-physical systems" , vol.
272, no. 10, p. 571-583, January 2018.
[18] Sant, Jeetendra, and Vinod Sharma. "Performance analysis of a slotted-
ALOHA protocol on a capture channel with fading." Queueing Systems
34.1-4, p. 1-35, 2000.
[19] Manshaei, Mohammad Hossein, et al. "Performance analysis of the
IEEE 802.11 MAC and physical layer protocol." World of Wireless
Mobile and Multimedia Networks, 2005. WoWMoM 2005. Sixth IEEE
International Symposium on a. IEEE, 2005.
[20] Nyandoro, Alfandika, Lavy Libman, and Mahbub Hassan. "Service
differentiation in wireless lans based on capture." Global Telecommuni-
cations Conference, 2005. GLOBECOM’05. IEEE. Vol. 6. IEEE, 2005.
[21] Hadzi-Velkov, Zoran, and Boris Spasenovski. "Capture effect in IEEE
802.11 basic service area under influence of Rayleigh fading and near/far
effect." Personal, Indoor and Mobile Radio Communications, 2002. The
13th IEEE International Symposium on. Vol. 1. IEEE, 2002.
[22] X Ge, J Yang, H Gharavi, Y Sun, "Energy Efficiency Challenges of 5G
Small Cell Networks" in IEEE Communications Magazine, vol. 55, no.
5, pp. 184 - 191, May 2017.
www.ijacsa.thesai.org 364 |Page

(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 9, No. 4, 2018
[23] Pelechrinis K, Iliofotou M, Krishnamurthy SV, "Denial of Service At-
tacks in wireless networks: The Case of Jammers" IEEE Communications
Surveys and Tutorials , vol. 13, no. 2, pp. 245 - 257, May 2010.
[24] SD Amuru, HS Dhillon, RM Buehrer, "On Jamming Against wireless
networks" IEEE Transactions on Wireless Communications, vol. 16, no.
1, pp. 412 - 428, November 2016.
[25] C. Kaufman and R. Perlman, M. Speciner, "Network security: private
communication in a public world". Prentice Hall Press, 2002.
[26] L. Jia, et al., "A Hierarchical Learning Solution for Anti-Jamming
Stackelberg Game With Discrete Power Strategies" IEEE Wireless Com-
munications Letters , vol. 6, no. 6, pp. 818 - 821, August 2017.
[27] Y. Li, L. Xiao, J. Liu and Y. Tang "Power control stackelberg Game in
cooperative anti-jamming communications". In Int. Conf. Game Theory
for Networks (GameNETS), pp. 1-6, Nov. 2014.
[28] D. Yang, J. Zhang, X. Fang and A. Richa "Optimal transmission power
control in the presence of a smart jammer". In : Global Communications
Conference (GLOBECOM), 2012 IEEE, p. 5506-5511, December 2012.
[29] XIAO, Liang, LI, Yan, LIU, Jinliang, et al., "Power control with
reinforcement learning in cooperative cognitive radio networks against
jamming". The Journal of Supercomputing, vol. 71, no. 9, p. 3237-3257,
2015.
[30] Boyd, Stephen, and Lieven Vandenberghe. "Convex optimization".
Cambridge university press, 2004.
www.ijacsa.thesai.org 365 |Page