Content uploaded by Mohammad Abrar Shakil Sejan
Author content
All content in this area was uploaded by Mohammad Abrar Shakil Sejan on Jun 29, 2023
Content may be subject to copyright.
Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2023.0322000
HyDNN: A Hybrid Deep Learning Framework
Based Multiuser Uplink Channel Estimation and
Signal Detection for NOMA-OFDM System
MD HABIBUR RAHMAN1,2 , MOHAMMAD ABRAR SHAKIL SEJAN1,2, MD ABDUL AZIZ1,2 ,
YOUNG-HWAN YOU2,3, and HYOUNG-KYU SONG1,2
1Department of Information and Communication Engineering, Sejong University, Seoul 05006, Republic of Korea
2Department of Convergence Engineering for Intelligent Drone, Sejong University, Seoul 05006, Republic of Korea
3Department of Computer Engineering, Sejong University, Seoul 05006, Republic of Korea
Corresponding author: Hyoung-Kyu Song (songhk@sejong.ac.kr).
This work was supported in part by the ICT R&D Program of MSIT/IITP [IITP-2021-0-01816, A Research on Core Technology of
Autonomous Twins for Metaverse] and in part by the Basic Science Research Program through the National Research Foundation of Korea
(NRF) funded by the Ministry of Education (2020R1A6A1A03038540)
ABSTRACT Deep learning (DL) techniques can significantly improve successive interference cancellation
(SIC) performance for the non-orthogonal multiple access (NOMA) system. The NOMA-orthogonal fre-
quency division multiplexing (OFDM) system is considered in this paper to develop a hybrid deep neural
network (HyDNN) model for multiuser uplink channel estimation (CE) and signal detection (SD). The
proposed HyDNN uses a combination of a bi-directional long short-term memory (BiLSTM) network and a
one-dimensional convolutional neural network (1D-CNN) to optimize errors in the system. The extraction of
input signal characteristics from OFDM is carried out using the 1D-CNN model and fed into the time series
BiLSTM network to infer the signal at the receiver terminal. The HyDNN model learns through the simulated
channel data during offline training. To optimize the loss during learning the model the Adam optimizer is
utilized. After successful training, the transmitted symbols in the online deployment are instantly recovered
with optimal prediction rates by using the proposed HyDNN model. In comparison to the traditional CE
and SD method for the NOMA scheme and other existing DL models, the proposed technique demonstrates
satisfactory performance enhancements. In addition, the simulation outcomes show robustness with different
training parameters such as minibatch sizes and learning rates.
INDEX TERMS 1D-CNN, BiLSTM, Symbol error performance, Uplink NOMA, OFDM, multiuser signal
detection (SD).
I. INTRODUCTION
THE non-orthogonal multiple access (NOMA) scheme
has been acknowledged as an effective method for im-
proving spectral efficacy and system performance [1]–[4].
The power domain and code domain are two subtypes of
NOMA. According to the separation among base stations
(BSs) and all users (Us) in the NOMA, the power domain
might receive low or high transmission power allocations.
By allowing the simultaneous sharing of subcarriers between
Us with perfect channel conditions and those with imperfect
channel conditions, NOMA maximizes the usage of available
bandwidth. As NOMA systems combine signals from multi-
ple Us, inter-user interference must be canceled to decode the
signal reliably at the receiver terminal. The state-of-the-art
multiuser detection is performed at the detectors of NOMA
systems by successive interference cancellation (SIC) with
the variations in power domain among Us [5]. Information
from various Us is decoded successively in decreasing order
of signal power during the SIC operation depending on the
channel state information (CSI) [6]. It is challenging to ac-
quire CSI in NOMA due to interference from pilot symbols
used for channel estimation (CE). It may drastically reduce
the accuracy of conventional CE procedures, including maxi-
mum likelihood (ML), least squares (LS), and minimum mean
square errors (MMSE).
To overcome the limitations of the above methods, deep
learning (DL) has attracted with great contributions to the
field of wireless communications [7] such as CE [8], signal
detection (SD) [9], constellation design [10], and modulation
recognition [11]. Furthermore, DL applications also have
VOLUME 11, 2023 1
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
been investigated in 6G system [12], [13] like mmWave [14],
reconfigurable reflecting surfaces (RIS) [15] UAV commu-
nications [16], respectively. The authors in [17] developed
a unique codebook-based architecture for RIS-assisted com-
munications that successfully overcomes the issues of high
implementation complexity and significant pilot overhead.
Due to NOMA potential as a future wireless network
technology and the aforementioned uses of DL solutions,
current research on DL based SD in NOMA-involved systems
has gained attention [18]–[20]. Using a deep neural network
(DNN), a system for estimation of channel parameters of
orthogonal frequency-division multiplexing (OFDM) and de-
tecting signals were given in [21]. The authors demonstrated a
notable improvement of the performance in terms of symbol-
error rate (SER) analysis. In [22], a study on signal detection
employing the NOMA technique is presented, which em-
ployed DNN based fully-connected model. A similar neural
network structure was previously proposed in [21].
By utilizing effective parallel computing techniques, the
convolutional neural network (CNN) is able to extract better
fundamental properties underlying the channel matrix from
the vast amount of data and offers the ability to estimate
the channel more precisely with less complexity [23]. This
study, utilized 1 dimensional (1D)-CNN model due to the
following advantageous reasons over 2D-CNN [24]: (1) A
1D-CNN has much less time complexity than a 2D-CNN
under comparable circumstances (same design, network, and
hyperparameters), (2) the training hardware demand of 2D-
CNN is special configuration (such as cloud computing and
GPU) whereas 1D-CNN training can possible quite quick
using any CPU structure over a conventional computer, (3)
Compact 1D-CNNs are highly suited for real-time and cost-
effective implementation because of their low computing re-
quirements, particularly on mobile or handheld gadgets. A
CNN technique was proposed in [25] for NOMA system to
instantly decode input from numerous Us in a cluster without
the usage of conventional signal processing. In [26], the
authors proposed 2D-CNN long short-term memory (LSTM)
based CE for the NOMA-OFDM system where the proposed
technique is applied for the downlink NOMA scenarios and
for CNN model flatten layers was used for output vectors.
The proposed study achieved a marginal performance than
others methods. To solve the imperfect CSI and Us section
problems, in [27], authors proposed a CNN-LSTM based
downlink NOMA system. The proposed system analyzed the
outage probability with a focus on imperfect and perfect CSI.
A feed-forward NN extension known as a recurrent (RNN)
may accept input sequences of different lengths. RNNs have
the ability to remember the memory of past events and use
this data to forecast future values [28]. A form of RNN
called LSTM is built with a unique gating mechanism control
for accessing memory cells [29]. In [30] authors proposed
a three-stage joint channel decomposition and prediction
framework based on the two-timescale property and the chan-
nel prediction to get the CSI of the time-varying channels
in a RIS-assisted system. Additionally, create a new NN
structure termed sparse-connected-LSTM to perform channel
decomposition and prediction. For an OFDM-NOMA system,
LSTM-based CE and SD were also proposed in [20]. The
four-layered DL architecture, which consists of one input
layer, two LSTM hidden layers, and one output layer, was
assessed for bit-error-rate analysis. LSTM based CE and SD
are carried out for multiuser NOMA system in [31]. The
presented approach evaluated the results based on different
cyclic prefixes (CP) and pilot numbers. BiLSTM network is
also practical in sequence classification as the data flow in
both directions compared with LSTM. Another related study
in [32], proposed Volterra-aided CNN and LSTM for mitigat-
ing nonlinearity and recovering transmitted signals in visible
light communication channel. CNN is used for extracting the
implicit feature of channel impairments and LSTM is used for
memory sequence prediction. In contrast, our study focused
on 1D-CNN with BiLSTM structure for robust feature extrac-
tion for radio frequency NOMA communication system. The
motivation behind BiLSTM is that it offers additional training
capability as the output layers receive information from past
(backward) and future (forward) instances simultaneously
to provide better accuracy as compared to LSTM [33]. As
the flow of information in the BiLSTM network grows, the
architecture is able to extract more features from the input
data and improve the training capability [34]. According to
the experiment results from the study in [35], compared the
performance of LSTM and BiLSTM, where BiLSTM has a
greater capacity for feature extraction. Our previous study
in [36], proposed a CE and SD system based on a BiLSTM
network for achieving higher SER output performance. The
CE and SD are significantly impacted by the performance
in the aforementioned studies. In the NOMA-OFDM based
communication systems, the combination of 1D-CNN and
BiLSTM may be a potential method to attain high-accuracy
performance. Motivated by the above significant advantages
of 1D-CNN and BiLSTM and the research gap, in this study,
we propose a hybrid (HyDNN) for multiuser uplink CE and
SD in NOMA-OFDM systems. The HyDNN consists of a
1D-CNN and a BiLSTM where for assuming a large amount
of training data, the 1D-CNN model is added in front of
BiLSTM for extraction of channel features thus the combined
model improves the learning performance and enhanced the
SER. The proposed HyDNN-based receiver solves the SER
performance problems of conventional methods more effi-
ciently.
The following is a summary of the study key contributions:
•The HyDNN framework is used in this study to develop
the uplink NOMA-OFDM CE and SD system over the
Rayleigh fading channel. The NOMA-OFDM problem
is solved by the design of the combined 1D-CNN fea-
ture extractor and BiLSTM layer. The 1D-CNN model
is performed to extract features of the received signal
and information to the BiLSTM model is fed for the
inference of the original signal at the receiver terminals.
•We segmented the data of the received signals using
2VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
a 1D-CNN based feature extractor, taking into account
inter-carrier interference (ICI) and inter-symbol interfer-
ence (ISI). The BiLSTM layers are offered for handling
the significant ISI generated by the multipath effect and
due to the bi-directional structure of BiLSTM, it pro-
vides additional training capability for learning the 1D-
CNN features with forward and backward directions.
Therefore, the output layers receive more feature infor-
mation for successful signal demodulation and provide
better prediction accuracy.
•After offline training, the proposed model can be applied
to a multi-user NOMA-OFDM system for online predic-
tion. The concept is validated by employing two users in
the simulation system.
•Finally, the effectiveness of HyDNN is performed
by calculating SER at different signal-to-noise ratios
(SNRs) using the Monte Carlo simulation. The simu-
lation output is compared by learning the model with
different minibatch sizes and learning rates for perfor-
mance analysis. Simulation results have proved that the
efficiency of the proposed method is comparable to the
conventional NOMA-SIC method outage performance.
Moreover, the proposed HyDNN model outperforms the
CNN and BiLSTM models, respectively.
The rest of this article is structured as follows: Section II
describes the system’s data transmission and channel con-
cept, while Section III explains the specifics of the proposed
HyDNN model. Section IV presents the simulation findings
and complexity. The finding summary is finally presented in
Section V.
Notations: The boldface letter in lower case and upper
case, respectively, stands for a vector and matrix; The ith
element of the vector xis represented by the subscript on the
lowercase letter xi;(·)H,(·)−1,⊗is the Hermitian transpose,
inverse, hadamard product, respectively; Erepresents the
statistical expectation.
II. DATA TRANSMISSION AND SYSTEM CHANNEL MODEL
High spectral efficiency in wireless communications is pro-
vided by the multi-carrier modulation technology known as
OFDM. Binary inputs are transformed into phase shift keying
modulation for mapping to Dparallel data streams as part of
the modulation process for OFDM. Let Xa[q]represent the
qth subcarrier’s ath transmit symbol where a= 0,1,2, ..., ∞
and q= 0,1,2, ..., Ns −1. Transforming signals from the
frequency domain to the time domain is done using the in-
verse fast Fourier transform (IFFT). To avoid interference
between symbols, a cyclic prefix (CP) is introduced to the
signal. Therefore, the transmitted symbol can be thought of
as follows:
Xa(n) =
Ns−1
X
q=0
Xa[q]ej(2πqn/Ns)n= 0,1,2. . . , Ns−1
(1)
where Nsis the FFT length.
FIGURE 1. The system architecture of NOMA multiuser data transmission
from the Us to BS.
In this paper, an uplink NOMA system is assumed. The
multiuser uplink NOMA system, which comprises of a BS
and two Us (Uu,u= 1,2, ....N), is depicted in Fig. 1. In this
system, it is assumed that all the nodes are constructed with
individual antenna and both Us transmit data at the same time
with identical frequency resources. The BS receives a super-
position of data symbols from two Us with channel noise from
the transmitter, which employs a traditional NOMA-OFDM
scheme. The power allocation is done with the assumption
that the transmitter and receiver are aware of CSI. Moreover,
for the estimation and detection of the channel, a pilot symbol
is inserted into the system. On the other hand, the multiuser
side employs the HyDNN approach for flexible CE and SD.
For the NOMA system, the superposition of data symbols
for the NUs can be expressed as follows [22]:
y=
N
X
u=1
√PuXu,(2)
where Puand Xuare defined as the power allocation and the
transmitted baseband symbol for Uu.
If the OFDM system is formatted with K-subcarriers and
has NUs, the following is an expression for the received
signal on subcarriers k:
y(k) =
N
X
u=1
√Pu(k)hu(k)Xu(k),(3)
where the FFT of the impulse response of a multipath channel
and the received frequency-domain signal respectively, are
hu(k)and y(k).Xu(k)is a representation of Uu’s transmitted
symbol.
For the ksubcarriers, at the receiver, the white additive
Gaussian noise (AWGN) W(k)is denoted by the symbol
VOLUME 11, 2023 3
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
CN(0, σ2). Following the addition of the AWGN, the re-
ceived signal may be written as follows:
y(k) =
N
X
u=1
√Pu(k)hu(k)Xu(k) + W(k).(4)
The power allocation Pufor the uUs with ksubcarriers is
rewritten as Pu(k). However, the total power Pwis allotted to
each of the ksubcarriers in the OFDM. The following is an
expression for the Uu’s power allocation coefficient:
δu(k) = Pu(k)
Pw
.(5)
The following formulation can be used to express the con-
straint of equation (5) [26]:
N
X
u=1
δu(k)=1.(6)
Therefore, the impulse response of a multipath channel
FFT hu(k)for the Uuis stated as follows [37]:
Hu(t) =
M
X
m=1
υu,mη(t−τu,m),(7)
where υu,mand τu,m, respectively, stand in for the complex
channel gain and associated time delay for the Us’ mth mul-
tipath parameters. The channel is represented by Rayleigh
fading in the proposed work, where the total number of
determined pathways Mis taken into account to be 20.
The signal is estimated and detected by the CSI using
conventional SIC techniques like LS and MMSE [38]. Ad-
ditionally, ML detectors are utilized to predict signals since
Uusignals are given greater power [39]. For the uplink CE of
LS and MMSE, pilot data transmission is employed. So the
conventional LS CE of (4) can be expressed as follows [38]:
ˆ
hLS =yp
Xp
,(8)
where Xpis the transmitted pilot sequences P=p1,p2, ...pN
and ypis the received pilot data for estimation of channel pa-
rameters. In addition for estimation of MMSE, the correction
coefficient RhhLS is calculated. The estimation of MMSE can
be formulated as follows [38]:
ˆ
hMMSEu =RhhLS R−1
hLS hLS
ˆ
hLS
=Rhh Rhh +σ2
s(XpXpH)−1−1ˆ
hLSu,(9)
where the signal is transmitted from the uth transmit antenna,
ˆ
hMMSEu is the MMSE estimated for the channel, ˆ
hLSu is the LS
estimation of the uth transmit antenna, and AWGN channel
noise variance is σs2.
These covariance matrices can be expressed as follows:
Rhh =E{hhH},(10)
RhhLS =E{hˆ
hLS
H},(11)
RhLS hLS =E{ˆ
hLS ˆ
hLS
H},(12)
FIGURE 2. Block-type pilot signal insertion structure.
where (10) is defined as channel autocorrelation matrix
of frequency-domain with expectation operator E, (11) is
the cross-correlation among the actual channel and predicted
channel which is estimated via LS estimator with the size of
FFT ×pilot, P. MMSE estimator can increase the accuracy
of CE because it considers the impact of noise and it needs the
prior information on channel characteristics which enhances
the computational complexity compared to LS. However,
each U transmits a pilot symbol P, to the BS, and this pilot is
used for the CE. The second U signal, y′
2(k), can be calculated
after the first U signal has been estimated which can be
formulated as follows:
y2
′(k) = y(k)−pP1ˆ
h(k)ˆ
X1(k).(13)
A. INSERTION OF PILOT DATA TO THE OFDM
The pilot signal is known as symbols which are inserted on
OFDM subcarriers to get the information for the channel
response. The operation of CE and SD is done based on
these pilot responses. Block-type pilot insertion strategy that
is most widely utilized and adopted [40]. Fig. 2 shows the
design of a block-type insertion of the pilot, where the green
and white squares indicate the value of the pilot signal and
data signal, respectively.
The pilot and data symbols of block-type are individually
inserted between each pair of subsequent OFDM signals. Ei-
ther just pilot or only data symbols are contained in the single
OFDM signal. Additionally, as shown by the obtained pilot
and data responses, the signal is detected at the conclusion of
two successive OFDM symbols [41].
III. PROPOSED HYDNN MODEL
In this paper, the proposed HyDNN network is formed by
combining 1D-CNN and BiLSTM models. The 1D-CNN
model output is cascaded with the BiLSTM model (i.e, the
input of the BiLSTM model is the model output of 1D-CNN).
The receiver objective is to retrieve the sent symbols for mul-
tiuser usage according to the presented model at the BS. The
proposed network is trained with the channel characteristics
4VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
using the OFDM simulation data which are generated with a
certain channel profile.
FIGURE 3. Training input dataset of the proposed model.
A. DATASET PREPARATION
To ensure optimal CSI and SD performance, data generation
and model construction of the DL network are very crucial
points. In this subsection, the dataset generation procedure is
discussed.
In this paper, the subcarrier length is 64 of the OFDM
system for the generation of the training dataset is consid-
ered. The data is transmitted from an OFDM packet and 1
OFDM packet contains 3OFDM symbols such as 1data
stream (ydu1,ydu2,ydu3, .....yduM ) and 2pilot data symbols
(ypu1,ypu2,ypu3, .....ypuM ), (y′
pu1,y′
pu2,y′
pu3, .....y′
puM ), respec-
tively as shown in Fig. 3. In the multiuser case, the first 2
OFDM symbols are generated by each U as 2pilot sequence,
and the third OFDM symbol occupies the transmitted 1data
symbol. The previous section II-A provides a description of
the specifics of pilot data insertion. Considered is the quadra-
ture phase shift-keying (QPSK) modulation, which uses 2
bits per subcarrier for each symbol. In order to create an
OFDM packet with fixed pilot sequences, QPSK random
data symbols are used in the training data preparation. The
Rayleigh fading channel is used to send the OFDM packets
to the receiving end. The BS on the receiving end receives
the combined OFDM packet from all Us together with extra
AWGN noise in order to decode the OFDM packets.
By constructing a feature vector called F, the received
OFDM packet is saved as a sample for the training data set.
The real, Re, and imaginary, Imvalues of each symbol in the
OFDM packet are combined to form the feature vector F.
The amount of the training sample is equal to the product of
the total number of data packets (T) and the total number of
labels (Nl). The proposed HyDNN network may be taught
to recover data on any subcarrier kby employing the cor-
responding L(k)in the training process. Multiuser transmis-
sion symbols are assigned to a single integer value label for
classification. There are a total of 24combinations or labels
Algorithm 1 Training Data Generation Process
1: Initialize data: Tis total packet, Nis number of OFDM
subcarriers, Nsc is pilot subcarrier, Mcp is the CP length,
EdB is SNR value
2: for each EdB value in EdB
3: for n=1:T, generate training data for each class, Nl.
4: Random generation of Rayleigh channel coefficients and
AWGN Wnoise.
5: Generate fixed pilot symbols and insert them to compose
the pilot-OFDM symbol.
6: Randomly generate Nlog2Mu,for Us u= 1,2, ...Nbits
and map them Xuby Mu-ary modulation. Then obtain y
symbols according to Eq. (2).
7: According to Eq. (4), calculate the received signal y
transmission through Rayleigh channel coefficients in step
4, and based on the y, obtain the (YRe
du ,YIm
du ,YRe
pu ,YIm
pu )
8: end for
9: end for
10: Outputs (YRe
du ,YIm
du ,YRe
pu ,YIm
pu )
11: Save the generated data
available in the system for Us transmitting QPSK symbols.
However, for Nl= 16, the entire label may be written as
L(k)=1,2,3,4.....Nl.1OFDM packet has three OFDM
symbols, 2active Us, and a total input size of 384 as the
64 subcarriers are taken into account. Data samples totaling
50000 ×Nl= 800000 are created for training, using 50000
data packets. The whole generated data sample is divided into
two portions, such as train and validation data size, in order
to create the model as effectively as possible. The size of the
training data sample is (4/5), or 640000, while the validation
data sample is (1/5), or 160000. The process of training data
generation is summarized in the Algorithm 1.
B. 1D CONVOLUTIONAL NEURAL NETWORK
DESCRIPTION
The proposed 1D-CNN network is depicted on the left side
in Fig. 4. 1D-CNN architecture model for extraction of signal
features is used. The input layer is fed into an OFDM data
symbol with an input dimension equal to the quantity of fea-
tures in the input. Convolutional 1D, ReLU, and normalizing
layers are the following layers. Every neuron in the convolu-
tional layer gets input characteristics from a rectangular area
of the previous layer, resulting in a rectangular grid of neurons
in this layer. In order to condense the output of the convolu-
tional layer into a single vector, a global average pooling 1D
layer is utilized last. The goal of the 1D-CNN network is to
local feature extraction and through the sharing parameters
reduce the number of weights. Thus, the calculation of the
overall network is effectively reduced by 1D-CNN.
C. BILSTM NETWORK DESCRIPTION
On the right side of Fig. 4 the BiLSTM model framework
is illustrated. The motivation for proposing the BiLSTM
VOLUME 11, 2023 5
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
FIGURE 4. The proposed HyDNN model architecture connection of its different layers: the 1D-CNN network structure is on the (left side) and the BiLSTM
network structure is on the (right side).
FIGURE 5. Internal cell structure of one layer: (a) LSTM; (b) BiLSTM.
network is as follows. The unidirectional LSTM networks
perform sequences in the past, without considering the future.
This is due to the fact that unidirectional LSTM only retains
information from earlier time steps since it only receives
input from the past [42]. On the other hand, the BiLSTM
is comprised of the forward (one from past to future) and
backward (one from future to past) directions of the uni-
directional LSTM. The input flows both ways, allowing it
to utilize both sides of the information, offering additional
training feature extractions and improving the prediction per-
formance. Fig. 5(a) and (b) display a schematic representation
of the internal cell architecture of the unidirectional LSTM
and BiLSTM, respectively. BiLSTM hidden, fully connected,
softmax function and classification layers are among the 4
layers that make up the proposed BiLSTM model. There
are 100 hidden units used to implement the BiLSTM hidden
layer. The fully connected layer is executed with a 16 number
of classes. The outputs for the end layer are obtained by using
the softmax function. The classification layer is used in the
final layer to convert the output to a vector probability.
D. MATHEMATICAL OPERATION OF CNN-BILSTM
Input layer: The input for 1D-CNN is comprised of the
sequence of real and imaginary values with corresponding
labels. Let the input sequence features matrix of F=[S1Re,
S1Im,S2Re,S2Im, ..., SNRe,SNIm], which is composed of two nu-
merical values together and their label classes. The ith output
of the sequence matrix is Si, so that S0=F. Each sequence
contained 384 real, Re, and imaginary, Imvalues and which
represent the label, Nl. The dimension of 384 ×1is assumed
to represent the size of the input features in the input layer.
The input layer of the CNN is fed the two numerical values
of sequence features with its label from the generated dataset,
where there are the same number of features in the input data
as the input size.
Convolutional 1D layer (i=1): The convolutional layer
function is fed to the sequence inputs and extracts local
features from it. A group of learnable filters makes up the
parameters for the convolutional layer (k×s, where kdenotes
the kernel size and sdenotes the dimension of the input data).
A total of 32 filters in 3×3different sizes are used in the
convolutional layer for the feature extraction. Accordingly,
the output feature matrix Sican be expressed as follows:
Si=f(Si−1⊗wi+bi),(14)
where wiis denotes weight for ith layers and biis denotes the
bias for ith layers. The ReLU (rectified linear unit) activation
function is used in the next layers which is a non-linear
or piece-wise linear function. The ReLU function output is
directly input if it is positive, else, the output will be 0. The
mathematical formulation of the ReLU function is as follows:
f(x) = max(0,x).
1-D global average pooling layer (i=2): To minimize the
information dimension and the likelihood of network over-
fitting, the pooling layer is primarily utilized to compress
the features that the convolutional layer has extracted. A 1-D
global average pooling method is used in this study where the
main role of this layer is the reduction of sequence features
computational time from prior hidden layers. The maximum
of the prior features matrix is produced by the pooling layer.
As a result, the output feature matrix Sican be written as
6VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
follows:
Si=fp(Si−1)(15)
where the pooling function is fp. The dimension of S2is m/z×
nwhich is obtained from the pooling layer, where zstands
in for the current layer’s scale value in the pooling layer, the
input data time steps are denoted by m, and the quantity of
filters is defined by n.
BiLSTM layer (i=3): The input gate, output gate, and forget
gate make up a BiLSTM cell. Each time slot sequence into
and out of the cell is controlled by these three gates. As the
BiLSTM network flows the information in both directions,
more training features from both directions are recorded for
mapping Us transmitting in the successive time slot. The
mathematical formulation of BiLSTM is as follows. For input
signal S2at the current time step t, the calculation of BiLSTM
layers in both directional flows can be expressed as follows:
−→
h3f=σ(W3fS2t+W3fh3t−1+b3f),(16)
←−
h3r=σ(W3rS2t+W3rh3t+1 +b3r),(17)
where σis the activation function, the time steps of forward
and backward represent t−1and t+ 1, the hidden state
of previous and next are h3t−1and h3t+1, respectively, the
weights and learnable bias of both directions is W3fand W3r
and b3fand b3r, respectively, and finally, the forward and
backward direction LSTM network outputs are −→
h3fand ←−
h3r
respectively. Therefore, the output S3of BiLSTM can be
formulated as follows:
S3=σ(WS3−→
h3f⊕WS3←−
h3r+bS3)
=σ(WS3˜
h3t+bS3),(18)
where the output weights of the BiLSTM network are WS3, the
learnable parameter of BiLSTM output bias is bS3, the con-
centration of the hidden state at both directions of BiLSTM
is ˜
h3t.
Fully connected layer (i=4): The fully connected layer is
very important and its work is to classify whereas a method
of activation, the softmax is chosen. Particularly, the final
classification is performed by the fully connected layer. Here,
the model estimates the probability that each sample presently
belongs to each class and then derives a features expression
(Ypred ) that can be stated as follows:
Ypred (i) = f(L=li|S3; (W,b)) (19)
where the features from the BiLSTM layer are S3and the soft-
max activation function is f(·).lirepresents the calculation
output of the ith classes of the input data, and the weight and
bias value are represented by Wand b, respectively.
The cross-entropy operation calculates the cross-entropy
loss for single-label and multi-label classification tasks be-
tween network predictions and target values. When working
with models of output probabilities, cross-entropy is typically
the best option. Additionally, L2regularization can be seen as
a successful compromise between locating small weights and
lowering the cost function [43]. Cross-entropy and L2reg-
ularization are therefore used to avoid overfitting. Reducing
the computation of loss is the goal of training the model that
is formulated as follows:
Loss(W,b) = −
ns
X
i=1
c
X
t=1
(Y(t)(i)∗log(Y(t)
pred (i))+ λ
2
ns
X
i=1
W2
i
(20)
where (Y(t)(i)stands for the prospect of the known goal,
(Y(t)
pred (i)) is the probability that the ith sample belongs to the
tth class, the sample numbers is ns, the class size is c, and
finally, λis used to define the regularization coefficient of
L2. The Adam optimization technique is applied to reduce
the loss [44].
E. HYDNN MODEL TRAINING AND TESTING PROCESS
Based on the design and data generation of the proposed DL
network architecture, the training procedure is done in the
offline. The offline training of the proposed HyDNN model
is shown in the upper part of Fig. 6. The generated dataset is
split into two portions: training and validation for learning of
the model. According to Fig. 6 in the upper part, the proposed
HyDNN model is loaded as sequential inputs of the training
and validation data to the input layer of the 1D CNN model
and the corresponding labels as supervised information. The
dimension of sequence input of CNN is 384 ×1.
The input layer is fed into these sequential data with label
values together. The vector of input features is then learned
by the convolutional 1D layer with the learnable parameters
(weights=3×384 ×32 and basis=1×32). Then the learnable
CNN is fed into BiLSTM input. The total hidden units of
BiLSTM are summed up forward and backward which is
100 + 100 = 200, where the weights of input dimension are
800 ×32, weights of recurrent dimension are 800 ×100, and
bias is 800 ×1. The fully connected layer is then classified,
and the softmax is used as an activation function. The output
dimension of the model is 16 ×1. Thus, the operation of the
HyDNN model is done and it is called for training as HyDNN
described in Algorithm 2.
Table 1 shows the training and optimized parameters for
training setting the HyDNN. The training and validation per-
formance is done with a total of 100 epochs to learn the
model. Furthermore, it has been found that the learning rate
performance of the proposed model is extremely robust in
relation to the iteration and epoch values. The testing method
begins in the online stage once the proposed HyDNN model
has undergone consecutive training. With the training settings
of 500 for the minibatch size and 0.01 for the learning rate,
the validation accuracy is attained at a rate of 99.93. The
lower part of Fig. 6 shows the online testing process that
the testing datasets test the trained HyDNN model where the
channel characteristics are tested as input to match the true
and predicted values for optimal prediction calculation.
VOLUME 11, 2023 7
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
FIGURE 6. Training and testing procedure of the proposed HyDNN model: the training mechanism in the offline stage (upper part); the testing mechanism
in the online stage (bottom part).
TABLE 1. The simulation parameters of proposed HyDNN model.
Parameters Value
Subcarriers of OFDM 64
Number of Pilots 64
Multipath numbers 20
Channel Noise AWGN
Number of cyclic prefixes (CP) 20
fading channel Rayleigh channel
U numbers 2
Modulation scheme QPSK
Numbers of total packets 50,000
Numbers of layers in model 9
Total Epochs 100
Fully connected layer 16
Hidden state units 100
Learning rate (LR) 0.01, 0.001
Size of minibatch 500, 1000, 2000, 5000, 8000
Optimizer Adam
In the online deployment process, for evaluating the system
performance, the trained model is loaded and initialized with
the parameters of testing data of the NOMA-OFDM signal
over the Rayleigh fading channel. Then the SER simulation
results of the proposed model are obtained in different SNR
values. The details of the situation evaluations are represented
in section IV. The training and testing procedure for the
proposed network overview is provided on Algorithm 2.
IV. SIMULATION PERFORMANCE EVALUATION
In this paper, the simulation work is conducted on the Win-
dows 10 Pro operating system. The program is performed
with MATLAB. The DL network formation is performed
by interconnecting DL layers which are provided in the DL
ToolboxTM. The DL ToolboxTM also provides Us with the
creation of DL models and monitors the training process. An
NVIDIA graphic card is utilized to enhance training perfor-
mance. The simulation outcomes are done using the various
simulation settings listed in Table 1 for the proposed HyDNN-
based multiuser CE and SD. The training data information
Algorithm 2 HyDNN Training and Testing Process
Training procedure:
1: Load the data symbols: assuming (YRe
du ,YIm
du ,YRe
pu ,YIm
pu )
2: split the data samples into a training and validation set
at a ratio of 80% and 20%.
3: Passing processed data to build the model and configure
the layers.
4: Setting the model parameters like learning rate, maxi-
mum epochs, and minibatch size.
5: Loss function calculation by (20).
6: Using the Adam optimization technique, find the best
solution while updating the parameters and computing the
correction parameter.
7: Save the model.
8: Output: HyDNN model.
Testing procedure:
9: Load the trained HyDNN model.
10: Initialize the all parameters.
11: for e: 1 to Iteration do:
12: for n: 1 to SNR do:
13: Data symbols transmit through channel matrix.
14: Received data symbols.
15: Generate test label classes from received data symbols
16: Match the label classes with the trained model to clas-
sify.
17: SER performance with different SNR.
18: end for
19: end for
20: Output: SER results.
in the previous section III-A was explained. For generating
the training dataset, the SNR value is considered at 30 dB in
the offline stage. Evaluation of the simulation results with an
SNR range of [0: 2: 30] dB is done in order to test the trained
model in the online stage. To infer the model and obtain the
8VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
FIGURE 7. Different hyperparameters tuning effect of HyDNN and CNN models with; (a) Epoch=100, minibatch (Mb) size=500 and learning rate=0.01 and
0.001, (b) Epoch=100, minibatch (Mb) size=1000 and learning rate=0.01 and 0.001, (c) Epoch=100, minibatch (Mb) size=2000 and learning rate=0.01 and
0.001,(d) Epoch=100, minibatch (Mb) size=5000 and learning rate=0.01 and 0.001, (e) Epoch=100, minibatch (Mb) size=8000 and learning rate=0.01 and
0.001
optimal results, 3000 packets for testing the trained model are
used. For evaluating the SER performance of the proposed
HyDNN network, Monte Carlo simulations are provided. 64
pilots in each transmitted package, and a CP size of 20 during
training and testing are utilized to compare the proposed
model with traditional MMSE, LS, and ML methods and
CNN and BiLSTM models.
To illustrate the system performance, the proposed network
with conventional SIC approaches such as MMSE, LS, ML,
as well CNN, and BiLSTM models are used. In this proposed
system, two different Us are considered. As a result of the ICI
and ISI not being totally eliminated, the MMSE-SIC findings
are not ideal for the distorted practical settings [22].
A. PERFORMANCE EVALUATION
To get the optimal prediction performance, the tuning of
hyperparameters is very important at the time of learning the
model. To get the best-learned model, the proposed model
is trained with a variety of hyperparameters. Figure 7(a)
and (b) shows the hyperparameters comparison of HyDNN
and CNN models with loss function versus different epochs
value of 100, minibatch=500,1000 and learning rates=0.01
and 0.001, respectively. In addition, Figure 7(c), (d), and
(e) shows the hyperparameters comparison of HyDNN and
CNN models with loss function versus different epochs value
of 100, minibatch=2000,5000,8000 and learning rates=0.01
and 0.001, respectively. The loss function of these results is
taken under SNR 30 dB. It is seen from the above graph, by
increasing the learning rate and minibatch size, the perfor-
mance is achieved better. With the decrease in learning rate
and increased minibatch, the convergence time is decreased.
However, the loss value is increased which caused the degra-
dation of CE estimation performance. For getting better CE
estimation and avoiding over-fitting risk, it is kept learning
rates=0.01 and 0.001 and minibatch=500.
The simulation results are conducted by comparison of
different traditional signal estimation and detection schemes.
The results of a simulation using a confusion matrix show
how robust the model symbol categorization is during test-
ing. Fig. 8 illustrates the confusion matrix and correlation
for symbol categorization based on the number of labels.
In Fig. 8(a) and (b), the confusion matrix results for the
minibatch size 500 with the learning rates of 0.01 and 0.001
are shown, respectively. In contrast, correlation results of true
and predicted values for minibatch size 500 with the learning
rates of 0.01 and 0.001, are shown in Fig. 8(c) and (d). The
proposed HyDNN model has a very high symbol decoding
and classification rate, which grounds true class and predicted
class matching with the exception of a few small missing
classes.
FIGURE 8. (a), (b): Confusion matrix results for multi class symbol
classification according to true class and predicted class at minibatch 500
with a learning rate of 0.01 and 0.001. (c), (d): Correlation of true and
predicted results for multi classes symbol classification according to true
class and predicted class at minibatch 500 with a learning rate of 0.01
and 0.001.
In Fig. 9(a), (b), (c), (d), and (e) the SER performances
of the HyDNN model are compared with the conventional
(MMSE, LS)-SIC, ML method, CNN and BiLSTM model
for U-1 and U-2 are shown respectively. With a minibatch
size of 500 and a learning rate of 0.01, the proposed model is
VOLUME 11, 2023 9
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
FIGURE 9. SER achievement of the proposed HyDNN model with two Us for NOMA-OFDM system; (a), (b), (c) and (d) are SNR versus SER achievement for
the proposed model and traditional (MMSE, LS, ML) methods, CNN model at minibatch (Mb) size 500, and learning rate 0.01 during training.
FIGURE 10. SER achievement of the proposed HyDNN model with two Us for NOMA-OFDM system; (a), (b), and (c) (d) are SNR versus SER achievement for
the proposed model and traditional (MMSE, LS, ML) methods, CNN model at minibatch (Mb) size 500, and learning rate 0.001 during training.
compared against existing approaches. The proposed model
is compared to the ML approach with the ideal scenario of
perfect CSI circumstances. According to Fig. 9(a) and (b),
the proposed model achieved 16 dB SNR whereas the MMSE
and LS achieved 20 dB SNR. In addition, in Fig. 9(c), the
ML also achieved less performance than the HyDNN model.
The performance of the proposed HYDNN is also evaluated
in comparison to that of the CNN and BiLTSM models to
determine its efficacy. It is observed that the CNN model
achieved 22 dB SNR and the proposed model gained 6dB
SNR more than the CNN model. It is also seen that the
proposed HyDNN model outperforms the BiLSTM model as
well.
On the other hand, Fig. 10(a), (b), (c), (d), and (e) depict
the SER performance of the proposed model with MMSE,
LS, ML, CNN, and BiLSTM at the minibatch size of 500
and a learning rate of 0.001 for both Us. Additionally, it is
stated that the proposed model consistently outperforms CNN
and BiLSTM models as well as conventional approaches in
terms of SER performance. However, Fig. 10(c) shows that
the SER accuracy of the ML methods is a little higher than
the proposed network at the end of the curve with high SNR
for U-2. In Fig. 10(c), the ML performance is higher after
the SNR of 28 dB for U-2. This small degradation happens
due to the low learning rates of the model and the ideal case
of the ML technique. With the lower learning rate and same
minibatch size, the performance of the proposed model is
achieved 18 dB SNR which is 2dB degradation than the
learning rate of 0.01. However, though the performance of the
proposed model is a little degraded with less learning rate,
still the SER performance is higher than all other methods
as well as the CNN and BiLSTM models. From the above
discussion, it can be stated that the proposed HyDNN-based
receivers can handle the ICI and the ISI using 1D-CNN and
BiLSTM based networks very effectively. It is also observed
that information about the interconnections among subcarri-
ers by the convolution for the input sequence of the proposed
model is also possible to extract. With the exception of a little
deterioration, the proposed detection network outperforms all
of the approaches in the overall cases as shown in Fig. 11(a)
and (b).
In order to examine the learning capacity of the proposed
HyDNN model, the simulation outcomes are also done in
terms of testing precision utilizing SNR (0 −30) dB values
of the final Monte Carlo simulations. Table 2 provides an
illustration of the testing accuracy findings using various SNR
settings. The proposed model is trained with the minibatch
size 500,1000,2000,5000, and 8000 is considered by the
learning rate of 0.01 and 0.001. The evaluation of the pro-
posed HyDNN model performance for the SER using SNR
values demonstrated that testing accuracy varies very little
with varied minibatch and learning rates. However, the aver-
age variation of accuracy is not much different with different
parameter settings. The testing accuracy for different SNR
values is 99.93 and 99.56 percent for 500 minibatch and
learning rates of 0.01 and 0.001, respectively. The achieved
accuracy with simulation parameters is showing best among
all of the outcomes which shows the robustness of the infer-
10 VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
FIGURE 11. Overview of SER performance of the proposed HyDNN model
with two Us for NOMA-OFDM system; (a) with learning rate 0.01, (b) with
learning rate 0.001.
ence capacity of the proposed model.
FIGURE 12. Comparison results of the proposed system in terms of SER
achievement and different SNR values.
The simulation results of various schemes are shown in
Fig. 12, where the curves show the effectiveness for chan-
nel estimation and signal detection of the proposed HyDNN
model, DNN in [21], DNN in [45] and CNN-LSTM in [26]
for U-1 and U-2, respectively. According to the figure, the
proposed HyDNN model has a similar trend with [45] and
provides better performance as compared to other models for
U-2 in terms of SER. In the case of U-1, the proposed method
outperforms the existing models by 10dB SNR gain. The SER
versus SNR graphs demonstrate that the performance of the
FIGURE 13. Comparison of the proposed system results regarding SER
achievement and different SNR values with different system
configurations.
proposed HyDNN network with the other existing approaches
is relatively higher for U-1 with the ranges of lower SNR
values.
Figure 13 shows the comparison of the proposed system
results regarding SER achievement and different SNR values
with different system configurations. The simulation results
are taken with a learning rate of 0.01 during the model
training. To justify of ICI and ISI handling capability of the
proposed model, different CP values and pilot numbers are
considered for system configuration. From Fig. 13, it is indi-
cated that with the CP=20, pilot=64 and CP=20, pilot=32, the
proposed model showed almost the same SER performance
with SNRs for the U-1 and U-2, respectively. In addition, with
a fixed pilot of 64 and changing the CP to 16, the proposed
model performance is not much different. Furthermore, with
the value of CP=16 and pilot=32, the proposed model showed
a little bit less performance than other configurations for both
Us.
TABLE 2. Testing performance evaluation of the proposed HyDNN
Mini-batch
(Mb)
Epoch (E) Learning
Rate (LR)
Testing Acc. (%)
0.01 99.93
500 100 0.001 99.56
0.01 99.86
1000 100 0.001 99.46
0.01 99.83
2000 100 0.001 99.30
0.01 99.73
5000 100 0.001 98.60
0.01 99.60
8000 100 0.001 96.60
VOLUME 11, 2023 11
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
B. COMPLEXITY ANALYSIS
In this section, the proposed model complexity is explained.
By taking into account the quantity of floating-point oper-
ations, computational complexity is quantified in terms of
time. The complexity of the 1-D CNN model can be described
as: O(N,K)[24] where Nstands for the number of the filters
and Kfor the convolutional kernel. The complexity of the
LSTM is given by: O(L)[46] where Lis the weight size
of hidden layers. So, Consequently, the complexity of the
BiLSTM model can be written as O(2L)as it flows in a
bi-directional mode. However, the HyDNN detection model
complexity can be represented as: O(N,K+2L). In the tradi-
tional LS and MMSE-SIC detection method, the complexity
is as: O(4M+ 2), where Mstands for the modulation order.
The complexity of the ML approach is further represented as
follows: O(2M). The above complexity of the HyDNN model
is involved in the offline training process which required a
long running time during the training of the model. However,
the proposed HyDNN model includes many parameters, the
complexity can be decreased by using parallelization of the
graphics processing unit (GPU) [47] in the actual online pro-
cess. Table 3 shows the justification of each method’s result
of the online process which is executed by proposed algo-
rithms. In addition, floating point operations (FLOPs) are also
presented in the table. Although the proposed model required
higher complexity than others, its estimation performance is
higher than all and the computational time can be reduced by
using GPU parallel computing.
V. CONCLUSIONS
A multiuser CE and SD method in uplink transmission for
the NOMA-OFDM is presented in this study using a HyDNN
network. The HyDNN model is constructed by a 1D-CNN
and a BiLSTM model. The proposed HyDNN model works
better than the traditional SIC-based techniques in terms of
symbol recovery rate.It is demonstrated that the proposed
HyDNN network is more efficient for radio resources like the
strength of signals, pilot symbols, and CP data than traditional
CE approaches like MMSE, LS, and ML. Additionally, the
proposed model outperforms the CNN and BiLSTM mod-
els when using the same channel parameters. The proposed
model shows high learning ability even though the model with
training less learning rate. Additionally, the proposed system
SER detection performance rate is significantly higher than
that of existing approaches. Future applications of this tech-
nique include more intricate systems like the NOMA system
with MIMO communication. It can also be used in physical
layers applications like reconfigurable reflecting surfaces-
based wireless networks.
REFERENCES
[1] X. Chen, G. Liu, Z. Ma, X. Zhang, W. Xu, and P. Fan, ‘‘Optimal power
allocations for non-orthogonal multiple access over 5g full/half-duplex
relaying mobile wireless networks,’’ IEEE transactions on Wireless com-
munications, vol. 18, no. 1, pp. 77–92, Oct. 2018.
[2] Y. Saito, Y. Kishiyama, A. Benjebbour, T. Nakamura, A. Li, and
K. Higuchi, ‘‘Non-orthogonal multiple access (noma) for cellular future
radio access,’’ in 2013 IEEE 77th vehicular technology conference (VTC
Spring). IEEE, Jun. 2013, pp. 1–5.
[3] Y.-R. Lee, W.-S. Lee, J.-S. Jung, C.-Y. Park, Y.-H. You, and H.-K. Song,
‘‘Hybrid beamforming with reduced rf chain based on pzf and pd-noma in
mmwave massive mimo systems,’’ IEEE Access, vol. 9, pp. 60 695–60 703,
Apr. 2021.
[4] D. Hwang, J. Yang, S. S. Nam, and H.-K. Song, ‘‘Optimal multi-antenna
transmission for the cooperative non-orthogonal multiple-access system,’’
Applied Sciences, vol. 11, no. 5, p. 2203, Mar. 2021.
[5] X. Wang, P. Zhu, D. Li, Y. Xu, and X. You, ‘‘Pilot-assisted simo-noma
signal detection with learnable successive interference cancellation,’’ IEEE
Communications Letters, vol. 25, no. 7, pp. 2385–2389, Apr. 2021.
[6] K. Higuchi and A. Benjebbour, ‘‘Non-orthogonal multiple access (noma)
with successive interference cancellation for future radio access,’’ IEICE
Transactions on Communications, vol. 98, no. 3, pp. 403–414, Mar. 2015.
[7] V. Andiappan and V. Ponnusamy, ‘‘Deep learning enhanced noma system:
A survey on future scope and challenges,’’ Wireless Personal Communica-
tions, vol. 123, no. 1, pp. 839–877, Sep. 2022.
[8] C.-J. Chun, J.-M. Kang, and I.-M. Kim, ‘‘Deep learning-based channel
estimation for massive mimo systems,’’ IEEE Wireless Communications
Letters, vol. 8, no. 4, pp. 1228–1231, Apr. 2019.
[9] H. Hua, X. Wang, and Y. Xu, ‘‘Signal detection in uplink pilot-assisted
multi-user mimo systems with deep learning,’’ in 2019 Computing, Com-
munications and IoT Applications (ComComAp). IEEE, Oct. 2019, pp.
369–373.
[10] X. Miao, D. Guo, and X. Li, ‘‘Grant-free noma with deviceactivity learning
using long short-term memory,’’ IEEE Wireless Communications Letters,
vol. 9, no. 7, pp. 981–984, Feb. 2020.
[11] T. O’shea and J. Hoydis, ‘‘An introduction to deep learning for the physical
layer,’’ IEEE Transactions on Cognitive Communications and Networking,
vol. 3, no. 4, pp. 563–575, Oct. 2017.
[12] T. Zhao, F. Li, and P. Tian, ‘‘A deep-learning method for device activity
detection in mmtc under imperfect csi based on variational-autoencoder,’’
IEEE Transactions on Vehicular Technology, vol. 69, no. 7, pp. 7981–7986,
May 2020.
[13] C. She, R. Dong, Z. Gu, Z. Hou, Y. Li, W. Hardjawana, C. Yang, L. Song,
and B. Vucetic, ‘‘Deep learning for ultra-reliable and low-latency commu-
nications in 6g networks,’’ IEEE network, vol. 34, no. 5, pp. 219–225, Jul.
2020.
[14] M. S. Sim, Y.-G. Lim, S. H. Park, L. Dai, and C.-B. Chae, ‘‘Deep learning-
based mmwave beam selection for 5g nr/6g with sub-6 ghz channel in-
formation: Algorithms and prototype validation,’’ IEEE Access, vol. 8, pp.
51 634–51646, Mar. 2020.
[15] Z. Mao, X. Liu, and M. Peng, ‘‘Channel estimation for intelligent reflecting
surface assisted massive mimo systems–a deep learning approach,’’ IEEE
Communications Letters, Jan. 2022.
[16] I. A. Nemer, T. R. Sheltami, S. Belhaiza, and A. S. Mahmoud, ‘‘Energy-
efficient uav movement control for fair communication coverage: A deep
reinforcement learning approach,’’ Sensors, vol. 22, no. 5, p. 1919, Mar.
2022.
[17] J. An, C. Xu, Q. Wu, D. W. K. Ng, M. Di Renzo, C. Yuen, and L. Hanzo,
‘‘Codebook-based solutions for reconfigurable intelligent surfaces and
their open challenges,’’ IEEE Wireless Communications, pp. 1–8, Nov.
2022.
[18] M. AbdelMoniem, S. M. Gasser, M. S. El-Mahallawy, M. W. Fakhr, and
A. Soliman, ‘‘Enhanced noma system using adaptive coding and modula-
tion based on lstm neural network channel estimation,’’ Applied Sciences,
vol. 9, no. 15, p. 3022, Jul. 2019.
[19] Y. Lu, P. Cheng, Z. Chen, W. H. Mow, Y. Li, and B. Vucetic, ‘‘Deep multi-
task learning for cooperative noma: System design and principles,’’ IEEE
Journal on Selected Areas in Communications, vol. 39, no. 1, pp. 61–78,
Nov. 2020.
[20] A. Emir, F. Kara, H. Kaya, and H. Yanikomeroglu, ‘‘Deep learning empow-
ered semi-blind joint detection in cooperative noma,’’ IEEE Access, vol. 9,
pp. 61 832–61852, Apr. 2021.
[21] H. Ye, G. Y. Li, and B.-H. Juang, ‘‘Power of deep learning for channel
estimation and signal detection in ofdm systems,’’ IEEE Wireless Commu-
nications Letters, vol. 7, no. 1, pp. 114–117, Sep. 2018.
[22] C. Lin, Q. Chang, and X. Li, ‘‘A deep learning approach for mimo-noma
downlink signal detection,’’ Sensors, vol. 19, no. 11, p. 2526, Jun. 2019.
[23] P. Dong, H. Zhang, G. Y. Li, I. S. Gaspar, and N. NaderiAlizadeh, ‘‘Deep
cnn-based channel estimation for mmwave massive mimo systems,’’ IEEE
Journal of Selected Topics in Signal Processing, vol. 13, no. 5, pp. 989–
1000, Jul. 2019.
12 VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
TABLE 3. Online computational time (in seconds) and FLOPs for the different estimation methods.
Methods Online Estimation Time Offline Training Time FLOPs Number
LS+MMSE 33.30907 - 4.5×108
ML 3.60645 - 9.4×106
HyDNN For LR=0.01, RT=9.85619;
For LR=0.001, RT=9.91830
For LR=0.01, RT=18214.45;
For LR=0.001, RT=29274.60 7.9×108
RT=Computational time; LR=Learning rate;
[24] S. Kiranyaz, O. Avci, O. Abdeljaber, T. Ince, M. Gabbouj, and D. J.
Inman, ‘‘1d convolutional neural networks and applications: A survey,’’
Mechanical systems and signal processing, vol. 151, p. 107398, Nov. 2021.
[25] L. Chuan, C. Qing, and L. Xianxu, ‘‘Uplink noma signal transmission with
convolutional neural networks approach,’’ Journal of Systems Engineering
and Electronics, vol. 31, no. 5, pp. 890–898, Oct. 2020.
[26] Y. Xie, K. C. Teh, and A. C. Kot, ‘‘Deep learning-based joint detection for
ofdm-noma scheme,’’ IEEE Communications Letters, vol. 25, no. 8, pp.
2609–2613, May 2021.
[27] Z. Yin, J. Chen, G. Li, H. Wang, W. He, and Y. Ni, ‘‘A deep learning-based
user selection scheme for cooperative noma system with imperfect csi,’’
Wireless Communications and Mobile Computing, vol. 2022, May 2022.
[28] D. Haviv, A. Rivkind, and O. Barak, ‘‘Understanding and controlling
memory in recurrent neural networks,’’ in International conference on
machine learning. PMLR, Sep. 2019, pp. 2663–2671.
[29] S. Hochreiter and J. Schmidhuber, ‘‘Long short-term memory,’’ Neural
Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
[30] W. Xu, J. An, Y. Xu, C. Huang, L. Gan, and C. Yuen, ‘‘Time-varying
channel prediction for ris-assisted mu-miso networks via deep learning,’’
IEEE Transactions on Cognitive Communications and Networking, vol. 8,
no. 4, pp. 1802–1815, Jul. 2022.
[31] S. Pandya, M. A. Wakchaure, R. Shankar, and J. R. Annam, ‘‘Analysis of
noma-ofdm 5g wireless system using deep neural network,’’ The Journal of
Defense Modeling and Simulation, vol. 19, no. 4, pp. 799–806, Mar. 2022.
[32] D. Tian, P. Miao, H. Peng, W. Yin, and X. Li, ‘‘Volterra-aided neural
network equalization for channel impairment compensation in visible light
communication system,’’ Photonics, vol. 9, no. 11, p. 845, Nov. 2022.
[33] M. H. Rahman, M. A. S. Sejan, M. A. Aziz, J.-I. Baik, D.-S. Kim, and H.-
K. Song, ‘‘Deep learning based improved cascaded channel estimation and
signal detection for reconfigurable intelligent surfaces-assisted mu-miso
systems,’’ IEEE Transactions on Green Communications and Networking,
pp. 1–1, Jan. 2023.
[34] R. L. Abduljabbar, H. Dia, and P.-W. Tsai, ‘‘Unidirectional and bidirec-
tional lstm models for short-term traffic prediction,’’ Journal of Advanced
Transportation, vol. 2021, Mar. 2021.
[35] S. Siami-Namini, N. Tavakoli, and A. S. Namin, ‘‘The performance of
lstm and bilstm in forecasting time series,’’ in 2019 IEEE International
Conference on Big Data (Big Data). IEEE, Feb. 2019, pp. 3285–3292.
[36] M. H. Rahman, M. A. S. Sejan, S.-G. Yoo, M.-A. Kim, Y.-H. You, and
H.-K. Song, ‘‘Multi-user joint detection using bi-directional deep neural
network framework in noma-ofdm system,’’ Sensors, vol. 22, no. 18, p.
6994, Sep. 2022.
[37] A. Le Ha, T. Van Chien, T. H. Nguyen, W. Choi et al., ‘‘Deep learning-aided
5g channel estimation,’’ in 2021 15th international conference on ubiqui-
tous information management and communication (IMCOM). IEEE, Mar.
2021, pp. 1–7.
[38] Q. Bai, J. Wang, Y. Zhang, and J. Song, ‘‘Deep learning-based channel esti-
mation algorithm over time selective fading channels,’’ IEEE Transactions
on Cognitive Communications and Networking, vol. 6, no. 1, pp. 125–134,
Sep. 2019.
[39] P. Chen and H. Kobayashi, ‘‘Maximum likelihood channel estimation and
signal detection for ofdm systems,’’ in 2002 IEEE International Confer-
ence on Communications. Conference Proceedings. ICC 2002 (Cat. No.
02CH37333), vol. 3. IEEE, Aug. 2002, pp. 1640–1645.
[40] M.-H. Hsieh and C.-H. Wei, ‘‘Channel estimation for ofdm systems based
on comb-type pilot arrangement in frequency selective fading channels,’’
IEEE Transactions on Consumer Electronics, vol. 44, no. 1, pp. 217–225,
Feb. 1998.
[41] L. Shi, B. Guo, and L. Zhao, ‘‘Block-type pilot channel estimation for ofdm
systems under frequency selective fading channels,’’ pp. 21–24, Jul. 2009.
[42] S. Khan, S. Durrani, M. B. Shahab, S. J. Johnson, and S. Camtepe, ‘‘Joint
user and data detection in grant-free noma with attention-based bilstm
network,’’ arXiv preprint arXiv:2209.06392, Sep. 2022.
[43] T. Van Laarhoven, ‘‘L2 regularization versus batch and weight normaliza-
tion,’’ arXiv preprint arXiv:1706.05350, Jun. 2017.
[44] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’
arXiv preprint arXiv:1412.6980, Jan. 2014.
[45] A. Kumar and K. Kumar, ‘‘Deep learning-based joint noma signal detection
and power allocation in cognitive radio networks,’’ IEEE Transactions on
Cognitive Communications and Networking, vol. 8, no. 4, pp. 1743–1752,
Jul. 2022.
[46] S. Hochreiter and J. Schmidhuber, ‘‘Long short-term memory,’’ Neural
computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
[47] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press,
Nov. 2016.
MD HABIBUR RAHMAN received B.S. in elec-
trical and electronic engineering from Islamic Uni-
versity, Kushtia, Bangladesh, in 2014 and he re-
ceived M.S. degree in electronic engineering from
Pukyong National University, Busan, the Republic
of Korea, in August 2020. Currently, he is pursuing
his Ph.D. degree in the department of information
and communication engineering and department
of convergence engineering for intelligent drone
at Sejong University, Seoul, Republic of Korea.
His research interests include wireless sensor networks, machine learning,
wireless communication, intelligent reflective surfaces, visible light commu-
nication, and IoT applications.
MOHAMMAD ABRAR SHAKIL SEJAN received
B.S. and M.S. in Computer Science and Engineer-
ing from Islamic University, Kushtia, Bangladesh
in 2014 and 2016 respectively. Dr. Sejan received
his Ph.D. degree from Pukyong National Univer-
sity, Busan, the Republic of Korea in electronic
engineering in February 2022. Currently, he is a
postdoctoral fellow at the Information and Com-
munication Engineering department and Depart-
ment of Convergence Engineering for Intelligent
Drone, Sejong University, Seoul, 05006, Republic of Korea. His research
interests include visible light communication, optical communication, wire-
less sensor networks, reconfigurable intelligent surface, protocol design for
communication, and modulation techniques in VLC and IoT applications. Dr.
Sejan serves as a reviewer in different journals and international conferences.
VOLUME 11, 2023 13
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
Rahman et al.: HyDNN: A Hybrid Deep Learning Framework Based Multiuser Uplink Channel Estimation and Signal Detection
MD ABDUL AZIZ received B.S. and M.S. in elec-
trical and electronic engineering from Islamic Uni-
versity, Kushtia, Bangladesh, in 2014 and 2018,
respectively. Currently, he is pursuing his Ph.D.
degree in the department of information and com-
munication engineering and department of conver-
gence engineering for intelligent drone at Sejong
University, Seoul, Republic of Korea. His research
interests include machine learning, wireless com-
munication, and intelligent reflective surfaces.
YOUNG-HWAN YOU received the B.S., M.S.,
and Ph.D. degrees in electronic engineering from
Yonsei University, Seoul, South Korea, in 1993,
1995, and 1999, respectively. From 1999 to 2002,
he was a Senior Researcher with the Wireless
PAN Technology Project Office, Korea Electron-
ics Technology Institute, Seongnam, South Korea.
Since 2002, he has been a Professor with the De-
partment of Computer Engineering and the Depart-
ment of Convergence Engineering for Intelligent
Drone, Sejong University, Seoul. His research interests include wireless
communications and signal processing with particular focus on 4G LTE, the
NB-IoT, 5G new radio, and ultrareliable low-latency communication.
HYOUNG-KYU SONG received B.S., M.S., and
Ph.D. degrees in electronic engineering from Yon-
sei University, Seoul, Korea, in 1990, 1992, and
1996, respectively. From 1996 to 2000 he had been
managerial engineer in Korea Electronics Tech-
nology Institute (KETI), Korea. Since 2000, he
has been a professor of the Department of In-
formation and Communication Engineering, and
Convergence Engineering for Intelligent Drone,
Sejong University, Seoul, Korea. His research in-
terests include digital and data communications, information theory and their
applications with an emphasis on mobile communications.
14 VOLUME 11, 2023
This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3290217
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
