Content uploaded by Alaa Awad Abdellatif
Author content
All content in this area was uploaded by Alaa Awad Abdellatif on Feb 22, 2021
Content may be subject to copyright.
A preview of the PDF is not available
Deep Reinforcement Learning for Network Selection Over Heterogeneous Health Systems
Abstract and Figures
Smart health systems improve our quality of life by integrating information and technology into health and medical practices. Such technologies can significantly improve the existing health services. However, reliability, latency, and limited network resources are among the many challenges hindering the realization of smart health systems. Thus, in this paper, we leverage the dense heterogeneous network (HetNet) architecture over 5G network to enhance network capacity and provide seamless connectivity for smart health systems. However, network selection in HetNets is still a challenging problem that needs to be addressed. Inspired by the success of Deep Reinforcement Learning (DRL) in solving complicated control problems, we present a novel DRL model for solving the network selection problem with the aim of improving medical data delivery over heterogeneous health systems. Specifically, we integrate the network selection problem with adaptive compression at the edge to formulate an optimization model that aims at minimizing the transmission energy consumption while meeting diverse applications' Quality of service (QoS) requirements. Our experimental results show that the proposed DRL model could minimize the energy consumption and cost compared to the greedy techniques while meeting different users' demands in high dynamics environments.
Figures - uploaded by Alaa Awad Abdellatif
Author content
All figure content in this area was uploaded by Alaa Awad Abdellatif
Content may be subject to copyright.
... The positive effects of edge computing for smart alternatives for distributed computation as well as the evaluation of smart IoT medical sensing were underlined in this paper [48] by the authors. This means that by fully or partially educating edge intelligent nodes at the edge of the network level, system latency can be decreased. ...
... Model QoS Metric Sensors [46] Matching based model Latency, energy consumption IoT [47] Deep Reinforcement Learning (DRL) model Energy consumption and latency Mobile sensors [48] DL models Energy consumption and latency IoT [49] Monte Carlo Throughput and energy efficiency Medical and motion [50] Hybrid fog-cloud of offloading (HFCO) delay IoT [51] Confident information coverage (CIC) model Energy consumption Mobile sensors [52] Cluster-based hierarchical approach Energy consumption Smart Sensors [53] Cloud Based Models Energy consumption N/A [54] Agent-based modeling and Ontology Overall QoS Body sensors [55] Secure human-centric mobility-aware (SHM) model Throughput and latency CPS sensors [56] Grey Filter Bayesian Convolution Neural Network (GFB-CNN) Delay and latency Smart IoT sensors [58] Clustered federated learning (CFL) model Latency Smart IoT sensors [59] Network model Latency IoT sensors ...
... 23, x FOR PEER REVIEW 20 of This pa ern shows that QoS issues in smart healthcare are receiving increased a ention from the research community[46][47][48][49][50][51][52][53][54][55][56]. ...
Smart healthcare is altering the delivery of healthcare by combining the benefits of IoT, mobile, and cloud computing. Cloud computing has tremendously helped the health industry connect healthcare facilities, caregivers, and patients for information sharing. The main drivers for implementing effective healthcare systems are low latency and faster response times. Thus, quick responses among healthcare organizations are important in general, but in an emergency, significant latency at different stakeholders might result in disastrous situations. Thus, cutting-edge approaches like edge computing and artificial intelligence (AI) can deal with such problems. A packet cannot be sent from one location to another unless the “quality of service” (QoS) specifications are met. The term QoS refers to how well a service works for users. QoS parameters like throughput, bandwidth, transmission delay, availability, jitter, latency, and packet loss are crucial in this regard. Our focus is on the individual devices present at different levels of the smart healthcare infrastructure and the QoS requirements of the healthcare system as a whole. The contribution of this paper is five-fold: first, a novel pre-SLR method for comprehensive keyword research on subject-related themes for mining pertinent research papers for quality SLR; second, SLR on QoS improvement in smart healthcare apps; third a review of several QoS techniques used in current smart healthcare apps; fourth, the examination of the most important QoS measures in contemporary smart healthcare apps; fifth, offering solutions to the problems encountered in delivering QoS in smart healthcare IoT applications to improve healthcare services.
... In [8], DRL is used to allocate spectrum resources in cognitive radio networks. In the domain of dense heterogeneous networks (HetNets) over 5G, research highlighted in [9] employs DRL to navigate the network selection challenge, striving to enhance medical data delivery within smart health systems. This model seeks to bolster energy consumption and latency, and satisfy a spectrum of Quality of Service (QoS) requirements. ...
... This metric represents the mean number of videos uploaded or downloaded by each node. Table 1 presents the simulation and the DRL parameters respectively [9], [34]. ...
The traditional healthcare system is increasingly challenged by its dependence on in-person consultations and manual monitoring, struggling with issues of scalability, the immediacy of care, and efficient resource allocation. As the global population ages and chronic conditions proliferate, the demand for healthcare systems capable of delivering efficient and remote care is becoming more pressing. In this context, Deep Reinforcement Learning (DRL) emerges as a technological advancement that improves the healthcare by enabling smart, adaptive, and real-time decision-making processes. Existing DRL applications in resource allocation, however, face significant challenges. They often lack the adaptability required to respond to the dynamic and complex nature of healthcare environments, struggle with optimizing latency, and fail to address specific node capacity constraints key factors that impacts the effectiveness of healthcare applications. Addressing these challenges, this paper introduces the Deep Reinforcement Learning for Live Video Transmission (DRL-LVT) framework. This new technique optimizes video resource allocation in Device-to-Device (D2D) networks within healthcare settings. By formulating the video resource allocation challenge as a multi-objective optimization problem, the framework aims to minimize network delays while respecting node capacity limitations. The core of DRL-LVT is its novel algorithm that leverages Deep Reinforcement Learning (DRL) to dynamically adapt to changing environmental conditions, facilitating real-time decisions that consider node capacities, latency, and the overall network dynamics. We evaluate the performance of our proposed model and benchmark it against existing state-of-the-art techniques. Our results demonstrate significant improvements in efficiency, reliability, and adaptability, making the DRL-LVT framework a robust solution for real-time remote patient monitoring in smart healthcare systems.
... In this context, RL has been used to allow each user to learn from his/her previous experience and define the optimal RAN-selection policy. RL-based network selection solutions in [86], [87] have depicted the efficiency of RL in achieving swift convergence to near-optimal performance. Specifically, the authors in [87] present a DRL-based solution that leverages dense HetNet within the smart health system to provide seamless connectivity for healthcare applications. ...
... RL-based network selection solutions in [86], [87] have depicted the efficiency of RL in achieving swift convergence to near-optimal performance. Specifically, the authors in [87] present a DRL-based solution that leverages dense HetNet within the smart health system to provide seamless connectivity for healthcare applications. In particular, a multi-objective optimization problem was formulated to minimize, for each user, the transmission energy consumption, monetary cost, latency, and signal distortion, while satisfying the applications' QoS requirements. ...
The rise of chronic disease patients and the pandemic pose immediate threats to healthcare expenditure and mortality rates. This calls for transforming healthcare systems away from one-on-one patient treatment into intelligent health systems, leveraging the recent advances of IoT and smart sensors. Meanwhile, Reinforcement Learning (RL) has witnessed an intrinsic breakthrough in solving a variety of complex problems for distinct applications and services. Thus, this paper presents a comprehensive survey of the recent models and techniques of RL that have been developed/used for supporting Intelligent-healthcare (I-health) systems. It can guide the readers to deeply understand the state-of-the-art regarding the use of RL in the context of I-health. Specifically, we first present an overview of the I-health systems’ challenges, architecture, and how RL can benefit these systems. We then review the background and mathematical modeling of different RL, Deep RL (DRL), and multi-agent RL models. We highlight important guidelines on how to select the appropriate RL model for a given problem, and provide quantitative comparisons, showing the results of deploying key RL models in two scenarios that can be followed in monitoring applications. After that, we conduct an in-depth literature review on RL’s applications in I-health systems, covering edge intelligence, smart core network, and dynamic treatment regimes. Finally, we highlight emerging challenges and future research directions to enhance RL’s success in I-health systems, which opens the door for exploring some interesting and unsolved problems.
... UDS can be considered as a baseline solution that reduces the complexity of the scheduling problem by solving it from user-centric point of view. FRA scheme is another baseline solution (i.e., implements the idea of RAN selection in [43]) that allows each user to select its RAN while distributing the available communication and computational resources, on the selected RANs, equally between EUs. ...
5G-and-beyond networks are designed to fulfill the communication and computation requirements of various industries, which requires not only transporting the data, but also processing them to meet/address diverse key performance indicators (KPIs). Network Function Virtualization (NFV) has emerged to enable this vision by: (i) collecting the requirements of diverse services, using graphs of Virtual Network Functions (VNFs); and (ii) mapping these requirements into network management decisions. Because of the latter, we need to efficiently allocate computing and network resources to support the desired services, and because of the former such decisions must be jointly optimized considering all KPIs associated with supported services. Thus, this paper proposes an optimized, intelligent network slicing framework to maintain a high performance of network operation by supporting diverse and heterogeneous services, while meeting new KPIs, e.g., reliability, energy consumption, and data quality. Different from the existing works, which are mainly designed considering traditional metrics like throughput and latency, we present a novel methodology and resource allocation schemes that enable high-quality selection of radio points of access, VNF placement and data routing, as well as data compression ratios, from the end users to the cloud. Our results depict the efficiency of the proposed framework in enhancing the network performance when compared to baseline approaches that consider partial network view or fair resource allocation.
... S is the state space of the agent, A is the action space, P : S × A × S → [0, 1] is the transition function, r : S × A → R is the reward function, and γ ∈ [0, 1] is the discount factor indicating the importance of future rewards on the current state. In addition to autonomous driving, stochastic sequential decision making problem occur in a range of realworld tasks such as game playing [41], health services [42], military planning [43], and water supplies [44]. ...
Human driving decisions are the leading cause of road fatalities. Autonomous driving naturally eliminates such incompetent decisions and thus can improve traffic safety and efficiency. Deep reinforcement learning (DRL) has shown great potential in learning complex tasks. Recently, researchers investigated various DRL-based approaches for autonomous driving. However, exploiting multi-modal fusion to generate perception and motion prediction and then leveraging these predictions to train a latent DRL has not been targeted yet. To that end, we propose enhancing urban autonomous driving using multi-modal fusion with latent DRL. A single LIDAR sensor is used to extract bird's-eye view (BEV), range view (RV), and residual input images. These images are passed into LiCaNext, a real-time multi-modal fusion network, to produce accurate joint perception and motion prediction. Next, predictions are fed with another simple BEV image into the latent DRL to learn a complex end-to-end driving policy ensuring safety, efficiency, and comfort. A sequential latent model is deployed to learn more compact representations from inputs, leading to improved sampling efficiency for reinforcement learning. Our experiments are simulated on CARLA and evaluated against state-of-the-art DRL models. Results manifest that our method learns a better driving policy that outperforms other prevailing models. Further experiments are conducted to reveal the effectiveness of our proposed approach under different environments and varying weather conditions.
Next-generation wireless networks (NGWNs) are moving towards a more advanced and exemplary environment encompassing data-intensive, delay-sensitive and energy-efficient services. To accommodate the stringent requirements of these radical and multifarious services, Next-generation wireless heterogeneous networks (NGWHNs) have been envisioned as an exemplary connectivity solution which enables flow based association among user devices (UDs) and radio access technologies (RATs). However, designing an intelligent RAT association scheme for NGWHNs is a significant challenge as the network heterogeneity tends to intensify in terms of access technologies and niche Quality of Service (QoS) requirements. Recently, substantial research endeavours have been carried out in this line of work but they are insufficient in sustaining adequate service levels and RAT capacity constraints concurrently. Inspired by the pitfalls in the pertinent work, an intelligent bilateral RAT selection framework has been developed. The proposed solution facilitates QoS provisioning while adhering to the RAT capacity limitations through well-defined preference functions. Within this paradigm, the proposal explores the diversity of matching game theory and double deep reinforcement learning (DDRL) that facilitates faster convergence to stable and balanced RAT selection policy. Finally, the proposed solution validated with the help of extensive simulations exhibited a significant gain of 42% and 46.35% respectively in terms of system utility and robustness in comparison to other schemes. Eventually, the performance assessment underlines the supremacy of the proposed solution by 22% in terms of system satisfaction with the varying network size.
Future networks are shifting towards more diversified and exemplary applications for societal benefits. To realise the functionalities of these advanced applications and services, Next-generation heterogeneous networks (HetNets) emerged as an exemplary connectivity solution, enabling each user device (UD) to associate with a radio access technology (RAT) in accordance with the demanded service. However, the design of an intelligent RAT association scheme for Beyond 5G (B5G) networks is becoming a crucial challenge due to the rapid growth of network heterogeneity in terms of RATs and personalised service requirements. Lately, considerable research endeavour has been noted in this direction but they are inadequate in maintaining Quality of Service (QoS) levels and RAT constraints simultaneously. Motivated by the gaps in the existing literature, an intelligent multi-connectivity approach has been proposed that facilitates agile differentiated service provisioning on the premise of ensuring the RAT capacity constraints. In particular, the proposed approach leverages the synergetic integration of matching game theory (MGT) and double deep reinforcement learning (DDRL) to attain a fine-grained UD-RAT association policy. Eventually, the proposed approach corroborated through the rigorous simulations demonstrated a substantial improvement in fairness, robustness and system satisfaction with the increase in network size.
The COVID-19 outburst has encouraged the adoption of Internet of Medical Things (IoMT) network to empower the antiquated healthcare system and alleviate the health care costs. To realise the functionalities of the IoMT network, 5G heterogeneous networks emerged as an exemplary connectivity solution as it facilitates diversified service provisioning in the service delivery model at more convenient care. However, the crucial challenge for 5G heterogeneous wireless connectivity solution is to facilitate agile differentiated service provisioning. Lately, considerable research endeavour has been noted in this direction but multiservice consideration and battery optimisation have not been addressed. Motivated by the gaps in the existing literature, an intelligent radio access technology selection approach has been proposed to ensure Quality of Service provisioning in a multiservice scenario on the premise of battery optimisation. In particular, the proposed approach leverages the concept of Double Deep Reinforcement Learning to attain an optimal network selection policy. Eventually, the proposed approach corroborated by the rigorous simulations demonstrated a substantial improvement in the overall system utility. Subsequently, the performance evaluation underlines the efficacy of the proposed scheme in terms of convergence and complexity.
The rapid production of mobile devices along with the wireless applications boom is continuing to evolve daily. This motivates the exploitation of wireless spectrum using multiple Radio Access Technologies (multi-RAT) and developing innovative network selection techniques to cope with such intensive demand while improving Quality of Service (QoS). Thus, we propose a distributed framework for dynamic network selection at the edge level, and resource allocation at the Radio Access Network (RAN) level, while taking into consideration diverse applications’ characteristics. In particular, our framework employs a deep Multi-Agent Reinforcement Learning (DMARL) algorithm, that aims to maximize the edge nodes’ quality of experience while extending the battery lifetime of the nodes and leveraging adaptive compression schemes. Indeed, our framework enables data transfer from the network’s edge nodes, with multi-RAT capabilities, to the cloud in a cost and energy-efficient manner, while maintaining QoS requirements of different supported applications. Our results depict that our solution outperforms state-of-the-art techniques of network selection in terms of energy consumption, latency, and cost.
Parkinson's disease (PD) can gradually affect people's lives thus attracting tremendous attention. Early PD detection and treatment can help control the disease progress, relief from the symptoms and improve the patients' life quality. However, the current practice of PD diagnosis is conducted in a clinical setup and administrated by a PD specialist due to the early signs of PD are not noticeable in daily life. According to the report of CDC/NIH, the diagnosed time of PD ranges from 2-10 years after onset. Therefore, a more accessible PD diagnosis approach is urgently demanded. In recent years, mobile health (for short mHealth) technology has been intensively investigated for preventive medicine, particularly in chronic disease management. Notably, many types of research have explored the possibility of using mobile and wearable personal devices to detect the symptom of PD and shown promising results. It provides opportunities for transforming early PD detection from clinical to daily life. This survey paper attempts to conduct a comprehensive review of mHealth technologies for PD detection from 2000 to 2019, and compares their pros and cons in practical applications and provides insights to close the performance gap between state-of-the-art clinical approaches and mHealth technologies.
In the wake of the recent pandemic of Corona Virus Disease 2019 (COVID-19), with confirmed cases having crossed 750,000, health systems across the world are getting overwhelmed; making it strenuous to maintain essential health services. Several changes were implemented in our acute mental health care service using a collaborative approach to maintain a balance between preventive measures to 'flatten the curve' and to provide care to those who were in need. Mode of service delivery was changed predominantly to tele-medicine, amongst others. It was found to be a workable model, albeit further follow up will be required to better understand its viability and feasibility to withstand the COVID-19 cataclysm.
Network selection is one of important techniques in cognitive radio networks. Multiple authorized networks can provide secondary users with more spectrum resources by network selection. Network selection is the key to spectrum sharing between cognitive radio networks and multiple primary networks.Traditional network selection algorithms are off-line selection methods which are based on prior knowledge of primary networks. However, in the complex network environment, it is impossible to get the prior knowledge from multiple primary networks, so the off-line network selection methods are lack of efficiency. In order to meet these challenges, this paper aims at improving the quality of service of cognitive users, and based on reinforcement learning method and the achievements of dynamic spectrum access of cognitive radio in single primary network environment, proposed a deep reinforcement learning based online network selection method of cognitive radio networks with multiple primary networks.
Recent developments of edge computing and content caching in wireless networks enable the Intelligent Transportation System (ITS) to provide high-quality services for vehicles. However, a variety of vehicular applications and time-varying network status make it challenging for ITS to allocate resources efficiently. Artificial intelligence algorithms, owning the cognitive capability for diverse and time-varying features of Internet of Connected Vehicles (IoCVs), enable an intent-based networking for ITS to tackle the above-mentioned challenges. In this paper, we develop an intent-based traffic control system by investigating Deep Reinforcement Learning (DRL) for 5G-envisioned IoCVs, which can dynamically orchestrate edge computing and content caching to improve the profits of Mobile Network Operator (MNO). By jointly analyzing MNO's revenue and users' quality of experience, we define a profit function to calculate the MNO's profits. After that, we formulate a joint optimization problem to maximize MNO's profits, and develop an intelligent traffic control scheme by investigating DRL, which can improve system profits of the MNO and allocate network resources effectively. Experimental results based on real traffic data demonstrate our designed system is efficient and well-performed.
The emerging 5G vehicular networks can satisfy various requirements of vehicles by traffic offloading. However, limited cellular spectrum and energy supplies restrict the development of 5G-enabled applications in vehicular networks. In this paper, we construct an intelligent offloading framework for 5G vehicular networks, by jointly utilizing licensed cellular spectrum and unlicensed channels. An optimization problem aiming at minimizing the offloading cost is formulated, and further decomposed into two subproblems due to its complexity. For the first subproblem, a two-sided matching algorithm is proposed to schedule the unlicensed spectrum while satisfying the latency constraint of users. Then, a deep reinforcement learning based method is investigated for the second one, where the system state is simplified to realize distributed offloading. Performance analyses based on real-world traces of taxies demonstrate the effectiveness of our solution.
The increased complexity and heterogeneity of emerging 5G and B5G wireless networks will require a paradigm shift from traditional resource allocation mechanisms. Deep learning (DL) is a powerful tool where a multi-layer neural network can be trained to model a resource management algorithm using network data.Therefore, resource allocation decisions can be obtained without intensive online computations which would be required otherwise for the solution of resource allocation problems. In this context, this article focuses on the application of DL to obtain solutions for the radio resource allocation problems in multi-cell networks. Starting with a brief overview of a DNN as a DL model, relevant DNN architectures and the data training procedure, we provide an overview of existing state-of-the-art applying DL in the context of radio resource allocation. A qualitative comparison is provided in terms of their objectives, inputs/outputs, learning and data training methods. Then, we present a supervised DL model to solve the sub-band and power allocation problem in a multi-cell network. Using the data generated by a genetic algorithm, we first train the model and then test the accuracy of the proposed model in predicting the resource allocation solutions. Simulation results show that the trained DL model is able to provide the desired optimal solution 86.3 percent of the time.
The development of smart vehicles brings drivers and passengers a comfortable and safe environment. Various emerging applications are promising to enrich users' traveling experiences and daily life. However, how to execute computing-intensive applications on resource-constrained vehicles still faces huge challenges. In this paper, we construct an intelligent offloading system for vehicular edge computing by leveraging deep reinforcement learning. First, both the communication and computation states are modelled by finite Markov chains. Moreover, the task scheduling and resource allocation strategy is formulated as a joint optimization problem to maximize users' Quality of Experience (QoE). Due to its complexity, the original problem is further divided into two sub-optimization problems. A two-sided matching scheme and a deep reinforcement learning approach are developed to schedule offloading requests and allocate network resources, respectively. Performance evaluations illustrate the effectiveness and superiority of our constructed system. Additional Key Words and Phrases: Vehicular system, intelligent offloading, deep reinforcement learning, edge computing.