No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Big Data Issues in Smart Grids: A Survey
Abstract
The smart power systems are based upon information and communication technologies, which lead to a deluge of data originating from various sources. To address these challenges concerning accumulated voluminous data, big data analysis in smart power systems is inevitable. This article comprehensively surveys the literature related to the big data issues in smart power systems. The background and motivation of the big data paradigm in smart power systems are first provided, and then the major issues related to the architectures, the key technologies, and standardizations of big data analytics in smart power systems are analyzed. Also, the potential applications of big data in smart power systems based upon the state-of-the-art research are highlighted. Finally, the future issues and challenges of the big data issues in modern power systems are discussed.
... Data infrastructure supports data collection, processing, analysis, and storage [62]. Smart energy systems collect, analyse, and share large amounts of data. ...
... Internet of Things (IoT) (D04) refers to smart devices that are equipped with unique identifiers (UID) and can communicate with each other around the world without requiring interaction between people or between people and computers [68,81,82]. According to data, the IoT generates massive amounts of unstructured or semi-structured data from various sources, which makes it necessary to analyse this data for further applications [62]. The sensors, actuators, and gateways (D14) are essential for the operation of many drivers that allow for the detection, measurement, control, and adjustment of physical conditions within a system [34]. ...
... The big data are concentrated in data centres (D19). A data centre is a facility that houses many computer servers and other networking equipment used to store, process, and manage large amounts of data [62]. With the high presence of virtual machines in addition to physical ones, they can be called cloud data centres [163]. ...
The evolution of technology and the diversification of service offerings have significantly influenced how energy is produced, distributed, stored, and consumed in cities. These new platforms are part of the Smart Energy concept and are aligned with the UN's 7th SDG by creating new business models that promote the growth of investment in infrastructure for clean energy technology. This development has several drivers, along with the various services offered to users, such as energy forecasting, intelligent measurement, and demand response programs. In this sense, this article aims to identify the development of technological drivers of Smart Energy, map collaborative networks among these drivers, and discuss best management practices for establishing collaborative relationships. The research was developed from a systematic literature review and a meta-analysis carried out with the aid of statistical software. Six key drivers and 20 collaborative drivers were identified. These drivers integrate four collaborative networks structured to offer an understanding of how energy systems are organised under the concept of Smart Energy. From this, the article provides insights into the development, implementation, and management of these networks and drivers to accelerate the diffusion of Smart Energy to users at all levels.
... The need for tools that can harvest, store, process, and analyze the data generated rises to encounter the era of BD. Hadoop and Apache Spark are the most common and open-source BD ecosystems [9,16]. In [9], a comprehensive explanation of the components of these ecosystems is provided. ...
... As presented in Fig. 1, Spark has four distinctive engines. This research Table 1 BD sources, applications, and challenges in SG BD sources Smart meters [1,[8][9][10][11][12] Weather data [1,2,12] Gas turbines and wind turbines [13] Sensors (e.g., Internet of Things sensors, geographical information system data) [9,12] Substation data collected from: -Phasor Measurement Units (PMUs) [9,12], -Remote Terminal Units (RTUs) [9,12], and -Digital relays SCADA [8,9,12] BD applications LF and RES forecasting [1,2,9,12] to improve integration of RES [12] DR applications [8,12,14,15] Asset management [2,12] Preventive maintenance and health monitoring [9] Power quality monitoring [9] BD challenges BD management issues such as: -Data privacy and security [8,12] -Data storage [12] Data processing and analysis [2,16] with cost-effective solutions [12] Engine Components APIs Data Citizens utilizes Spark SQL and Spark MLlib. Spark SQL is used for descriptive analytics while Spark MLlib is used for predictive analytics to build scalable ML models using the power of distributed clusters. ...
Using recent information and communication technologies for monitoring and management initiates a revolution in the smart grid. These technologies generate massive data that can only be processed using big data tools. This paper emphasizes the role of big data in resolving load forecasting, renewable energy sources integration, and demand response as significant aspects of smart grids. Meters data from the Low Carbon London Project is investigated as a case study. Because of the immense stream of meters' readings and exogenous data added to load forecasting models, addressing the problem is in the context of big data. Descriptive analytics are developed using Spark SQL to get insights regarding household energy consumption. Spark MLlib is utilized for predictive analytics by building scalable machine learning models accommodating meters' data streams. Multivariate polynomial regression and decision tree models are preferred here based on the big data point of view and the literature that ensures they are accurate and interpretable. The results confirmed the descriptive analytics and data visualization capabilities to provide valuable insights, guide the feature selection process, and enhance load forecasting models' accuracy. Accordingly, proper evaluation of demand response programs and integration of renewable energy resources is accomplished using achieved load forecasting results.
... Based on the frequency of data collection, the space needed to store it varies greatly, according to [82]- [85]. By considering one million smart meters and that each data collection occupies a space of 5kB, then if data is collected once a day at the end of the year we would have 1.82 TB of data, on the other hand, if data is collected every 15 minutes, then the volume of data collected is 2920 TB. ...
... [84], [85], [90], [153], [154] Machine Learning and Artificial Intelligence ...
The primary contributions of this paper include the systematic organization of existing research and an in-depth analysis of significant milestones in the field of CPPS, leading to the development of a taxonomy for categorizing various drivers. Furthermore, this paper elucidates current strategies for defining clear terms, models, methodologies, and attributes essential to cybersecurity policies within CPPS. The incorporation of relevant standards employed in the CPPS domain is also a paramount aspect of this review. In addition to these contributions, the paper identifies areas warranting further research and highlights emerging trends within CPPS, aiming to offer valuable insights to industry practitioners and to catalyze future research endeavors.
... In a centralized power grid, a single station's failure to provide power can cause a domino effect of power outages throughout the system [7], [8]. As a result, modern power grids are evolving towards decentralization and intelligence by implementing structural, monitoring, and topological modifications to prevent cascading failures [9], [10]. ...
... Inaccurate models can lead to inefficient and unreliable operation of the power grid, with potentially severe consequences for both the environment and society. It is crucial to develop models that can accurately capture the structure and dynamics of power grids in different locations, allowing for more effective planning, management, and control of these critical systems [10]. Watts Strogatz showed the characteristic path length and clustering in a power grid similar to the small-world network model [11], [12]. ...
Smart grid power networks are essential for addressing the global energy crisis and combating climate change. In the past few decades, information and communication infrastructure have improved a lot. As a result, studying the characteristics of smart grids has become important. To accurately represent the connectivity of different components in power networks, we need precise models. In this study, we introduce a new growth model called PowerX. This model is designed to capture the characteristics of real-world power networks. PowerX is a growth model that is designed to capture the characteristics of real-world power networks, by incorporating both random and ordered elements. Specifically, it is designed to accurately capture the degree distribution and clustering coefficient of power networks. To assess the effectiveness of PowerX, we compared it with existing growth models such as Watts Strogatz Small World model, Henneberg’s model, and Modified Henneberg’s model, using the US Western States Power Grid dataset consisting of 4789 nodes and 5571 edges. Our results show that PowerX precisely captures the degree distribution of the real dataset and its clustering coefficient is close to the actual dataset, outperforming the other models we evaluated. In addition, we used Gephi to demonstrate the features of the Western States power grid, including identifying the most important node of the network, community structure, and the strongest and weakest nodes. This research provides valuable insights into the characteristics of power networks and demonstrates the effectiveness of PowerX in accurately modeling them. The datasets and codes are publicly available for further research at: github.com/irfan2inform/powerX.
... Based on the frequency of data collection, the space needed to store it varies greatly, according to [82]- [85]. By considering one million smart meters and that each data collection occupies a space of 5kB, then if data is collected once a day at the end of the year we would have 1.82 TB of data, on the other hand, if data is collected every 15 minutes, then the volume of data collected is 2920 TB. ...
... [84], [85], [90], [153], [154] Machine Learning and Artificial Intelligence ...
The primary contributions of this paper include the systematic organization of existing research and an in-depth analysis of significant milestones in the field of CPPS, leading to the development of a taxonomy for categorizing various drivers. Furthermore, this paper elucidates current strategies for defining clear terms, models, methodologies, and attributes essential to cybersecurity policies within CPPS. The incorporation of relevant standards employed in the CPPS domain is also a paramount aspect of this review. In addition to these contributions, the paper identifies areas warranting further research and highlights emerging trends within CPPS, aiming to offer valuable insights to industry practitioners and to catalyze future research endeavors.
... A fifth strand of the literature presents big data (issues) for smart building energy utilization [26,119,120]. All solutions based on machine models require applicable data. ...
The United Nations launched sustainable development goals in 2015 that include goals for sustainable energy. From global energy consumption, households consume 20–30% of energy in Europe, North America and Asia; furthermore, the overall global energy consumption has steadily increased in the recent decades. Consequently, to meet the increased energy demand and to promote efficient energy consumption, there is a persistent need to develop applications enhancing utilization of energy in buildings. However, despite the potential significance of AI in this area, few surveys have systematically categorized these applications. Therefore, this paper presents a systematic review of the literature, and then creates a novel taxonomy for applications of smart building energy utilization. The contributions of this paper are (a) a systematic review of applications and machine learning methods for smart building energy utilization, (b) a novel taxonomy for the applications, (c) detailed analysis of these solutions and techniques used for the applications (electric grid, smart building energy management and control, maintenance and security, and personalization), and, finally, (d) a discussion on open issues and developments in the field.
... More specifically, smart grids are a future vision towards a more sustainable energy infrastructure that uses power grids which give priority to adaptability, efficiency, resilience, cleanliness, and eco-friendliness and are supported by intelligent systems to actively, autonomously, and pervasively manage, control, and monitor resources and systems [15,20,[22][23][24][25][26]. These self-sufficient systems [27] utilize heterogeneous data and a variety of data sources [28], focus on both consumers and prosumers [1], capitalize on renewable resources [7], enable a more effective delivery of energy and exchange of information [22,24,29], and operate in a more responsive, organic, and collaborative manner [7]. Thus, smart grids can enhance energy production, transmission, distribution, management, and consumption [30][31][32] and improve the effectiveness, performance, security, reliability, and availability of the power sector [33,34]. ...
To achieve sustainability and fulfill sustainable development goals, the digitalization of the power sector is vital. This study aims to examine how blockchain can be integrated into and enrich smart grids. In total, 10 research questions are explored. Scopus and Web of Science (WoS) were used to identify documents related to the topic. The study involves the analysis of 1041 scientific documents over the period 2015–2022. The related studies are analyzed from different dimensions including descriptive statistics, identification of the most common keywords and most widely used outlets, examination of the annual scientific production, the analysis of the most impactful and productive authors, countries, and affiliations. The advancement of the research focus and the most popular topics are also examined. Additionally, the results are analyzed, the main findings are discussed, open issues and challenges are presented, and suggestions for new research directions are provided. Based on the results, it was evident that blockchain plays a vital role in securing smart grids and realizing power sector digitalization, as well as in achieving sustainability and successfully meeting sustainable development goals.
... The article applies time series and regression analyses to understand the relationship between weather patterns and wind power output. Furthermore, in another research article [30], data analysis techniques for demand response in smart grids, such as clustering analysis, pattern recognition, and regression analysis, are used to explore the use of data analysis to analyze energy consumption patterns, customer behavior, and grid conditions for effective demand response programs. ...
The rapid adoption of hydrogen as an eco-friendly energy source has necessitated the development of intelligent power management systems capable of efficiently utilizing hydrogen resources. However, guaranteeing the security and integrity of hydrogen-related data has become a significant challenge. This paper proposes a pioneering approach to ensure secure hydrogen data analysis by integrating blockchain technology, enhancing trust, transparency, and privacy in handling hydrogen-related information. Combining blockchain with intelligent power management systems makes the efficient utilization of hydrogen resources feasible. Using smart contracts and distributed ledger technology facilitates secure data analysis (SDA), real-time monitoring, prediction, and optimization of hydrogen-based power systems. The effectiveness and performance of the proposed approach are demonstrated through comprehensive case studies and simulations. Notably, our prediction models, including ABiLSTM, ALSTM, and ARNN, consistently delivered high accuracy with MAE values of approximately 0.154, 0.151, and 0.151, respectively, enhancing the security and efficiency of hydrogen consumption forecasts. The blockchain-based solution offers enhanced security, integrity, and privacy for hydrogen data analysis, thus advancing clean and sustainable energy systems. Additionally, the research identifies existing challenges and outlines future directions for further enhancing the proposed system. This study adds to the growing body of research on blockchain applications in the energy sector, specifically on secure hydrogen data analysis and intelligent power management systems.
... Augmented reality smart glasses are computer-capable wearable glasses, which add additional data, primarily threedimensional images and data like videos and animations [31]. For example, Hexagon's augmented reality smart glassmonitoring solutions utilises AR in order to assist manufacturing operation via maintenance process and key operations [32]. ...
This study has clearly proved that a well-designed power monitoring system is quite effective and necessary in the current year. During the same time, the integration of IOT devices with this system increased its effectiveness and use in a variety of disciplines. On the other hand, using this monitoring system makes it simple to maintain direct and effective connection with GSM and embedded controllers. Several sensors in smart design monitoring systems come in handy in this circumstance. Aside from that, the study's critical discussion revealed that smart power monitoring systems have a favourable influence on motion direction, home security, augmented reality, and technological disruption. The researcher gets data from 100 respondents out of a total population of 500. The sample for the study is chosen by the researcher using the basic random selection approach. The researcher prefers simple random sampling because it is the simplest way for selecting samples for the study. The researcher distributes the questionnaire to the study's respondents and so collects their responses in order to perform the research.
... can be found in [4]. In this context, the accumulated, voluminous and continuously generated data, whose volume is quite larger than that of traditional ones in common data processing systems, make real-time data analysis in smart grid systems very challenging [5]. According to [6], a smart grid system with 2 million customers will generate about 22 Gigabytes of data each day. ...
... The concepts and applications of big data analytics have also been the subject of numerous surveys and given a survey of methods and tools for large data management [23]. Representative surveys may be found and big data analytics have also been used in intelligent transportation and smart grid systems [24,25]. In conclusion, blockchain may enhance big data by improving data integrity, security, and privacy, enabling real-time data analytics, enhancing data sharing, and enhancing big data quality. ...
The last several years have seen a significant increase in interest in big data across a range of scientific and engineering fields. Despite having several benefits and applications, big data still has some difficulties that must be overcome for a higher level of service, such as big data analytics, big data management, and big data privacy and security. Big data services and apps stand to greatly benefit from blockchain decentralisation and security features. In this article, we present an overview of blockchain for big data with an emphasis on current methods, possibilities, and upcoming trends. We begin by providing a succinct explanation of big data, blockchain, and the purpose of their integration. After that, we look at different types of blockchain assistance for big data, such as blockchain for security in big data collection, data privacy protection, storage, and collection. Next, we examine the latest work on the utilization of blockchain applications for big data across different industries, including smart grid apps and applications, smart city applications, and smart healthcare applications. A few illustrative blockchain-big data initiatives are given and discussed for a good understanding. Finally, difficulties and potential directions are examined to advance research in an exciting field.
... The article applies time series and regression analyses to understand the relationship between weather patterns and wind power output. Furthermore, in another research article [46], data analysis techniques for demand response in smart grids, such as clustering analysis, pattern recognition, and regression analysis used 5 of 34 to explore the use of data analysis to analyze energy consumption patterns, customer behavior, and grid conditions for effective demand response programs. ...
The rapid adoption of hydrogen as an eco-friendly energy source has necessitated the development of intelligent power management systems capable of efficiently utilizing hydrogen resources. However, guaranteeing the security and integrity of hydrogen-related data has become a significant challenge. This paper proposes a pioneering approach to ensure secure hydrogen data analysis through the integration of blockchain technology, enhancing trust, transparency, and privacy in handling hydrogen-related information. By combining blockchain with intelligent power management systems, the efficient utilization of hydrogen resources becomes feasible. The utilization of smart contracts and distributed ledger technology facilitates secure data analysis, real-time monitoring, prediction, and optimization of hydrogen-based power systems. The effectiveness and performance of the proposed approach are demonstrated through comprehensive case studies and simulations. Notably, our prediction models, including ABiLSTM, ALSTM, and ARNN, consistently delivered high accuracy with MAE values of approximately 0.154, 0.151, and 0.151, respectively, enhancing the security and efficiency of hydrogen consumption forecasts. The blockchain-based solution offers enhanced security, integrity, and privacy for hydrogen data analysis, thus contributing to the advancement of clean and sustainable energy systems. Additionally, the research identifies existing challenges and outlines potential future directions for further enhancing the proposed system. This study adds to the growing body of research on blockchain applications in the energy sector, with a specific focus on secure hydrogen data analysis and intelligent power management systems.
... Due to the widespread usage of shared networks and the resulting risks in smart grid computing systems, security risks have dramatically risen (Sahani et al., 2023). Ghorbanian et al., (2019) addressed the large data issues in smart power systems is extensively evaluated. The context and driving forces behind the big data paradigm in smart power systems are presented first, followed by an analysis of the significant problems with big data architectures, essential technologies, and standardizations in such systems. ...
Based on the continuous growth of the economy, widespread adoption of intermittent renewable energy sources, and extensive use of information and communication technologies, conventional electric power systems are no longer able to meet the enormous demands of the information age. Diverse renewable energy technologies have been quickly developed to address the energy issue and environmental damage. However, since renewable energy sources are unpredictable and erratic, the widespread use of different renewable energy technologies has consequently put significant strain on the security and dependability of conventional power networks. The Smart Grid (SG) is a modernized electrical network that makes use of cutting-edge communication, control, and information technology to facilitate the integration of renewable energy sources, increase energy efficiency, and improve dependability and security. The invention of computational modeling and evaluation methodologies for SG energy transmission and production networks is the main topic of the research. The Internet of Energy (IoE), which will eventually replace the conventional power production and distribution networks, increases the need to be familiar with the proper computing tools in order to conduct any future SG investigation. The software for simulation that is significant to the modeling and analysis of electrical power production, transmission, distribution, and related systems is examined in this research. The study's conclusions are anticipated to aid in the creation of power generation and transmission systems that are more effective, dependable, and sustainable.
... Real-time implementation can be achieved by Strom [247], Spark [198], and VoltDb. However, the system was implemented by Spark. ...
Smart Cities (SCs) have recently opened new lifestyles as they introduce effective approaches for improving urban management. These cities integrate Industry 4.0 technologies and new organizations into a social and technical system. A Smart City (SC) finally aims to integrate urban business, transportation, water, energy, and other subsystems by analyzing the data collected from sensors and Information and Communication Technology (ICT). In this regard, the current article proposes a hybrid structure based on Cloud Computing (CC), cloud processing, and internet technology to develop and manage SCs and urban planning. Moreover, the development and analysis of sensors to construct an SC, as well as supporting technologies, are addressed. Therefore, the present research is focused on defining and reviewing large data and objects of the Internet and CC to identify the current challenges and limitations. This research, thus, introduced a novel approach in the context of the Internet of objects, whose data are gathered from different geographic locations through devices and sensors to be used in a new system. CC services in a city can analyze and make smart decisions to optimally manage SCs and improve the welfare of citizens. This study consequently sets the path for further investigation into the problems and difficulties associated with Big Data (BD) applications in SCs.
... More specifically, smart grids constitute selfsufficient systems [18] that put emphasis on both prosumers and consumers to generate and distribute energy [2], use heterogeneous data and various data sources [19], enable renewable energy resources to be integrated into the grid [1], and can dynamically respond to various conditions and events and address issues that appear throughout the network and grid [7]. Smart grids are characterized as the next-generation power grid and are radically different from traditional grids as they capitalize on ICT to enable bidirectional and more efficient electricity delivery and information exchange [16] [20] and facilitate and improve energy generation, transmission, distribution, and control [21]. ...
The realization of sustainable development and sustainable development goals achievement are essential. Hence, the power sector digitalization is imminent. This bibliometric and mapping study aims to explore the use of artificial intelligence in smart grids and how the topic has evolved over the years. In total, ten research questions are set to be explored. The analysis includes 1,926 articles that were identified and retrieved from Scopus and Web of Science (WoS) over the period 2005 to 2022. The analysis includes the descriptive statistics of the related studies and the annual scientific production, the identification of the most relevant and impactful authors, articles, outlets, affiliations, and countries, and the examination of the most commonly used keywords. The most popular topics and the advancement of the research focus are also explored. The study examines the results, discusses the main findings, presents open issues, and suggests new research directions. The significant role of artificial intelligence in the realization of smart grids and the digitalization of the power sector to enable sustainable development and the achievement of sustainable development goals was evident.
... In this regard, AI and ML methods are being developed day by day. In addition, the exchange of information also has a high cyber security [56]. ...
... At the same time, as more intelligent devices have been connected to the power system, the grid has stored massive and high-dimensional operational data during the operation [5,6]. These data have the characteristics of 4Vs [7,8], which contain rich and valuable information and are closely related to the stable operation of the power system [9][10][11]. ...
Bulk power grid interconnection and the access of various smart devices make the current grid highly complex. Timely and accurately identifying the power grid operation state is crucial for monitoring the operation stability of the power grid. For this purpose, an evaluation method of the power grid operation state based on random matrix theory and qualitative trend analysis is proposed. This method constructs two evaluation indicators based on the operation data of the power grid, which cannot only find out whether the current state of the power grid is stable but can also find out whether there is a bad operation trend in the current power grid. Compared with the traditional method, this method analyzes the power grid’s operation state from the big data perspective. It does not need to consider the complex network structure and operation mechanism of the actual power grid. Finally, the effectiveness and feasibility of the method are verified by the simulations of the IEEE 118-bus system.
... • High speed data processing: In modern power grids, certain decisions or actions must be taken in a matter of milliseconds, e.g., fault clearing; hence, high data processing speeds are required. Advanced analytics must be employed to meet the real-time processing requirements of big data (Ghorbanian et al., 2019). • Considering the importance of data visualization to power systems operators, the difficulty in meaningfully representing multisource data is a crucial challenge. ...
... In order to handle such large sensor data, smart grids must have a reliable secure communication infrastructure. A high data rate requires wide bandwidth and that must be satisfied by the communication infrastructure [5]. Also, the infrastructure must be adaptive to changes [6]. ...
... The popularity of DG, based on renewable energy sources, modifies the traditional structure of the electric utility grid and opens the way for self-sustainable entities called microgrids [9,10]. Subsequently, the power industry model will be transformed into a system based on local self-balancing energy areas and smart grids [11,12]. It introduces the need for analyzing and simulating an increasing amount of data. ...
Power quality (PQ) becomes a more and more pressing issue for the operation stability of power systems with renewable energy sources. An important aspect of PQ monitoring of distribution networks is to compare the PQ indicators in different operating conditions. This paper evaluates the impact of a microgrid implementation in a real distribution network on power quality indicators at the point of common coupling in an LV network. The study includes a classical assessment of the long-term PQ parameters according to the EN 50160 standard, such as nominal frequency deviations, voltage RMS variations, voltage fluctuations (represented by long-term flicker severity), voltage unbalance and total harmonic distortion. The PQ evaluation is extended in statistical assessment based on cluster analysis. The case study contains 5 weeks of power quality observation results obtained at the assessment point in two different working conditions of the distribution system: before and after implementing the microgrid. The study allows establishing general conclusions regarding a microgrid interconnection in order not to exceed power quality limits and considering the influence of photovoltaic generation on power quality parameters.
... Big data analysis is utilised by the PMU for the purposes of dynamic model calibration, and state estimation, transmission grid visualisation [53]. Big data facilitates SG applications [48]. The topic of investigation in [49] is 5G in SGs. ...
The goals of "smart cities" include relieving the burden of growing urbanisation, cutting down on energy use, protecting the environment, boosting the local economy and people's standard of living, and facilitating wider access to and use of cutting-edge Information and Communication Technologies (ICT). In smart cities, ICT is essential for policymaking, decision making, plan execution, and the delivery of useful services. The primary goal of this analysis is to investigate the part that AI and machine learning play. Examples of ed tech include Deep Reinforcement Learning (DRL) and Machine Learning (ML). In a complicated smart city setting, the aforementioned methods can be used to develop the best possible rules. Smart transportation, cyber-security, energy-efficient usage of smart grids (SG), efficient use of Unmanned Aerial Vehicles (UAVs) to guarantee the best 5G and beyond 5G (B5G) communications, and a smart health monitoring are all discussed in length in this paper. Finally, we discuss the research challenges that have yet to be met and potential future research directions that could bring the concept of a "smart city" closer to fruition.
... In the above context, the authors in [8] enumerate how the use of big data analysis in conjunction with intelligent models helps to resolve the issue of processing these huge data in an SG. Various applications of big data in the SG perspective are listed in [9]. reat to the communication network in the form of covert data integrity assault (CDIA) can be detrimental to the reliability and safety of smart grid functionalities. ...
The smart grid is considered a conventional application domain of cyber-physical system (CPS) tools in the electrical utility industry. The physical system dynamics of SG with the assistance of CPS are generally controlled by connected sensors and controllers via a communication link. These CPSs, which rely heavily on an expansive communication network and intelligent computing algorithms, are susceptible to cyber-physical attacks and are also sensitive to various technical, economical, and social factors compromising their stability. Assessment and prediction of the stability of CPSs are very vital in this context. In this work, a novel optimized (memetic algorithm-based) extreme learning machine model for smart grid-CPS stability prediction has been proposed. Here, the teaching-learning-based optimization and simulated annealing techniques are used to design the memetic algorithm. The experimental result regarding the proposed model is then compared with other contemporary machine learning and deep learning models.
... In this aspect, Zerdoumi et al. (2018) illustrates the use of big data analysis along with intelligent computational models in resolving the issue of bulk data processing in a SG. Different possible applications of big data in SG context have been listed out in Ghorbanian et al. (2019). Covert data integrity assault (CDIA) poses a serious threat to the reliability and safety of smart grid functionalities, as they have the potential to infringe the traditional bad-data detectors used SG control points. ...
The cutting-edge electrical utility industry is quick advancing into a keen blend of best-in-class digital advancements with the actual foundation, famously alluded to as cyber-physical system (CPS). These CPSs, which depend entirely on an extensive intelligent computing algorithm-based communication network, are vulnerable to digital assaults and are likewise touchy to various technical and socioeconomic variables trading off its stability. Evaluation and forecast of the stability of these CPSs are fundamental in this unique situation. Figuring such countless variables while designing a framework of stability assessment is humanly inconceivable and hence deployment of cutting-edge computational methods, for example, the machine learning-based model, is found generally reasonable for such purposes. In this work, a novel improved genetic algorithm (GA)-based extreme learning machine (ELM) model for smart-grid-CPS stability forecast has been proposed. The exploratory outcome in regards to the proposed model is then contrasted with other contemporary AI and profound learning models.
... In this aspect, Zerdoumi et al. (2018) illustrates the use of big data analysis along with intelligent computational models in resolving the issue of bulk data processing in a SG. Different possible applications of big data in SG context have been listed out in Ghorbanian et al. (2019). Covert data integrity assault (CDIA) poses a serious threat to the reliability and safety of smart grid functionalities, as they have the potential to infringe the traditional bad-data detectors used SG control points. ...
The ability of the power system to withstand, respond, and recover from a catastrophic event is an important factor often used to define the resilience of a power system. Environmental threats and human threats, such as cyber security attacks, may trigger these types of incidents. Cost-benefit analysis (CBA) is a proven method to assess the economic feasibility of development interventions. The cost of doing any project can be compared by using CBA and by knowing their net benefit and efficiency. A flexible framework for cost-benefit analysis can assist in assessing and prioritizing investments to improve the resiliency of the energy system. This chapter deals with the economic approach for calculating the benefit and cost of any project. Cost-benefit analysis should be seen as a preferred choice by proving that benefit outweighs the cost and providing significance to the community. As benefit and cost are difficult to quantify with uncertainty, when comparing the project resilience option, a certain degree of ambiguity or sensitivity should be considered.
This research explores the characterization of data in time series in Smart Grids, considering the importance of data as a basis for information and knowledge. The analysis, based on real data from a Smart Grid, focused on quantities such as temperature, voltage and current. Characteristics such as stationarity, linearity, complexity, cyclicality, mutability and randomness were addressed. The application of these characteristics made it possible to identify specific patterns and behaviors in each piece of data. Stationarity, linearity, and randomness are properties that can vary over time, and it is crucial to analyze time series at different periods. In addition, additional Big Data characteristics, such as trueness, value, variability, and others, amplify the complexity of the analysis. The research provides relevant insights to understand and address the challenges in analyzing large volumes of smart power grid data.
Data plays a crucial role in understanding several problems, including those related to electrical micro, smart grids and power consumption. In addition to electrical meters, data can be sourced from different channels such as environmental conditions, user expertise, maintenance history, and inventory registries. This paper introduces a Python-based analysis tool designed to search for equipment categories that exhibit constant power consumption within the asset inventory database of the University of Campinas. The software effectively identifies and categorizes items as air conditioners, refrigerators, computers, uninterrupted power supplies and internet routers, providing detailed insights into their specific characteristics. The tool generates a comprehensive PDF report featuring item discrimination through values, charts, organized and university units. Additionally, the software incorporates identification item lists and logs, aiding in the identification of missing or mismatched data throughout the process. These reports has been utilized to establish internal guidelines for optimizing in power consumption and already help the university to improve its GreenMetric index.
Data plays a crucial role in understanding a scientific problem which includes those related to electrical micro, smart grids, and power consumption. In this scenario, one aspect that requires analysis is the consumption data that can be retrieve into the monthly utility invoices. However, the analysis of utility invoices in Brazil poses an unique challenge due to variations in invoice document formats across different services, utilities and contract types. This article addresses this challenge by developing a software based in Regular Expression to extract and standardized data from diverse invoice models. Through the establishment of a database and the use of a Business Intelligence platform, the retrieval and analysis of pertinent information have been significantly improved. The focus of this article is on understanding and extracting utility invoice data, including electricity, water, piped gas, and telephony services, which have direct or indirect impacts on natural resources and human well-being. The analysis is carried out on public buildings in Brazil, specifically those associated with the São Paulo Center for Energy Transition Studies (CPTEn). The data presented in this paper encompasses consumption invoices from three universities: the State University of Campinas (Unicamp), São Paulo State University campus of São João da Boa Vista (Unesp-SJBV), and the Federal University of Goiás (UFG). It is important to note that all the analyzed information is publicly available under Brazilian law number 12 527 from November 18, 2011.
In recent years, there has been a growing trend towards integrating data analytics into smart grids as a method of providing increased performance, optimised energy use, and deeper insights into network operations. The term “data analytics” refers to the process of collecting and analysing data from a wide variety of sources, including “smart metres,” “distribution grids,” and “industrial plants.” This information is then put to use in the development of models and algorithms that can assist in recognising and predicting patterns and trends in energy consumption. This may help to improve customer service while also lowering operational expenses and optimising energy efficiency. In addition, data analytics can be used to detect and prevent energy theft, minimise energy waste, detect system failures, and indicate areas in which improvements could be made. Smart grids have the potential to become energy networks that are more intelligent, efficient, and reliable if data analytics are utilised.
The rapid growth of the Internet of Things (IoT) has led to its widespread adoption in various industries, enabling enhanced productivity and efficient services. Integrating IoT systems with existing enterprise application systems has become common practice. However, this integration necessitates reevaluating and reworking current Enterprise Architecture (EA) models and Expert Systems (ES) to accommodate IoT and cloud technologies. Enterprises must adopt a multifaceted view and automate various aspects, including operations, data management, and technology infrastructure. Machine Learning (ML) is a powerful IoT and smart automation tool within EA. Despite its potential, a need for dedicated work focuses on ML applications for IoT services and systems. With IoT being a significant field, analyzing IoT‐generated data and IoT‐based networks is crucial. Many studies have explored how ML can solve specific IoT‐related challenges. These mutually reinforcing technologies allow IoT applications to leverage sensor data for ML model improvement, leading to enhanced IoT operations and practices. Furthermore, ML techniques empower IoT systems with knowledge and enable suspicious activity detection in smart systems and objects. This survey paper conducts a comprehensive study on the role of ML in IoT applications, particularly in the domains of automation and security. It provides an in‐depth analysis of the state‐of‐the‐art ML approaches within the context of IoT, highlighting their contributions, challenges, and potential applications.
Cryptocurrencies have been gaining popularity in India, but the lack of reliable and safe platforms for trading has been a significant concern. This research paper proposes a decentralized exchange (DEX) platform that enables Indian users to trade cryptocurrencies for Indian Rupees (INR). The system uses the Chainlink decentralized oracle network to convert INR to the equivalent ETH, the base currency for trading on the DEX platform. The proposed platform aims to provide secure and transparent trading to users, ensuring that transactions are executed instantly and at a fair market price. The paper outlines the technical details of the proposed DEX platform, including the integration with Chainlink oracles for price feeds and the development of smart contracts for trading. The platform utilizes the Ethereum blockchain and is built using Solidity, a programming language for smart contracts. Finally, the paper concludes with an evaluation of the proposed DEX platform, highlighting its potential benefits for Indian users and its potential impact on the broader cryptocurrency ecosystem in India. This research paper’s findings can guide the development of more secure and reliable DEX platforms in India and beyond.
Demand side management (DSM) has become one of the major concerns of the smart grids to cope with the penetration of renewable energy. The availability of new communication technologies can enhance the resilient operation of smart grids for DSM. However, the false injection data and adversarial attacks that are against the operation of signal analysis models represent the biggest challenge against the resiliency of energy management operations by deploying these technologies in the smart grids. In this regard, this paper proposes a new reliable industrial Internet of Things (IoT) architecture and deep convolution neural network (CNN) with an image processing strategy based on continuous wavelet transform (CWT) for DSM and resilience operation of energy management. The main contribution involved in this paper includes establishing a real-time signal processing model and developing an industrial IoT platform with CWT-based CNN; verifying the IoT architecture with different levels of adversarial attacks; providing a cybersecurity analysis for the smart buildings with DSM considering the device-level attacks, and developing defense strategies from the aspects of detection, mitigation, and prevention. In addition, the proposed deep CNN is designed with proper hyperparameters to counter adversarial attacks. The proposed CWT-based CNN performed the highest accuracy compared with different machine learning and deep learning models. Various testing scenarios are presented and executed to ensure and demonstrate the performance and robustness of the proposed method under different levels of adversarial attacks.
In today’s energy industry, the use of data analytics in modern digital energy system operation is an important research and innovation area. Data analysis has a key role in any modern industries and is a significant part of the optimal modern operation and planning in different industries, especially in smart power systems. This is a motivation for efficient data monitoring and processing methods to operate the digital energy system. This chapter aims to present algorithms and tools in the area of data analysis as well as the application of these tools to solve problems and challenges in modern electric power systems. In addition, basic concepts in data analysis methods, technical approaches, and research opportunities for analyzing energy data and its application in digital electric energy systems operation are discussed. Moreover, data security challenges, data management, and visualization with analysis of system input data are introduced.
Analysis of electric circuits in nonsinusoidal, nonlinear conditions has been a talking point for a long time, and there are different approaches to analysis methods.
Since its beginning and to date, the electric power system has been getting complex through the embedding of power electronics (PE) equipment, converters, and renewable energy (RE) sources. Electronic equipment has revolutionized the power system and brought numerous benefits to industrial applications. Still, unfortunately, this is at the cost of distortion in signals (voltage, current, power). This further necessitates understanding power flow in nonsinusoidal linear and nonlinear circuit conditions. A novel mathematical framework of analysis in such an environment can help to achieve a consensus regarding the standards that can comply with the well-known, established standards in sinusoidal conditions.
This chapter summarizes the application of geometric algebra (GA) as a mathematical tool for circuit analysis instead of complex algebra. Some basic examples with results are exhibited, attained with this tool, and compared with the harmonic domain (HD) to draw a comparative conclusion.
DESCRIPTION
The digitalization of power systems incorporating IOT and advanced metering infrastructures produces large volumes of data which contain hidden insights that can be extracted using appropriate data analysis tools and techniques. This chapter discusses data science along with its classifications. Specific applications of various classifications of data science in the energy industry are enumerated. Also discussed are the challenges of the Energy 4, especially in relation to data and artificial intelligence.
Considering the history of energy systems since the second industrial revolution, (Zhou et al., 2016a) identifies four development stages. These start from an early decentralized phase, pass through the well-established centralized phase, and evolve into a distributed phase characterized by the rapid development of renewable energy technologies. The fourth phase, which today is materializing, consists of a smart and connected energy system permeated by new technologies and reshaping society and businesses in various ways. Digitalization, decarbonization, and decentralization are three pillars of the ongoing transformations in the electricity industry. In this chapter, we first review some key electrical infrastructure futures envisioned in the recent literature (e.g., based on smart grid or super grid principles). We then discuss several analysis macroareas, which gained attention in the scientific community. These include the assessment of new technologies, the development of energy data analytics, modeling/simulation tools, and testbeds, as well as the exploration of broader socio-technical-economic perspectives on energy systems.
Addressing data anomalies (e.g., garbage data, outliers, redundant data, and missing data) plays a vital role in performing accurate analytics (billing, forecasting, load profiling, etc.) on smart homes’ energy consumption data. From the literature, it has been identified that the data imputation with machine learning (ML)-based single-classifier approaches are used to address data quality issues. However, these approaches are not effective to address the hidden issues of smart home energy consumption data due to the presence of a variety of anomalies. Hence, this paper proposes ML-based ensemble classifiers using random forest (RF), support vector machine (SVM), decision tree (DT), naive Bayes, K-nearest neighbor, and neural networks to handle all the possible anomalies in smart home energy consumption data. The proposed approach initially identifies all anomalies and removes them, and then imputes this removed/missing information. The entire implementation consists of four parts. Part 1 presents anomaly detection and removal, part 2 presents data imputation, part 3 presents single-classifier approaches, and part 4 presents ensemble classifiers approaches. To assess the classifiers’ performance, various metrics, namely, accuracy, precision, recall/sensitivity, specificity, and F1 score are computed. From these metrics, it is identified that the ensemble classifier “RF+SVM+DT” has shown superior performance over the conventional single classifiers as well the other ensemble classifiers for anomaly handling.
In the last years, the term Big Data (BD) is becoming ubiquitous in almost every scientific field given the information era we are living. In this respect, power systems is not the exception. This overflood of information can be overwhelming for the different energy stakeholders, nonetheless, numerous opportunities can also be derived from this BD. To do so, it is necessary to perform the corresponding knowledge extraction to this large data. This has been referred as Big Data Analytics (BDA) in the recent literature. In this respect, this work provides a comprehensive understanding of the attributes of BD in the power sector, along with the most representative tools, techniques, applications and challenges related to the incorporation of BDA in this field.
The Blockchain technology has revolutionized the digital currency space with the pioneering cryptocurrency platform named Bitcoin. From an abstract perspective, a blockchain is a distributed ledger capable of maintaining an immutable log of transactions happening in a network. In recent years, this technology has attracted significant scientific interest in research areas beyond the financial sector, one of them being the Internet of Things (IoT). In this context, the Blockchain is seen as the missing link towards building a truly decentralized, trustless and secure environment for the IoT and, in this survey, we aim to shape a coherent and comprehensive picture of the current state-of-the-art efforts in this direction. We start with fundamental working principles of blockchains and how blockchain-based systems achieve the characteristics of decentralization, security, and auditability. From there, we build our narrative on the challenges posed by the current centralized IoT models, followed by recent advances made both in industry and research to solve these challenges and effectively use blockchains to provide a decentralized, secure medium for the IoT.
Cyber security in the context of big data is known to be a critical problem and presents a great challenge to the research community. Machine learning algorithms have been suggested as candidates for handling big data security problems. Among these algorithms, support vector machines (SVMs) have achieved remarkable success on various classification problems. However, to establish an effective SVM, the user needs to define the proper SVM configuration in advance, which is a challenging task that requires expert knowledge and a large amount of manual effort for trial and error. In this work, we formulate the SVM configuration process as a bi-objective optimisation problem in which accuracy and model complexity are considered as two conflicting objectives. We propose a novel hyperheuristic framework for bi-objective optimisation that is independent of the problem domain. This is the first time that a hyper-heuristic has been developed for this problem. The proposed hyper-heuristic framework consists of a high-level strategy and low-level heuristics. The high-level strategy uses the search performance to control the selection of which low-level heuristic should be used to generate a new SVM configuration. The low-level heuristics each use different rules to effectively explore the SVM configuration search space. To address bi-objective optimisation, the proposed framework adaptively integrates the strengths of decomposition- and Pareto-based approaches to approximate the Pareto set of SVM configurations. The effectiveness of the proposed framework has been evaluated on two cyber security problems: Microsoft malware big data classification and anomaly intrusion detection. The obtained results demonstrate that the proposed framework is very effective, if not superior, compared with its counterparts and other algorithms.
Big data is considered to be the key to unlocking the next great waves of growth in productivity. The amount of collected data in our world has been exploding due to a number of new applications and technologies that permeate our daily lives, including mobile and social networking applications, and Internet of Thing (IoT)-based smart-world systems (smart grid, smart transportation, smart cities, etc.). With the exponential growth of data, how to efficiently utilize the data becomes a critical issue. This calls for the development of a big data market that enables efficient data trading. Via pushing data as a kind of commodity into a digital market, the data owners and consumers are able to connect with each other, sharing and further increasing the utility of data. Nonetheless, to enable such an effective market for data trading, several challenges need to be addressed, such as determining proper pricing for the data to be sold or purchased, designing a trading platform and schemes to enable the maximization of social welfare of trading participants with efficiency and privacy preservation, and protecting the traded data from being resold to maintain the value of the data. In this paper, we conduct a comprehensive survey on the lifecycle of data and data trading. To be specific, we first study a variety of data pricing models, categorize them into different groups, and conduct a comprehensive comparison of the pros and cons of these models. Then, we focus on the design of data trading platforms and schemes, supporting efficient, secure, and privacy-preserving data trading. Finally, we review digital copyright protection mechanisms, including Digital Copyright Identifier (DCI), Digital Rights Management (DRM), Digital Encryption, Watermarking, and others, and outline challenges in data protection in the data trading lifecycle. OAPA
The widespread popularity of smart meters enables an immense amount of fine-grained electricity consumption data to be collected. Meanwhile, the deregulation of the power industry, particularly on the delivery side, has continuously been moving forward worldwide. How to employ massive smart meter data to promote and enhance the efficiency and sustainability of the power grid is a pressing issue. To date, substantial works have been conducted on smart meter data analytics. To provide a comprehensive overview of the current research and to identify challenges for future research, this paper conducts an application-oriented review of smart meter data analytics. Following the three stages of analytics, namely, descriptive, predictive and prescriptive analytics, we identify the key application areas as load analysis, load forecasting, and load management. We also review the techniques and methodologies adopted or developed to address each application. In addition, we also discuss some research trends, such as big data issues, novel machine learning technologies, new business models, the transition of energy systems, and data privacy and security.
This paper presents a high-level overview of Emerging Services and Analytics in Cloud Computing virtual reality including the rationale, technologies involved, selected examples, and contributions. Examples in health care, business, climate change, and natural disaster have been demonstrated to support Emerging Services and Analytics by virtual reality, which can allow scientists to understand the complexity behind science and process thousands of data within seconds. Not only scientists with deep knowledge but also the general public can understand part of the outputs and illustrations from different types of examples and outputs from other disciplines. Businesses can achieve business agility, improvement in efficiency, monitoring the daily work, and analyzing key performance indicators up to date and in real time. We present the case of Emerging Services and Analytics by virtual reality to discuss its contributions and future directions.
A smart grid is an intelligent electricity grid that optimizes the generation, distribution and consumption of electricity through the introduction of Information and Communication Technologies on the electricity grid. In essence, smart grids bring profound changes in the information systems that drive them: new information flows coming from the electricity grid, new players such as decentralized producers of renewable energies, new uses such as electric vehicles and connected houses and new communicating equipments such as smart meters, sensors and remote control points. All this will cause a deluge of data that the energy companies will have to face. Big Data technologies offers suitable solutions for utilities, but the decision about which Big Data technology to use is critical. In this paper, we provide an overview of data management for smart grids, summarise the added value of Big Data technologies for this kind of data, and discuss the technical requirements, the tools and the main steps to implement Big Data solutions in the smart grid context.
Energy internet, as a major trend in power system, can provide an open framework for integrating equipments of energy generation, transmission, storage and consumption, etc., so that global energy can be managed and controlled efficiently by information and communication technologies. In this paper, we focus on the coordinated management of renewable and traditional energy, which is a typical issue on energy connections. We consider a conventional power system consisting of the utility company, the energy storage company, the microgrid, and electricity users. Firstly, we formulate the energy management problem as a three-stage Stackelberg game, and every player in the electricity market aims to maximize its individual payoff while guaranteeing the system reliability and satisfying users’ electricity demands. We employ the backward induction method to solve the three-stage non-cooperative game problem, and give the closed-form expressions of the optimal strategies for each stage. Next, we study the big data-based power generation forecasting techniques, and introduce a scheme of the wind power forecasting, which can assist the microgrid to make strategies. Furthermore, we prove the properties of the proposed energy management algorithm including the existence and uniqueness of Nash equilibrium and Stackelberg equilibrium. Simulation results show that accurate prediction results of wind power is conducive to better energy management.
Intelligent buildings are quickly becoming cohesive and integral inhabitants of cyberphysical ecosystems. Modern buildings adapt to internal and external elements and thrive on ever-increasing data sources, such as ubiquitous smart devices and sensors, while mimicking various approaches previously known in software, hardware, and bioinspired systems. This article provides an overview of intelligent buildings of the future from a range of perspectives. It discusses everything from the prospects of U.S. and world energy consumption to insights into the future of intelligent buildings based on the latest technological advancements in U.S. industry and government.
Energy is one of the most important parts in human life. As a significant application of energy, smart grid is a complicated interconnected power grid that involves sensors, deployment strategies, smart meters, and real-time data processing. It continuously generates data with large volume, high velocity, and diverse variety. In this paper, we first give a brief introduction on big data, smart grid, and big data application in the smart grid scenario. Then, recent studies and developments are summarized in the context of integrated architecture and key enabling technologies. Meanwhile, security issues are specifically addressed. Finally, we introduce several typical big data applications and point out future challenges in the energy domain.
Smart Grids are electric networks that employ advanced monitoring, control, and communication technologies to deliver reliable and secure energy supply, enhance operation efficiency for generators and distributors, and provide flexible choices for prosumers. Smart Grids are a combination of complex physical network systems and cyber systems that face many technological challenges. In this paper, we will first present an overview of these challenges in the context of Cyber-Physical Systems. We will then outline potential contributions that
Cyber-Physical Systems can make to Smart Grids, as well as the challenges that Smart Grids present to Cyber-Physical Systems research. Finally, implications of current technological advances to Smart Grids are outlined.
Despite being popularly referred to as the ultimate solution for all problems of our current electric power system, smart grid is still a growing and unstable concept. It is usually considered as a set of advanced features powered by promising technological solutions. In this paper, we describe smart grid as a socio-technical transition and illustrate the evolutionary path on which a smart grid can be realized. Through this conceptual lens, we reveal the role of big data, and how it can fuel the organic growth of smart grid. We also provide a rough estimate of how much data will be potentially generated from different data sources, which helps clarify the big data challenges during the evolutionary process.
In this paper, we address a big-data analysis method for estimating the driving range of an electric vehicle (EV), allowing drivers to overcome range anxiety. First, we present an estimating approach to project the life of battery pack for 1600 cycles (i.e., 8 years/160 000 km) based on the data collected from a cycle-life test. This approach has the merit of simplicity. In addition, it considers several critical issues that occur inside battery packs, such as the dependence of internal resistance and the state-of-health. Subsequently, we describe our work on driving pattern analysis of an EV, using a machine-learning approach, namely growing hierarchical self-organizing maps, to cluster the collected EV big data. This paper contains the analysis of energy consumption and driving range estimation for EVs, including powertrain simulation and driving behavior analysis. The experimental results, including both simulating battery degradation and analysis of driving behaviors, demonstrate a feasible solution for improving driving range estimation by the EV big data.
Model-based analysis tools, built on assumptions and simplifications, are difficult to handle smart grids with data characterized by volume, velocity, variety, and veracity (i.e. 4Vs data). This paper, using random matrix theory (RMT), motivates data-driven tools to perceive the complex grids in highdimension; meanwhile, an architecture with detailed procedures is proposed. In algorithm perspective, the architecture performs a high-dimensional analysis, and compares the findings with RMT predictions to conduct anomaly detections. Mean Spectral Radius (MSR), as a statistical indicator, is defined to reflect the correlations of system data in different dimensions. In management mode perspective, a group-work mode is discussed for smart grids operation. This mode breaks through regional limitations for energy flows and data flows, and makes advanced
big data analyses possible. For a specific large-scale zone-dividing system with multiple connected utilities, each site, operating under the group-work mode, is able to work out the regional MSR only with its own measured/simulated data. The large-scale interconnected system, in this way, is naturally decoupled from statistical parameters perspective, rather than from engineering models perspective. Furthermore, a comparative analysis of these
distributed MSRs, even with imperceptible different raw data, will produce a contour line to detect the event and locate the source. It demonstrates that the architecture is compatible with the block calculation only using the regional small database; beyond that, this architecture, as a data-driven solution, is sensitive to system situation awareness, and practical for real large-scale interconnected systems. Five case studies and their
visualizations validate the designed architecture in various fields of power systems. To our best knowledge, this study is the first attempt to apply big data technology into smart grids.
The performance of prediction models is often based on "abstract metrics"
that estimate the model's ability to limit residual errors between the observed
and predicted values. However, meaningful evaluation and selection of
prediction models for end-user domains requires holistic and
application-sensitive performance measures. Inspired by energy consumption
prediction models used in the emerging "big data" domain of Smart Power Grids,
we propose a suite of performance measures to rationally compare models along
the dimensions of scale independence, reliability, volatility and cost. We
include both application independent and dependent measures, the latter
parameterized to allow customization by domain experts to fit their scenario.
While our measures are generalizable to other domains, we offer an empirical
analysis using real energy use data for three Smart Grid applications:
planning, customer education and demand response, which are relevant for energy
sustainability. Our results underscore the value of the proposed measures to
offer a deeper insight into models' behavior and their impact on real
applications, which benefit both data mining researchers and practitioners.
The promise of data-driven decision-making is now being recognized broadly, and there is growing enthusiasm for the notion of "Big Data," including the recent announcement from the White House about new funding initiatives across different agencies, that target research for Big Data. While the promise of Big Data is real -- for example, it is estimated that Google alone contributed 54 billion dollars to the US economy in 2009 -- there is no clear consensus on what is Big Data. In fact, there have been many controversial statements about Big Data, such as "Size is the only thing that matters." In this panel we will try to explore the controversies and debunk the myths surrounding Big Data.
The fast-paced development of power systems necessitates smart grids to facilitate real-time control and monitoring with bidirectional communication and electricity flows. Future smart grids are expected to have reliable, efficient, secured, and cost-effective
power management with the implementation of distributed architecture. To focus on these requirements, we provide a comprehensive survey on different cloud computing applications for the smart grid architecture, in three different areas — energy management,
information management, and security. In these areas, the utility of cloud computing applications is discussed, while giving directions on future opportunities for the development of the smart grid. We also highlight different challenges existing in the conventional smart grid (without cloud application) that can be overcome using cloud. In this survey, we present a synthesized overview of the current state of research on smart grid development. We also identify the current research problems in the areas of cloud-based energy management, information management, and security in smart grid.
A smart grid is a new form of electricity network with high fidelity power-flow control, self-healing, and energy reliability and energy security using digital communications and control technology. To upgrade an existing power grid into a smart grid, it requires significant dependence on intelligent and secure communication infrastructures. It requires security frameworks for distributed communications, pervasive computing and sensing technologies in smart grid. However, as many of the communication technologies currently recommended to use by a smart grid is vulnerable in cyber security, it could lead to unreliable system operations, causing unnecessary expenditure, even consequential disaster to both utilities and consumers. In this paper, we summarize the cyber security requirements and the possible vulnerabilities in smart grid communications and survey the current solutions on cyber security for smart grid communications.
Advanced information and communication infrastructures are essential to successfully operate smart grids (SGs) and provide efficient, reliable, and sustainable electricity to the customers. After providing the background of the communication paradigm in SGs, a comprehensive survey of the major issues related to the architectures, the key technologies, and the requirements of the SG communication infrastructure are analyzed in this article. The role of cloud computing and Internet of Things in SGs is also discussed. Finally, the standardization, potential applications, and future fruitful research issues of the communication infrastructures of the SGs are classified and discussed.
Cloud-based content distribution networks (CDNs) consist of multiple servers that consume large amounts of energy. However, with the development of a cloud-based software defined network (SDN), a new paradigm of the virtual content distribution network (vCDN) has emerged. In an emerging cloud-based vCDN environment, the development and adjustment of vCDN components has become easier with the aid of SDN technology. This transformation provides the opportunity to use vCDNs to reduce energy consumption by adjusting the scale of the vCDN components. Energy costs can be reduced by deactivating the commercial servers carrying the software components of the vCDN, such as replica servers, the firewall or routers. In addition, the CDN requires a high service level agreement (SLA) to respond to clients' requests, potentially consuming large amounts of energy. In this research, we focus on the issue of the energy savings of a CDN in a cloud computing environment while maintaining a high quality of service (QoS). We propose an approximate algorithm termed max flow forecast (MFF) to determine the number of software components in the vCDN. Additionally, we use a real traffic trace from a website to assess our algorithm. The experimental results show that MFF can produce a larger energy reduction than the existing algorithms for an identical SLA. We fully justify our research as a good example for the emerging cloud.
Increased intelligence and automation in smart grid results in many heterogeneous applications benefiting from the Internet of Things, such as demand response, energy delivery efficiency/reliability, and fault recovery. However, vulnerabilities in smart grid arise due to public communication infrastructure and Internet-based protocols. To deal with security threats, energy big data should be thoughtfully stored and processed to extract critical information, and security and blackout warnings should be given in an early stage. This work gives a comprehensive tutorial and survey to highlight research challenges on the aforementioned issues in the Internet-of-Things-based smart grid. We demonstrate that a stealthy and blind energy big data attack can be launched using a replay scheme. Also, we elucidate an intuitive geometric viewpoint for this type of attack. The proposed attack can bypass bad data detection successfully using either DC or AC state estimation.
Electricity price forecasting is a significant part of smart grid because it makes smart grid cost efficient. Nevertheless, existing methods for price forecasting may be difficult to handle with huge price data in the grid, since the redundancy from feature selection cannot be averted and an integrated infrastructure is also lacked for coordinating the procedures in electricity price forecasting. To solve such a problem, a novel electricity price forecasting model is developed. Specifically, three modules are integrated in the proposed model. First, by merging of Random Forest (RF) and Relief-F algorithm, we propose a hybrid feature selector based on Grey Correlation Analysis (GCA) to eliminate the feature redundancy. Second, an integration of Kernel function and Principle Component Analysis (KPCA) is used in feature extraction process to realize the dimensionality reduction. Finally, to forecast price classification, we put forward a differential evolution (DE) based Support Vector Machine (SVM) classifier. Our proposed electricity price forecasting model is realized via these three parts. Numerical results show that our proposal has superior performance than other methods.
There are both economic and environmental urges for transition from the current outdated power grid to a sensor-embedded smart grid that monitors system stability, integrates distributed energy and schedules energy consumption for household users. Especially with the proliferation of intelligent measurement devices, exponential growth of data empowers this transition and brings new tools for the development of different applications in power system. Under this context, this paper presents a holistically overview on the state-of-the-art of big data technology in smart grid integration. First, the features of smart grid and the multisource of energy data are discussed. Then, this paper comprehensive summarizes the applications leveraged by big data in smart grid, which also contains some brand new applications with the latest big data technologies. Furthermore, some mainstream platforms and knowledge extraction techniques are looked to promote the big data insights. Finally, challenges and opportunities are pointed out in this paper as well.
Privacy preservation is one of the greatest concerns in big data. As one of extensive applications in big data, privacy preserving data publication (PPDP) has been an important research field. One of the fundamental challenges in PPDP is the trade-off problem between privacy and utility of the single and independent data set. However, recent research has shown that the advanced privacy mechanism, i.e., differential privacy, is vulnerable when multiple data sets are correlated. In this case, the trade-off problem between privacy and utility is evolved into a game problem, in which payoff of each player is dependent on his and his neighbors’ privacy parameters. In this paper, we firstly present the definition of correlated differential privacy to evaluate the real privacy level of a single data set influenced by the other data sets. Then, we construct a game model of multiple players, in which each publishes data set sanitized by differential privacy. Next, we analyze the existence and uniqueness of the pure Nash Equilibrium. We refer to a notion, i.e., the price of anarchy, to evaluate efficiency of the pure Nash Equilibrium. Finally, we show the correctness of our game analysis via simulation experiments.
As both the scale of mobile networks and the population of mobile users keep increasing, the applications of mobile social big data have emerged where mobile social users can use their mobile devices to exchange and share contents with each other. The security resource is needed to protect mobile social big data during the delivery. However, due to the limited security resource, how to allocate the security resource becomes a new challenge. Therefore, in this paper we propose a joint match-coalitional game based security-aware resource allocation scheme to deliver mobile social big data. In the proposed scheme, firstly a coalition game model is introduced for base stations (BSs) to form groups to provide both wireless and security resource, where the resource efficiency and profits can be improved. Secondly, a matching theory based model is employed to determine the selecting process between communities and the coalitions of BSs so that mobile social users can form communities to select the optimal coalition to obtain security resource. Thirdly, a joint matching-coalition algorithm is presented to obtain the stable security-aware resource allocation. At last, the simulation experiments prove that the proposal scheme outperforms other existing schemes.
Real-time monitoring and control of smart grids is critical to the enhancement of reliability and operational efficiency of power utilities. We develop a real-time anomaly detection framework, which can be built based upon smart meter data collected at the consumers’ premises. The model is designed to detect the occurrence of anomalous events and abnormal conditions at both lateral and customer levels. We propose a generative model for anomaly detection that takes into account the hierarchical structure of the network and the data collected from smart meters. We also address three challenges existing in smart grid analytics: (i) large-scale multivariate count measurements, (ii) missing points, and (iii) variable selection. We present the effectiveness of our approach with numerical experiments.
The development of smart grid brings great improvement in the efficiency, reliability, and economics to power grid. However, at the same time, the volume and complexity of data in the grid explode. To address this challenge, big data technology is a strong candidate for the analysis and processing of smart grid data. In this article, we propose a big data computing architecture for smart grid analytics, which involves data resources, transmission, storage, and analysis. In order to enable big data computing in smart grid, a communication architecture is then described consisting of four main domains. Key technologies to enable big-data-Aware wireless communication for smart grid are investigated. As a case study of the proposed architecture, we introduce a big-data-enabled storage planning scheme based on wireless big data computing. A hybrid approach is adopted for the optimization including GA for storage planning and a game theoretic inner optimization for daily energy scheduling. Simulation results indicate that the proposed storage planning scheme greatly reduce.
The key challenge for household load forecasting lies in the high volatility and uncertainty of load profiles. Traditional methods tend to avoid such uncertainty by load aggregation (to offset uncertainties), customer classification (to cluster uncertainties) and spectral analysis (to filter out uncertainties). This paper, for the first time, aims to directly learn the uncertainty by applying a new breed of machine learning algorithms – deep learning. However simply adding layers in neural networks will cap the forecasting performance due to the occurrence of overfitting. A novel pooling-based deep recurrent neural network (PDRNN) is proposed in this paper which batches a group of customers’ load profiles into a pool of inputs. Essentially the model could address the over-fitting issue by increasing data diversity and volume. This work reports the first attempts to develop a bespoke deep learning application for household load forecasting and achieved preliminary success. The developed method is implemented on Tensorflow deep learning platform and tested on 920 smart metered customers from Ireland. Compared with the state-of-art techniques in household load forecasting, the proposed method outperforms ARIMA by 19.5%, SVR by 13.1% and classical deep RNN by 6.5% in terms of RMSE.
We describe the background and an analytical framework for a mathematical optimization model for home energy management systems (HEMS) to manage electricity demand on the smart grid by efficiently shifting electricity loads of households from peak times to off-peak times. We illustrate the flexibility of the model by modularizing various available technologies such as plug-in electric vehicles, battery storage, and automatic windows. First, the analysis shows that the end-user can accrue economic benefits by shifting consumer loads away from higher-priced periods. Specifically, we assessed the most likely sources of value to be derived from demand response technologies. Therefore, wide adoption of such modeling could create significant cost savings for consumers. Second, the findings are promising for the further development of more intelligent HEMS in the residential sector. Third, we formulated a smart grid valuation framework that is helpful for interpreting the model's results concerning the efficiency of current smart appliances and their respective prices. Finally, we explain the model’s benefits, the major concerns when the model is applied in the real world, and the possible future areas that can be explored.
Advanced communications and data processing technologies bring great benefits to the smart grid. However, cyber-security threats also extend from the information system to the smart grid. The existing security works for smart grid focus on traditional protection and detection methods. However, a lot of threats occur in a very short time and overlooked by exiting security components. These threats usually have huge impacts on smart gird and disturb its normal operation. Moreover, it is too late to take action to defend against the threats once they are detected, and damages could be difficult to repair. To address this issue, this paper proposes a security situational awareness mechanism based on the analysis of big data in the smart grid. Fuzzy cluster based analytical method, game theory and reinforcement learning are integrated seamlessly to perform the security situational analysis for the smart grid. The simulation and experimental results show the advantages of our scheme in terms of high efficiency and low error rate for security situational awareness.
Growth of the internet, Lead to Cloud Computing, Mobile Network and Internet of Things increasing rapidly, big data is becoming a hot-spot in recent years. Data generated from many sources made huge demand for storing, managing, processing and querying on various stream. This gives rise to data processing in our daily life such as mobile devices, RFID and Wireless sensors, which aims at dealing with billions of users interactive data. At real time processing is intently needed integrated system. An entire system is built on one such application Storm, associated with Sql Stream. In this survey use this static data stream and some logs stream and also click stream to show some basic result. To ensure the practical applicability and high efficiency, A Simulation System is established and shown acceptable performance in various expressions using data sheet. It proved that data analysis system for stream and real time processing based on storm can be used in various computing environment.
The limited available fossil fuels and the call for sustainable environment have brought about new technologies for the high efficiency in the use of fossil fuels and introduction of renewable energy. Smart grid is an emerging technology that can fulfill such demands by incorporating advanced information and communications technology (ICT). The pervasive deployment of the advanced ICT, especially the smart metering, will generate big energy data in terms of volume, velocity, and variety. The generated big data can bring huge benefits to the better energy planning, efficient energy generation, and distribution. As such data involve end users' privacy and secure operation of the critical infrastructure, there will be new security issues. This paper is to survey and discuss new findings and developments in the existing big energy data analytics and security. Several taxonomies have been proposed to express the intriguing relationships of various variables in the field.
Nowadays, there are two significant tendencies, how to process the enormous amount of data, big data, and how to deal with the green issues related to sustainability and environmental concerns. An interesting question is whether there are inherent correlations between the two tendencies in general. To answer this question, this paper firstly makes a comprehensive literature survey on how to green big data systems in terms of the whole life cycle of big data processing, and then this paper studies the relevance between big data and green metrics and proposes two new metrics, effective energy efficiency and effective resource efficiency in order to bring new views and potentials of green metrics for the future times of big data.
The ν-support vector classification has the advantage of using a regularization parameter ν to control the number of support vectors and margin errors. Recently, a regularization path algorithm for ν-support vector classification (ν-SvcPath) suffers exceptions and singularities in some special cases. In this brief, we first present a new equivalent dual formulation for ν-SVC and, then, propose a robust ν-SvcPath, based on lower upper decomposition with partial pivoting. Theoretical analysis and experimental results verify that our proposed robust regularization path algorithm can avoid the exceptions completely, handle the singularities in the key matrix, and fit the entire solution path in a finite number of steps. Experimental results also show that our proposed algorithm fits the entire solution path with fewer steps and less running time than original one does.
The integration of the distributed generation (DG) systems into the power grid is a significant issue to provide a reliable operation of the power system. DG systems must meet some technical requirements to achieve a successful grid connection. Islanding is also a vital issue for a reliable integration of DG systems with the grid. There have been many islanding detection methods researched in the literature, but most of them have some boundaries related to the local load and the inverter. In this study, a new remote islanding detection method is introduced for a developed wind–solar hybrid power plant, and a practical solution is researched by classifying the current methods. The proposed method monitors and controls the grid, local load and the output of the PV inverter in real time with the communication of circuit breakers. The proposed remote control system detects the changes in the currents of the circuit breakers, frequency and the active powers by checking the defined threshold values at all electrical branches of the hybrid DG system. When the breaker current goes to zero, or they are under/over defined threshold values, the circuit breakers are tripped by using a real-time control system that is developed with Labview. The proposed method also checks the frequency, active powers, and reactive powers with the currents in real-time, so it is independent of the load, and it is not inverter resident. Islanding detection time is just a cycle, and it is a considerably short response time according to the current standards. Non-detection zone (NDZ) is also zero in the proposed method. The experimental results prove that the developed remote islanding detection method is easily implemented in wind–solar DG systems, and it is also suitable for real system applications.
2 Abstract: Growth of the internet, Lead to Cloud Computing, Mobile Network and Internet of Things increasing rapidly, big data is becoming a hot-spot in recent years. Data generated from many sources made huge demand for storing, managing, processing and querying on various stream. This gives rise to data processing in our daily life such as mobile devices, RFID and Wireless sensors, which aims at dealing with billions of users interactive data. At the same time real time processing is intently needed in integrated system. An entire system is built on one such application Storm, associated with Sql Stream. To ensure the practical applicability and high efficiency, A Simulation System is established and shown acceptable performance in various expressions using data sheet. It proved that data analysis system for stream and real time processing based on storm can be used in various computing environment.
In this letter, we present an algorithm for rotor angle stability monitoring of power system in real-time. The proposed algorithm is model-free and can make use of high resolution phasor measurement units (PMUs) data to provide reliable, timely information about the system's stability. The theoretical basis behind the proposed algorithm is adopted from dynamical systems theory. In particular, the algorithm approximately computes the system's Lyapunov exponent (LE), thereby measuring the exponential convergence or divergence rate of the rotor angle trajectories. The LE serves as a certificate of stability where the positive (negative) value of the LE implies exponential divergence (convergence) of nearby system trajectories, hence, unstable (stable) rotor angle dynamics. We also show the proposed model-free algorithm can be used for the identification of the coherent sets of generators. The simulation results are presented to verify the developed results in the paper on the modified IEEE 162-bus system.
Smart sensor networks provide numerous opportunities for smart grid applications including power monitoring, demand-side energy management, coordination of distributed storage, and integration of renewable energy generators. Because of their low cost and ease-of-deployment, smart sensor networks are likely to be used on a large scale in future of smart power grids. The result is a huge volume of different variety of data sets. Processing and analyzing these data reveals deeper insights that can help expert to improve the operation of power grid to achieve better performance. The technology to collect massive amounts of data is available today, but managing the data efficiently and extracting the most useful information out of it remains a challenge. This paper discusses and provides recommendations and practices to be used in the future of smart grid and Internet of things. We explore the different applications of smart sensor networks in the domain of smart power grid. Also we discuss the techniques used to manage big data generated by sensors and meters for application processing.
Big Data concerns massive, heterogeneous, autonomous sources with distributed and decentralized control. These characteristics make it an extreme challenge for organizations using traditional data management mechanism to store and process these huge datasets. It is required to define a new paradigm and re-evaluate current system to manage and process Big Data. In this paper, the important characteristics, issues and challenges related to Big Data management has been explored. Various open source Big Data analytics frameworks that deal with Big Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed.
Support vector ordinal regression (SVOR) is a popular method to tackle ordinal regression problems. However, until now there were no effective algorithms proposed to address incremental SVOR learning due to the complicated formulations of SVOR. Recently, an interesting accurate on-line algorithm was proposed for training ν-support vector classification ( ν-SVC), which can handle a quadratic formulation with a pair of equality constraints. In this paper, we first present a modified SVOR formulation based on a sum-of-margins strategy. The formulation has multiple constraints, and each constraint includes a mixture of an equality and an inequality. Then, we extend the accurate on-line ν-SVC algorithm to the modified formulation, and propose an effective incremental SVOR algorithm. The algorithm can handle a quadratic formulation with multiple constraints, where each constraint is constituted of an equality and an inequality. More importantly, it tackles the conflicts between the equality and inequality constraints. We also provide the finite convergence analysis for the algorithm. Numerical experiments on the several benchmark and real-world data sets show that the incremental algorithm can converge to the optimal solution in a finite number of steps, and is faster than the existing batch and incremental SVOR algorithms. Meanwhile, the modified formulation has better accuracy than the existing incremental SVOR algorithm, and is as accurate as the sum-of-margins based formulation of Shashua and Levin.
Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.
In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.
