Table 3 - available via license: CC BY
Content may be subject to copyright.
Source publication
Energy efficiency in a data center is a challenge and has garnered researchers interest. In this study, we addressed the energy efficiency issue of a small scale data center by utilizing Single Board Computer (SBC)-based clusters. A compact layout was designed to build two clusters using 20 nodes each. Extensive testing was carried out to analyze t...
Contexts in source publication
Context 1
... the logs, the upper-bound wattage usage within a period of 23 h was taken as power consumption in the idle mode as well as the stress mode. Table 3 shows the power consumption for DM-Clusters in idle and stress modes. ...
Context 2
... approximation of energy consumption cost per year (C y ) can be given by Equation (1), where E is the specific power consumption for an event for 24 h a day and 365.25 days per year. The approximate cost for all the clusters computed based on values given in Table 3, whereas the cost per kilowatt-hour (P) was assumed to be 0.05 US$. The Bolzano Experiment [16] reports raspberry Pi cluster built using Raspberry Pi Model B (first generation) where each node is consuming 3 Watts in stress mode. ...
Context 3
... the computation of power consumption, we assumed max power utilization (stress mode) for each job, during a test run in the clusters. Based on the power consumption of each cluster and the dollar cost of maintaining the clusters (given in Table 3), a summary of average execution times, energy consumption and cost of running various benchmark tasks is presented in Table 9. Figure 8a shows the energy consumption (in watts) for all Hadoop benchmarks with lowest workloads. Although the power consumption of RPi Cluster is the lowest, the overall energy consumption by RPi Cluster is the highest compared to Xu20 and HDM Clusters due to the time inefficiency in job completion. ...
Similar publications
![Fig. 15.1 Role of cloud infrastructure in big data analysis [2]](profile/Basit-Qureshi-3/publication/333931578/figure/fig1/AS:819243130044416@1572334191299/Role-of-cloud-infrastructure-in-big-data-analysis-2_Q320.jpg)
In recent times, the commodity Single Board Computers (SBCs) have now become sufficiently powerful that they can run standard operating systems and mainstream workloads. In this chapter, we investigate the design and implementation of Single Board Computer (SBC)-based Hadoop clusters. We provide a compact design layout and build two clusters each u...
Physical data layout is an important performance factor for modern databases. Clustering, i.e., storing similar values in proximity, can lead to performance gains in several ways. We present an automated model to determine beneficial clustering columns and a clustering algorithm for the column-oriented, memory-resident database Hyrise. To automatic...
Citations
... First released in 2012, they are cost-effective, energy efficient, and widely accessible and have been used in various studies. A major drawback with earlier generation RPis was the computational capacity, as highlighted in our earlier work in [12]. With newer models 3B+, 4B, and the fifth generation, the use of improved onboard processors has significantly improved the performance of individual SBCs. ...
... It was able to handle up to two containers per node/device. In earlier works [5,12,19], the researchers note that the native YARN settings do discern the limited capabilities of these devices. When the number of containers exceeds two on an SBC node, it can overwhelm the task queue within the scheduler. ...
... Computation 2024, 12, 96 ...
Single-board computers (SBCs) are emerging as an efficient and economical solution for fog and edge computing, providing localized big data processing with lower energy consumption. Newer and faster SBCs deliver improved performance while still maintaining a compact form factor and cost-effectiveness. In recent times, researchers have addressed scheduling issues in Hadoop-based SBC clusters. Despite their potential, traditional Hadoop configurations struggle to optimize performance in heterogeneous SBC clusters due to disparities in computing resources. Consequently, we propose modifications to the scheduling mechanism to address these challenges. In this paper, we leverage the use of node labels introduced in Hadoop 3+ and define a Frugality Index that categorizes and labels SBC nodes based on their physical capabilities, such as CPU, memory, disk space, etc. Next, an adaptive configuration policy modifies the native fair scheduling policy by dynamically adjusting resource allocation in response to workload and cluster conditions. Furthermore, the proposed frugal configuration policy considers prioritizing the reduced tasks based on the Frugality Index to maximize parallelism. To evaluate our proposal, we construct a 13-node SBC cluster and conduct empirical evaluation using the Hadoop CPU and IO intensive microbenchmarks. The results demonstrate significant performance improvements compared to native Hadoop FIFO and capacity schedulers, with execution times 56% and 22% faster than the best_cap and best_fifo scenarios. Our findings underscore the effectiveness of our approach in managing the heterogeneous nature of SBC clusters and optimizing performance across various hardware configurations.
... One major challenge is the significant variance in computational capabilities among frugal nodes, which can lead to uneven workload distribution and resource contention. Frugal nodes typically have limited CPU processing power, memory, and storage, which can constrain the types and sizes of tasks they can effectively execute [17][18][19]. Moreover, these nodes may be deployed in edge or remote locations with unreliable network connectivity, posing challenges for communication and data transfer between nodes [20]. ...
... Taking inspiration from previous benchmark studies [10,12,16,18,27,29,30,38], we select wordcount and terasort workloads for the evaluation of AMS-ERA. ...
Efficient resource allocation is crucial in clusters with frugal Single-Board Computers (SBCs) possessing limited computational resources. These clusters are increasingly being deployed in edge computing environments in resource-constrained settings where energy efficiency and cost-effectiveness are paramount. A major challenge in Hadoop scheduling is load balancing, as frugal nodes within the cluster can become overwhelmed, resulting in degraded performance and frequent occurrences of out-of-memory errors, ultimately leading to job failures. In this study, we introduce an Adaptive Multi-criteria Selection for Efficient Resource Allocation (AMS-ERA) in Frugal Heterogeneous Hadoop Clusters. Our criterion considers CPU, memory, and disk requirements for jobs and aligns the requirements with available resources in the cluster for optimal resource allocation. To validate our approach, we deploy a heterogeneous SBC-based cluster consisting of 11 SBC nodes and conduct several experiments to evaluate the performance using Hadoop wordcount and terasort benchmark for various workload settings. The results are compared to the Hadoop-Fair, FOG, and IDaPS scheduling strategies. Our results demonstrate a significant improvement in performance with the proposed AMS-ERA, reducing execution time by 27.2%, 17.4%, and 7.6%, respectively, using terasort and wordcount benchmarks.
... One major challenge is the significant variance in computational capabilities among frugal nodes, which can lead to uneven workload distribution and resource contention. Frugal nodes typically have limited CPU processing power, memory, and storage, which can constrain the types and sizes of tasks they can effectively execute [18][19][20]. Moreover, these nodes may be deployed in edge or remote locations with unreliable network connectivity, posing challenges for communication and data transfer between nodes [21]. ...
... Taking inspiration from previous benchmark studies [11,13,17,19,28,31,32,42], we select wordcount and terasort workloads for evaluation of AMS-ERA. ...
Efficient resource allocation is crucial in clusters with frugal Single-Board Computers (SBCs) possessing limited computational resources. These clusters are increasingly being deployed in edge computing environments in resource-constrained settings where energy efficiency and cost-effectiveness are paramount. A major challenge in Hadoop YARN scheduling is load-balancing, as frugal nodes within the cluster can become overwhelmed, resulting in degraded performance and frequent occurrences of out-of-memory errors, ultimately leading to job failures. In this study, we introduce an Adaptive Multi-criteria Selection for Efficient Resource Allocation (AMS-ERA) in Frugal Heterogeneous Hadoop Clusters. Our criterion considers CPU, memory and disk requirements for jobs and aligns the requirements with available resources in the cluster for optimal resource allocation. To validate our approach, we deploy a heterogeneous SBC-based cluster consisting of 11 SBC nodes and conduct several experiments to evaluate the performance using Hadoop wordcount and terasort benchmark for various workload settings. The results are compared to the Hadoop-Fair, FOG and IDaPS scheduling strategies. Our results demonstrate a significant improvement in performance with the proposed AMS-ERA, reducing execution time by 27.2%, 17.4% and 7.6% respectively using terasort and wordcount benchmarks.
... Its distributed computing model enhances energy efficiency by enabling parallel processing of data across multiple nodes within a cluster. Lately, researchers in [9][10][11][12][13][14] have directed their attention towards achieving energy-efficient remote data processing through the utilization of clusters comprised of single-board computers (SBCs) like Raspberry Pi, coupled with the Hadoop framework for handling large-scale data processing tasks in various context including agriculture, smart cities, smart homes, healthcare etc. Qureshi et. al. in [11] developed a heterogenous cluster of 20 SBCs including Raspberry Pis and Ordoid Xu-4 for data analytics using Hadoop. ...
... Lately, researchers in [9][10][11][12][13][14] have directed their attention towards achieving energy-efficient remote data processing through the utilization of clusters comprised of single-board computers (SBCs) like Raspberry Pi, coupled with the Hadoop framework for handling large-scale data processing tasks in various context including agriculture, smart cities, smart homes, healthcare etc. Qureshi et. al. in [11] developed a heterogenous cluster of 20 SBCs including Raspberry Pis and Ordoid Xu-4 for data analytics using Hadoop. They conduct various experiments to analyze the performance and energy efficiency of the cluster for workloads of various sizes. ...
... First released in 2012, are cost-effective, energy efficient and are widely accessible and have been used in various studies. A major drawback with earlier generation RPi was the computational capacity as highlighted in our earlier work in [11]. With newer models 3B+, 4B and 5th generation, the use of improved on-board processors has significantly improved the performance of individual SBCs. ...
In the dynamic landscape of sustainable computing, use of edge devices is paramount for reducing the need for large-scale centralized data centers. By processing data locally, edge devices minimize the energy-intensive computing in data centers, improving the overall performance, cost-effectiveness whereas reducing the environmental impact. Edge devices may constitute edge clusters composed of resource frugal Single Board Computers (SBC) such as Raspberry Pi etc. The small form-factor and energy efficiency of these computers makes them ideal for processing large data on the edge. Despite their potential, traditional Hadoop configurations struggle to optimize performance in heterogeneous SBC clusters due to disparities in computing resources. Consequently, we propose modifications to the Yet Another Resource Negotiator (YARN) scheduling mechanism to address these challenges. Our proposed changes include the introduction of a Frugality Index and an adaptiveConfig policy. The Frugality Index categorizes SBC nodes based on their capabilities, enabling intelligent resource allocation. The adaptiveConfig policy dynamically adjusts resource allocation in response to workload and cluster conditions, enhancing system efficiency. Additionally, we introduce a fetch_threshold for reduce tasks to improve task prioritization based on locality and data processing efficiency. We evaluate our approach using a 13-node SBC cluster and conduct experiments with CPU-intensive and IO-intensive Hadoop benchmarks. The results demonstrate significant performance improvements compared to native YARN settings, with execution times 4.7 times faster than the worst_native and 1.9 times faster than the best_native scenarios. Furthermore, the proposed adaptiveConfig policy implementing the frugality index and a fetch_threshold outperforms the native YARN by 5.86 times and 1.79 times in Terasort and wordcount executions respectively. Our findings underscore the effectiveness of our approach in managing the heterogeneous nature of SBC clusters and optimizing performance across various hardware configurations. The adaptive policies prove well-suited to the frugal SBC-cluster context, yielding enhanced outcomes and paving the way for sustainable big data processing initiatives.
... And achieve high detection accuracy in which objects accomplish with intra-class variations, illuminations, and environmental disturbances. Furthermore, an algorithm must be able to achieve high efficiency on limited memory, speed, and computing capabilities on low-end mobile devices [16]- [18]. ...
span lang="EN-US">Implementation of artificial intelligence tends to be portable, mobile and embeds in embedded computer system (EBD). EBD is a special-purpose computer with limited capacity in a small-form size. Deep learning (DL) had known as cutting edges for object recognition. With DL, object feature extraction analysis is omitted. DL requires large computing resources and capacity. Implement DL algorithm on EBD goal to achieves high detection accuracy and high-efficiency resources. Hence, be able to cope with intra-class variations, and image disturbances. By those challenges and limitations, this study reports the performance of EBD to recognize an object which has high variations in their class, through an optimal raw-input dataset. The raw-input dataset performed optimization process with a supervisor. Yield is the proper optimal input dataset in size. The performance results observed begin from training dataset until evaluation stage of DL. The comparison performs in efficiency resources, loss, validation-loss, timesteps, and detection accuracy by multiclass confusion matrix analysis. This study shows through this purpose method efficient resources are highly archived. Shorter timesteps ensure training stage is successful, and detection accuracy is perfectly archived. In addition, this study proves DL method archived great performances in classifying object that has identical structure.</span
... Typically, when innovative educational technologies are introduced in the classroom [17] [20] [22] [21] [29], they can pose several challenges to traditional teaching and learning methods. Managing these challenges becomes the responsibility of education practitioners and policymakers [27]. ...
... Typically, when innovative educational technologies are introduced in the classroom [27][28][29][30][31], it can pose several challenges to the traditional teaching and learning methods. Managing these challenges becomes the responsibility of education practitioners and policymakers [33]. ...
The application of Artificial intelligence for teaching and learning in the academic sphere is a trending subject of interest in the computing education. ChatGPT, as an AI-based tool, provides various advantages, such as heightened student involvement, cooperation, accessibility and availability. This paper addresses the prospects and obstacles associated with utilizing ChatGPT as a tool for learning and assessment in undergraduate Computer Science curriculum in particular to teaching and learning fundamental programming courses. Students having completed the course work for a Data Structures and Algorithms (a sophomore level course) participated in this study. Two groups of students were given programming challenges to solve within a short period of time. The control group (group A) had access to text books and notes of programming courses, however no Internet access was provided. Group B students were given access to ChatGPT and were encouraged to use it to help solve the programming challenges. The challenge was conducted in a computer lab environment using PC2 environment. Each team of students address the problem by writing executable code that satisfies certain number of test cases. Student teams were scored based on their performance in terms of number of successful passed testcases. Results show that students using ChatGPT had an advantage in terms of earned scores, however there were inconsistencies and inaccuracies in the submitted code consequently affecting the overall performance. After a thorough analysis, the paper's findings indicate that incorporating AI in higher education brings about various opportunities and challenges.
... There is a growing amount of work related to the energy efficiency and performance evaluation of computing systems, mainly focused on aspects related to scheduling and distribution of tasks in clusters [27][28][29][30][31], data centers, and systems in the cloud [32][33][34][35][36]. Unlike the works cited, in this paper the analysis focuses on measuring the energy consumption of personal computers, using the Linpack benchmark to evaluate performance. The use of Linpack allows direct comparison of the energy efficiency figures obtained with those of the GREEN500 ranking, which is the list of the 500 most powerful computers and servers in the world. ...
The demand for electricity related to Information and Communications Technologies is constantly growing and significantly contributes to the increase in global greenhouse gas emissions. To reduce this harmful growth, it is necessary to address this problem from different perspectives. Among these is changing the computing scale, such as migrating, if possible, algorithms and processes to the most energy efficient resources. In this context, this paper explores the possibility of running scientific and engineering programs on personal computers and compares the obtained power efficiency on these systems with that of mainframe computers and even supercomputers. Anecdotally, this paper also shows how the power efficiency obtained for the same workloads on personal computers is similar to that obtained on supercomputers included in the Green500 ranking
... Once we have a good understanding of the energy usage patterns on a cluster of small board computers, we can then extrapolate and generalize these patterns to other cluster architectures. Others have used the same approach of using small board computers for similar types of experiments (e.g., [17,18]). ...
Scientific problems can be formulated as workflows to allow them to take advantage of cluster computing resources. Generally, the assumption is that the greater the resources dedicated to completing these tasks the better. This assumption does not take into account the energy cost of performing the computation and the specific characteristics of each workflow. In this paper, we present a unique approach to evaluating the energy consumption of scientific workflows on compute clusters. Two workflows from different domains, Astronomy and Bioinformatics, are presented and their execution is analyzed on a cluster of low powered small board computers. The paper presents a theoretical analysis of an energy-aware execution of workflows that can reduce the energy consumption of workflows by up to 68% compared to normal execution. We demonstrate that there are limitations to the benefits of increasing cluster sizes and there are trade-offs when considering energy vs. performance of the workflows and that the performance and energy consumption of any scientific workflow is heavily dependent on its underlying structure. The study concludes that the energy consumption of workflows can be optimized to improve both aspects of the workflow and motivates the development of an energy-aware scheduler.
... They reported metrics such as the performance in GFLOPS for different cluster sizes and memory utilization, the power efficiency in GFLOPS/Watt, and the value for money in GFLOPS/$. Qureshi and Koubaa [30] created three clusters: (1) a 20-node cluster based on the RPi 2 Model B, (2) a 20-node cluster based on the Odroid XU-4, and (3) a 4-node cluster made of regular PCs. The authors did extensive testing to analyze their performance using popular benchmarks for task execution time, memory/storage utilization, and energy consumption. ...
