Content uploaded by Gregor Ulm
Author content
All content in this area was uploaded by Gregor Ulm on Dec 11, 2018
Content may be subject to copyright.
A preview of the PDF is not available
A Performance Evaluation of Federated Learning Algorithms
Abstract and Figures
Federated learning is an approach to distributed machine learning where a global model is learned by aggregating models that have been trained locally on data-generating clients. Contrary to centralized optimization, clients can be very large in number and face challenges of data and network heterogeneity. Examples of clients include smartphones and connected vehicles, which highlights the practical relevance of federated learning. We benchmark three federated learning algorithms and compare their performance against a centralized approach where data resides on the server. The algorithms Federated Averaging (FedAvg), Federated Stochastic Variance Reduced Gradient, and CO-OP are evaluated on the MNIST dataset, using both i.i.d. and non-i.i.d. partitionings of the data. Our results show that FedAvg achieves the highest accuracy among the federated algorithms, regardless of how data was partitioned. Our comparison between FedAvg and centralized learning shows that they are practically equivalent when i.i.d. data is used. However, the centralized approach outperforms FedAvg with non-i.i.d. data.
Figures - uploaded by Gregor Ulm
Author content
All figure content in this area was uploaded by Gregor Ulm
Content may be subject to copyright.
... while Randomly select ε < ε h and V h f ̸ = ∅ do 34: n ′ = arg max n O k (n ∈ ℵ n',k V h f ) , and ρ train n ′ ,k = 1; 35: ...
... We use the MNIST and CIFAR-10 datasets to evaluate our proposed schemes. MNIST is a large database of handwritten digits and CIFAR-10 is a collection of 60,000 32 × 32 color images, which are commonly used for training various image-processing systems and have been widely leveraged for the performance evaluation of FL schemes [35]. We use the these two datasets instead of IoT images collected through camera sensors to test our designed schemes. ...
The booming growth of the internet of things has brought about widespread deployment of devices and massive amounts of sensing data to be processed. Federated learning (FL)-empowered mobile edge computing, known for pushing artificial intelligence to the network edge while preserving data privacy in learning cooperation, is a promising way to unleash the potential information of the data. However, FL’s multi-server collaborative operating architecture inevitably results in communication energy consumption between edge servers, which poses great challenges to servers with constrained energy budgets, especially wireless communication servers that rely on battery power. The device-to-device (D2D) communication mode developed for FL turns high-cost and long-distance server interactions into energy-efficient proximity delivery and multi-level aggregations, effectively alleviating the server energy constraints. A few studies have been devoted to D2D-enabled FL management, but most of them have neglected to investigate server computing power for FL operation, and they have all ignored the impact of dataset characteristics on model training, thus failing to fully exploit the data processing capabilities of energy-constrained edge servers. To fill this gap, in this paper we propose a D2D-assisted FL mechanism for energy-constrained edge computing, which jointly incorporates computing power allocation and dataset correlation into FL scheduling. In view of the variable impacts of computational power on model accuracy at different training stages, we design a partite graph-based FL scheme with adaptive D2D pairing and aperiodic variable local iterations of heterogeneous edge servers. Moreover, we leverage graph learning to exploit the performance gain of the dataset correlation between the edge servers in the model aggregation process, and we propose a graph-and-deep reinforcement learning-based D2D server pairing algorithm, which effectively reduces FL model error. The numerical results demonstrate that our designed FL schemes have great advantages in improving FL training accuracy under edge servers’ energy constraints.
... This cyclical process continues until the model convergence criteria are satisfied. FedAvg harnesses the collaborative potential of decentralized data sources while safeguarding the privacy of individual clients within the federated learning framework [12] [13]. ...
... FL allows participants to jointly train a model without any direct data sharing. The central server in the FL aggregates the updates received from multiple clients using some popular aggregation algorithms like FedAvg [39] to compute a global model as follows: ...
Generative Adversarial Network (GAN) exhibited significant capabilities in many applications including image enhancement and manipulation, language translation, generating images/videos from text, creating art and music, and so on. However, train ing GANs using large datasets remains highly computationally intensive for most of the standalone systems. Additionally, standalone GANs often exhibit poor synchronization between their generator and discriminator with unstable training , poor convergence along with a large number of mode collapse s and vanishing / exploding gradients. Standalone GANs also failed to learn in a decentralized environment, where the data is distributed among several client machines. Some researchers have lately used the most prevalent decentralized setting available today, called Federated Learning (FL) to develop distributed -GAN strategies as the possible solutions, although their implementations mostly failed to address the above issues mainly because of: the training instability within the distributed component s, which eventually leads to the poor synchronization among the generator s and discriminator s scattered over several machines.
In this work, we developed a computationally inexpensive Wasserstein conditional Distributed Relativistic Discriminator-GAN or DRD-GAN to alleviate the above issues. DRD-GAN stabilizes its train ing (with non-convex losses) by keeping a global generator in the central server and relativistic discriminator s in the local client s (one discriminator per client ), and uses Wasserstein-1 for computing local and global losses. It eventually avoids mode collapse s, vanishing/exploding gradient s (both in the presence of iid and non-iid samples) and helps DRD-GAN to produce high-quality fake images. Apart from that, the sheer unavailability of a capable conditional distributed -GAN model has become another motivation behind the current work. Essentially, we revisited the FL paradigms, locating one discriminator per client , and a generator in the central server that aggregates the updates coming from multiple discriminator s. Relativistic discriminator s in the client s are train ed on both iid and non-iid private data. We presented a detailed mathematical formulation of DRD-GAN and empirically evaluated our claims using CIFAR-10, MNIST, EuroSAT, and CelebA datasets.
... To initiate the registration contract, the task initiator obtains the PKIs root key (RK ′ pub ) from the CA and anchors it into the contract. The root key is required to validate that only the For the aggregation contract, the task initiator implements the aggregation strategy, e.g., averaged-based [35], and creates the initial learning parameters that will be input to the first learning cycle and then updated in each iteration. ...
Verifiable decentralized federated learning (FL) systems combining blockchains and zero-knowledge proofs (ZKP) make the computational integrity of local learning and global aggregation verifiable across workers. However, they are not end-to-end: data can still be corrupted prior to the learning. In this paper, we propose a verifiable decentralized FL system for end-to-end integrity and authenticity of data and computation extending verifiability to the data source. Addressing an inherent conflict of confidentiality and transparency, we introduce a two-step proving and verification (2PV) method that we apply to central system procedures: a registration workflow that enables non-disclosing verification of device certificates and a learning workflow that extends existing blockchain and ZKP-based FL systems through non-disclosing data authenticity proofs. Our evaluation on a prototypical implementation demonstrates the technical feasibility with only marginal overheads to state-of-the-art solutions.
... The weights of the model created are then transmitted to the base station. The base station does the averaging of the weights through the federated average (FedAvg) method [32]. The architecture of the CLAT model is shown in Fig. 3. ...
Machine failures during the manufacturing process can have severe consequences, causing extensive downtime and financial losses. Hence, predictive maintenance (PdM) plays a crucial role within the Industrial Internet of Things (IIoT) by estimating the remaining useful life (RUL) of machines so that proactive maintenance measures can be taken to mitigate potential failures and minimize disruptions. RUL estimation is enabled by gathering and processing the data sensed by sensors mounted on and around the machines at the central server (base station or cloud) after analyzing the failure patterns. However, this approach imposes a significant load on network bandwidth and leads to poor response time for the monitoring system because a large volume of sensed data has to be transmitted to the central server for processing. Moreover, due to the singularity of the computing resource, many problems, such as inefficient resource utilization, frequent offloading, single-point failure, etc., have become major challenges. To address these issues, this article proposes an edge computing-enabled predictive maintenance framework called “Collaborative Predictive Maintenance Framework" (CollabRULe), which first identifies the optimal locations for edge server placement in the deployment region by considering the QoS parameters, such as energy, delay and connectivity. Then, it uses federated learning for predictive maintenance by estimating the RUL of the machines. Simulation results show that the proposed mechanism effectively minimizes the overall network energy consumption and end-to-end delay by ≈60%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 60\%$$\end{document} and ≈35%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 35\%$$\end{document}, respectively, as compared to state-of-the-art approaches. Furthermore, it shows significant improvement in the accuracy of RUL prediction as compared to its counterparts.
... An optimization problem of federated learning in algorithm FedAvg you can see in this formula [29]: ...
In this paper, the authors propose a new machine learning paradigm, federated machine learning. This method produces accurate predictions without revealing private data. It requires less network traffic, reduces communication costs and enables private learning from device to device. Federated machine learning helps to build models and further the models are moved to the device. Applications are particularly prevalent in healthcare, finance, retail, etc., as regulations make it difficult to share sensitive information. Note that this method creates an opportunity to build models with huge amounts of data by combining multiple databases and devices. There are many algorithms available in this area of machine learning and new ones are constantly being created. Our paper presents a comparative analysis of algorithms: FedAdam, FedYogi and FedSparse. But we need to keep in mind that FedAvg is at the core of many federated machine learning algorithms. Data testing was conducted using the Flower and Kaggle platforms with the above algorithms. Federated machine learning technology is usable in smartphones and other devices where it can create accurate predictions without revealing raw personal data. In organizations, it can reduce network load and enable private learning between devices. Federated machine learning can help develop models for the Internet of Things that adapt to changes in the system while protecting user privacy. And it is also used to develop an AI model to meet the risk requirements of leaking client's personal data. The main aspects to consider are privacy and security of the data, the choice of the client to whom the algorithm itself will be directed to process the data, communication costs as well as its quality, and the platform for model aggregation.
Artificial Intelligence (AI) and Machine Learning (ML) methods have been applied significantly in modern healthcare systems in the last few years. AI and its subfields, such as ML, Deep Learning (DL), and Reinforcement Learning (RL), are driving a paradigm shift in modern healthcare systems, including disease detection, diagnosis, treatment, and outcome prediction, supported by good quality healthcare datasets. Research domains such as telemedicine, precision medicine, and healthcare monitoring have become pioneers in deploying AI methods for advancing medical sectors. Additionally, the emerging subfield of AI, Federated Learning (FL) removes the barrier of data sharing and enhances privacy which is gaining increasing attention as a mainstream technology in healthcare research and utilizing patients’ data efficiently. This paper provides a comprehensive study of the use of AI, ML, and FL methods in smart healthcare systems, their contributions to this paradigm shift, their current status, and their recent challenges. In addition, this study outlines a road map for future research in this domain.
The machine learning community adopted the use of null hypothesis significance testing (NHST) in order to ensure the statistical validity of results. Many scientific fields however realized the shortcomings of frequentist reasoning and in the most radical cases even banned its use in publications. We should do the same: just as we have embraced the Bayesian paradigm in the development of new machine learning methods, so we should also use it in the analysis of our own results. We argue for abandonment of NHST by exposing its fallacies and, more importantly, offer better - more sound and useful - alternatives for it.
We present a Bayesian approach for making statistical inference about the accuracy (or any other score) of two competing algorithms which have been assessed via cross-validation on multiple data sets. The approach is constituted by two pieces. The first is a novel correlated Bayesian \(t\) test for the analysis of the cross-validation results on a single data set which accounts for the correlation due to the overlapping training sets. The second piece merges the posterior probabilities computed by the Bayesian correlated \(t\) test on the different data sets to make inference on multiple data sets. It does so by adopting a Poisson-binomial model. The inferences on multiple data sets account for the different uncertainty of the cross-validation results on the different data sets. It is the first test able to achieve this goal. It is generally more powerful than the signed-rank test if ten runs of cross-validation are performed, as it is anyway generally recommended.
Long-Term Evolution (LTE) is the new standard recently specified by the 3GPP on the way towards fourth-generation mobile. This paper presents the main technical features of this standard as well as its performance in terms of peak bit rate and average cell throughput, among others. LTE entails a big technological improvement as compared with the previous 3G standard. However, this paper also demonstrates that LTE performance does not fulfil the technical requirements established by ITU-R to classify one radio access technology as a member of the IMT-Advanced family of standards. Thus, this paper describes the procedure followed by the 3GPP to address these challenging requirements. Through the design and optimization of new radio access techniques and a further evolution of the system, the 3GPP is laying down the foundations of the future LTE-Advanced standard, the 3GPP candidate for 4G. This paper offers a brief insight into these technological trends.
Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. Our analysis casts some light on why recent "High Throughput" methods achieve surprising success--they appear to search through a large number of hyper-parameters because most hyper-parameters do not matter much. We anticipate that growing interest in large hierarchical models will place an increasing burden on techniques for hyper-parameter optimization; this work shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms.
Keras' learning rate decay implementation
- François Chollet
- François
François Chollet et al. 2018. Keras' learning rate decay implementation.
Retrieved August 17, 2018 from https://github.com/keras-team/keras/blob/
29a22a8d59b5e2c4282f1e7f664d82595049eb9d/keras/optimizers.py#L178
Communication-efficient learning of deep networks from decentralized data
- Eider H Brendan Mcmahan
- Daniel Moore
- Seth Ramage
- Hampson
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. 2016.
Communication-efficient learning of deep networks from decentralized data.
arXiv:1602.05629
CO-OP: Cooperative Machine Learning from Mobile Devices
- Yushi Wang
- Wang Yushi
Yushi Wang. 2017. CO-OP: Cooperative Machine Learning from Mobile Devices.
Master's thesis. Dept. Elect. and Comput. Eng., Univ. Alberta, Edmonton, Canada.