No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Intent Recognition in Smart Living Through Deep Recurrent Neural Networks
Abstract
Electroencephalography (EEG) signal based intent recognition has recently attracted much attention in both academia and industries, due to helping the elderly or motor-disabled people controlling smart devices to communicate with outer world. However, the utilization of EEG signals is challenged by low accuracy, arduous and time-consuming feature extraction. This paper proposes a 7-layer deep learning model to classify raw EEG signals with the aim of recognizing subjects’ intents, to avoid the time consumed in pre-processing and feature extraction. The hyper-parameters are selected by an Orthogonal Array experiment method for efficiency. Our model is applied to an open EEG dataset provided by PhysioNet and achieves the accuracy of 0.9553 on the intent recognition. The applicability of our proposed model is further demonstrated by two use cases of smart living (assisted living with robotics and home automation).
... However, extracting data from long temporal sequences presents a formidable challenge in these contexts. Other research endeavors, as noted in [4] and [11][12][13][14][15][16][17], emphasize recurrent neural networks (RNNs). For instance, the authors of DRNN [11] achieved a remarkable 95.53% accuracy when classifying activities in the EEGmmidb dataset BCI2000. ...
... Other research endeavors, as noted in [4] and [11][12][13][14][15][16][17], emphasize recurrent neural networks (RNNs). For instance, the authors of DRNN [11] achieved a remarkable 95.53% accuracy when classifying activities in the EEGmmidb dataset BCI2000. This feat was accomplished by employing seven RNNs. ...
... Meanwhile, the research presented in [18] merges convolutional and RNNs, while [19] introduces the use of generative adversarial network convolutional neural network (GAN-CNN). It's worth noting that studies like [11,14], and [17] present evaluation results based solely on one dataset, which raises concerns regarding their generalizability. The research paper [17] exclusively concentrates on individuals with amputations, limiting its generalizability to the broader population of healthy individuals. ...
One of the critical challenges in brain-computer interfaces is the classification of brain activities through the analysis of EEG signals. This paper seeks to improve the efficacy of deep learning-based rehabilitation systems, aiming to deliver superior services for individuals with physical disabilities. The research introduces the time distributed long short-term memory (TD-LSTM) framework, which incorporates an LSTM and a time distributed approach to classify brain activities. Learning in TD-LSTM is achieved by uncovering time-dependent semantic dependencies within EEG signals over time. By extracting all discriminative and relevant spatiotemporal dependencies via TD-LSTM, valuable information on different time steps in each sequence has been obtained. Time distributed approach shortens the input time series, making learning from long time series sequences easier, and the learning process of complex temporal and spatial dependencies in time series multi-channel EEG signals becomes more efficient. The main contributions in this paper can be outlined as follows: (1) implementation of brain activity binary classification of motor imagery/execution tasks using time distributed approach via RNN network for the first time, (2) evaluation of the performance and generalizability of the proposed method on four benchmark datasets, (3) dual-purpose classification which providing an efficient ways for classifying both types of motor imagery/execution brain activity. The experimental results show that the proposed method performs well compared to several baseline research works.
... Best observed with eyes closed and in a state of physical calm and mental inaction. Discontinued by any mental or visual activity 4. Beta band: lies within the frequency range of (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30) Hz, with an amplitude range of (5-30) lV. It is dominant when the eyes are open and blocked by motor activity and the intention of motor movement. ...
... A significant number of research on MI signal classification using LSTM have been conducted in recent years. Researchers in [29] used PhysioNet dataset as large as (109 participants, 26.4 million samples) containing five MI motions. The authors obtained an accuracy of 95.53% for classification using seven RNN layers and two LSTM layers. ...
... Because the brain waves of interest are in the 0-30 Hz range, the data gathered are analyzed through the Fast Fourier Transform (FFT) between the overall 0-30 Hz frequency range. The analyzed data are subdivided into four frequency domains, delta (0-4 Hz), theta (4-8 Hz), alpha (8)(9)(10)(11)(12), and beta (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30). For Each domain, two aggregation functions are applied using weighted, and arithmetic mean values; this step enriched the information content and led to duplicating the data columns by two. ...
Recently, the discipline of Brain-Computer-Interface (BCI) has attracted attention to exploiting Electroencephalograph (EEG) mental activities such as Motor Imagery (MI). Neurons in the human brain are activated during these MI tasks and generate an electrical potential of small magnitude reached to the scalp as a signal. Classification of MI data is a primary problem in BCI systems. Classification accuracy of these biomedical signals emerges as a significant task in the scientific community. This work proposes two main ideas: a new preprocessing technique based on four EEG frequency bands and a new stacking method for three deep-learning architectures used to decode three classes of MI signals. The preprocessing stage was introduced using Fast Fourier Transform to perform frequency analysis and data aggregation functions to enhance the data view. Performance was evaluated using well-defined metrics: accuracy, precision, recall, and f1-score for multiple batch sizes, optimizers, and epochs. Experimental results were evaluated using a publicly available dataset (BCI Competition IV dataset 2a) and local data collected from four subjects using the EMOTIV EPOC headset. The highest f1-scores achieved with the R-CNN model were 94% and 84% using the aforementioned datasets. Our proposed models also outperform many related models studied in the literature.
... An equally interesting approach is presented in [6], where a systematic framework for recognizing realistic actions from videos "in the wild" is used. The topic of neural networks is also of interest in a variety of industries not only in terms of its application to video classification but also its ability to address problems related to sound, for which time continuity is also a concern [7]. 3D convolutional neural network (3DCNN) is a powerful and effective model utilizing spatial-temporal features, which is why they also use it in the article [8] for gesture recognition. ...
... The choice of hyperparameters affects the overall results. For example, in [7] authors show that the accuracy varies from 32.2% to 92.6% depending on the selected hyperparameter values. Hyperparameters such as filter size, size of the MaxPooling filter, or the number of filters, are set default values in the design. ...
... The F1 score is defined as a measure that provides a balance between recall and precision. The accuracy result obtained using the proposed 3DCNN network is compared with accuracy results previously reported for other networks (different architecture for motion, static and hybrid features) [7]. We also compared the results of the proposed architecture with previous research [22]. ...
Interest in utilizing neural networks in a variety of scientific and academic studies and in industrial applications is increasing. In addition to the growing interest in neural networks, there is also a rising interest in video classification. Object detection from an image is used as a tool for various applications and is the basis for video classification. Identifying objects in videos is more difficult than for single images, as the information in videos has a time continuity constraint. Common neural networks such as ConvLSTM (Convolutional Long Short-Term Memory) and 3DCNN (3D Convolutional Neural Network), as well as many others, have been used to detect objects from video. Here, we propose a 3DCNN for the detection of human activity from video data. The experimental results show that the optimized proposed 3DCNN provides better results than neural network architectures for motion, static and hybrid features. The proposed 3DCNN obtains the highest recognition precision of the methods considered, 87.4%. In contrast, the neural network architectures for motion, static and hybrid features achieve precisions of 65.4%, 63.1% and 71.2%, respectively. We also compare results with previous research. Previous 3DCNN architecture on database UCF Youtube Action worked worse than the architecture we proposed in this article, where the achieved result was 29%. The experimental results on the UCF YouTube Action dataset demonstrate the effectiveness of the proposed 3DCNN for recognition of human activity. For a more complex comparison of the proposed neural network, the modified UCF101 dataset, full UCF50 dataset and full UCF101 dataset were compared. An overall precision of 82.7% using modified UCF101 dataset was obtained. On the other hand, the precision using full UCF50 dataset and full UCF101 dataset was 80.6% and 78.5%, respectively.
... RNNs have the capability of propagating information from previous timesteps to the current process while LSTM networks are special architectures of RNNs, which proved to be highly accurate. Although, RNNs that are composed of LSTM modules are commonly used for pattern recognition in several application areas, such as speech and handwriting recognition [2][3][4][5], a number of works use RNNs in non-vocal applications, using electroencephalogram (EEG) [6][7][8][9][10]. ...
... In [9], the authors design a CRNN model, which is a combination of CNN and RNN/LSTM networks, for Parkinson's disease classification using EEG signals. The authors in [10] present a software application that assists people with mobility difficulties, such as elderly or people with motor neuron diseases. This system utilizes a recurrent neural network (RNN) and electroencephalography (EEG) signals to recognize and classify the intent of the user and control the moves of a robot as well as the operation of household appliances. ...
... This system utilizes a recurrent neural network (RNN) and electroencephalography (EEG) signals to recognize and classify the intent of the user and control the moves of a robot as well as the operation of household appliances. In this paper, the model proposed in [10] is exploited because it displays a noticeably high accuracy, utilizes raw EEG data, contrasting most mentioned designs that need preprocessing, and classifies the signals to five different classes, dissimilar with most that trigger binary values, resulting in a more functional and intriguing application. ...
In recent years, systems that monitor and control home environments, based on non-vocal and non-manual interfaces, have been introduced to improve the quality of life of people with mobility difficulties. In this work, we present the reconfigurable implementation and optimization of such a novel system that utilizes a recurrent neural network (RNN). As demonstrated in the real-world results, FPGAs have proved to be very efficient when implementing RNNs. In particular, our reconfigurable implementation is more than 150× faster than a high-end Intel Xeon CPU executing the reference inference tasks. Moreover, the proposed system achieves more than 300× the improvements, in terms of energy efficiency, when compared with the server CPU, while, in terms of the reported achieved GFLOPS/W, it outperforms even a server-tailored GPU. An additional important contribution of the work discussed in this study is that the implementation and optimization process demonstrated can also act as a reference to anyone implementing the inference tasks of RNNs in reconfigurable hardware; this is further facilitated by the fact that our C++ code, which is tailored for a high-level-synthesis (HLS) tool, is distributed in open-source, and can easily be incorporated to existing HLS libraries.
... Some are based on manually extracted features [62,63]. For instance, Lee et al. [64] and Zhang et al. [65] employed CNN and 2-D CNN, respectively, for classification; Zhang et al. [65] learned affective information from EEG signals to built a modified LSTM control smart home appliances. Others also used CNN for feature extraction [66]. ...
... Some are based on manually extracted features [62,63]. For instance, Lee et al. [64] and Zhang et al. [65] employed CNN and 2-D CNN, respectively, for classification; Zhang et al. [65] learned affective information from EEG signals to built a modified LSTM control smart home appliances. Others also used CNN for feature extraction [66]. ...
... MLP has also been applied for MI EEG recognition [68], which showed higher sensitivity to EEG phase features at earlier stages and higher sensitivity to EEG amplitude features at later stages. [51], [48], [25,50], [70,52] [ 49,54] [57] [52] [56] [55], [58] MI EEG [71], [68] [6], [61,65] [64], [72], [60], [63], [73], [ [112], [113], [114], [115], [116], [117], [118,119] [120] ...
Brain signals refer to the biometric information collected from the human brain. The research on brain signals aims to discover the underlying neurological or physical status of the individuals by signal decoding. The emerging deep learning techniques have improved the study of brain signals significantly in recent years. In this work, we first present a taxonomy of non-invasive brain signals and the basics of deep learning algorithms. Then, we provide a comprehensive survey of the frontiers of applying deep learning for non-invasive brain signals analysis, by summarizing a large number of recent publications. Moreover, upon the deep learning-powered brain signal studies, we report the potential real-world applications which benefit not only disabled people but also normal individuals. Finally, we discuss the opening challenges and future directions.
... Best performances with QDA. Zhang, Yao, Huang, et al. (2017) Consider all the motor imagery tasks plus the eyes closed condition on 10 subjects. Apply long short-term memory Recurrent Neural Network (RNN) and perform hyper parameter selection through orthogonal array. ...
... Apply the extreme gradient boosting classifier. (Dose et al., 2018;Lu et al., 2019;Wang et al., 2020;Zhang et al., 2019;Zhang, Yao, Huang, Sheng, & Wang, 2017), and exploit an Autoencoder for feature extraction or adaptation (Zhang et al., 2018). No other information are given regarding feature computation and selection. ...
... Three publications in our result set are of this type (Zhang et al., 2019b;Zhang et al., 2020a;Nguyen et al., 2018). Other applications of EEG control include driving assist (Mourad et al., 2020;Lu et al., 2020), embedded movement recognition , smart home control (Zhang et al., 2017), image reconstruction (Hernandez-Carmona et al., 2019) and virtual reality control (Bevilacqua et al., 2014;Karácsony et al., 2019). One publication uses EEG for wheelchair control (Zgallai et al., 2019) and another combines EEG with functional near-infrared spectroscopy to play a computer game (Makhrov et al., 2018). ...
... The Other category includes applications such as speller systems that allow paralyzed persons to communicate (Nguyen et al., 2018;Zhang et al., 2020a;Zhang et al., 2019b), virtual reality control systems (Karácsony et al., 2019;Bevilacqua et al., 2014), driving assist systems (Lu et al., 2020;Mourad et al., 2020), and smart home control (Zhang et al., 2017). Except for spellers, which are all EEG-based, these applications are the type that target biosignal control for everyday use by able-bodied users. ...
Biosignal control is an interaction modality that allows users to interact with electronic devices by decoding the biological signals emanating from the movements or thoughts of the user. This manner of interaction with devices can enhance the sense of agency for users and enable persons suffering from a paralyzing condition to interact with everyday devices that would otherwise be challenging for them to use. It can also improve control of prosthetic devices and exoskeletons by making the interaction feel more natural and intuitive. However, with the current state of the art, several issues still need to be addressed to reliably decode user intent from biosignals and provide an improved user experience over other interaction modalities. One solution is to leverage advances in Deep Learning (DL) methods to provide more reliable decoding at the expense of added computational complexity. This scoping review introduces the basic concepts of DL and assists readers in deploying DL methods to a real-time control system that should operate under real-world conditions. The scope of this review covers any electronic device, but with an emphasis on robotic devices, as this is the most active area of research in biosignal control. We review the literature pertaining to the implementation and evaluation of control systems that incorporate DL to identify the main gaps and issues in the field, and formulate suggestions on how to mitigate them. Additionally, we formulate guidelines on the best approach to designing, implementing and evaluating research prototypes that use DL in their biosignal control systems.
... After that, the XGBoost classifier has been used for classification. The method in [50] uses CNN and LSTM in serial fashion, while the technique in [52] uses a reinforced learning approach combined with CNN. Table 9 shows the performance of the proposed approach and the abovementioned three methods on the EEGMMIDB dataset. ...
... Table 9 shows the performance of the proposed model against the three comparison models on raw EEG of the EEGMMIDB dataset. The result shows the proposed model achieves an accuracy of 80.9% on the raw dataset which is 6% higher than the method proposed in [50]. The performance of the model against the other two approaches, i.e., [51] and [52], is approximately 6% lower on raw EEG, the reason being the lower number of parameters in the proposed model. ...
Electroencephalography (EEG)-based brain computer interface (BCI) enables people to interact directly with computing devices through their brain signals. A BCI typically interprets EEG signals to reflect the user's intent or other mental activity. Motor imagery (MI) is a commonly used technique in BCIs where a user is asked to imagine moving certain part of the body such as a hand or a foot. By correctly interpreting the signal, one can perform a multitude of tasks such as controlling wheel chair, playing computer games, or even typing text. However, the use of motor-imagery-based BCIs outside the laboratory environment is limited due to the lack of their reliability. This work focuses on another kind of mental imagery, namely, the visual imagery (VI). VI is the manipulation of visual information that comes from memory. This work presents a deep con-volutional neural network (DCNN)-based system for the recognition of visual/mental imagination of English alphabets so as to enable typing directly via brain signals. The DCNN learns to extract the spatial features hidden in the EEG signal. As opposed to many deep neural networks that use raw EEG signals for classification, this work transforms the raw signals into band powers using Morlet wavelet transformation. The proposed approach is evaluated on two publicly available benchmark MI-EEG datasets and a visual imagery dataset specifically collected for this work. The obtained results demonstrate that the proposed model performs better than the existing state-of-the-art methods for MI-EEG classification and yields an average accuracy of 99.45% on the two public MI-EEG datasets. The model also achieves an average recognition rate of 95.2% for the 26 English-language alphabets.
... Salazar-Varas et al. reported that by selecting the appropriate bandpass filter, the classification performance of an intended movement from EEG signals could be improved [41] . However, when features of EEG signals are considered in just one frequency [42][43][44] , relevant features from other frequencies that would have positively influenced the performance of the learning model might be lost. It is therefore necessary to evaluate the information within an EEG signal at different frequency ranges that is mostly associated with the specific neuronal activity. ...
... Further comparison was conducted with other methods involving the use of deep learning methods. For example, the application of LSTM-RNN for assisted living with robotics and home automation achieved the accuracy of 95.53% when tested on motor tasks [44] . The application of cascaded convolutional recurrent neural network (CasRNN) yielded the accuracy of 93% [67] . ...
Background and Objective
Recognition of motor intention based on electroencephalogram (EEG) signals has attracted considerable research interest in the field of pattern recognition due to its notable application of non-muscular communication and control for those with severe motor disabilities. In analysis of EEG data, achieving a higher classification performance is dependent on the appropriate representation of EEG features which is mostly characterized by one unique frequency before applying a learning model. Neglecting other frequencies of EEG signals could deteriorate the recognition performance of the model because each frequency has its unique advantages. Motivated by this idea, we propose to obtain distinguishable features with different frequencies by introducing an integrated deep learning model to accurately classify multiple classes of upper limb movement intentions.
Methods
The proposed model is a combination of long short-term memory (LSTM) and stacked autoencoder (SAE). To validate the method, four high-level amputees were recruited to perform five motor intention tasks. The acquired EEG signals were first preprocessed before exploring the consequence of input representation on the performance of LSTM-SAE by feeding four frequency bands related to the tasks into the model. The learning model was further improved by t-distributed stochastic neighbor embedding (t-SNE) to eliminate feature redundancy, and to enhance the motor intention recognition.
Results
The experimental results of the classification performance showed that the proposed model achieves an average performance of 99.01% for accuracy, 99.10% for precision, 99.09% for recall, 99.09% for f1_score, 99.77% for specificity, and 99.0% for Cohen's kappa, across multi-subject and multi-class scenarios. Further evaluation with 2-dimensional t-SNE revealed that the signal decomposition has a distinct multi-class separability in the feature space.
Conclusion
This study demonstrated the predominance of the proposed model in its ability to accurately classify upper limb movements from multiple classes of EEG signals, and its potential application in the development of a more intuitive and naturalistic prosthetic control.
... Hence, our low-intrusive and low-cost solution could be used in an interactive system with a reduced response time of 2 s. Sensors 2020, 20, 6730 2 of 18 systems in several healthcare areas, such as healthy ageing, to help the elderly living independently, increasing their autonomy and improving their quality of life [5,6]; and impaired persons [2,7,8]. ...
... Additionally, they gather big quantities of data in an objective and precise way. Wearables have been used in evaluation and intervention as healthy ageing, to help the elderly living independently, increasing their autonomy and improving their quality of life [5,6]; and impaired persons [2,7,8]. ...
Electroencephalography (EEG) signals to detect motor imagery have been used to help patients with low mobility. However, the regular brain computer interfaces (BCI) capturing the EEG signals usually require intrusive devices and cables linked to machines. Recently, some commercial low-intrusive BCI headbands have appeared, but with less electrodes than the regular BCIs. Some works have proved the ability of the headbands to detect basic motor imagery. However, all of these works have focused on the accuracy of the detection, using session sizes larger than 10 s, in order to improve the accuracy. These session sizes prevent actuators using the headbands to interact with the user within an adequate response time. In this work, we explore the reduction of time-response in a low-intrusive device with only 4 electrodes using deep learning to detect right/left hand motion imagery. The obtained model is able to lower the detection time while maintaining an acceptable accuracy in the detection. Our findings report an accuracy above 83.8% for response time of 2 s overcoming the related works with both low-and high-intrusive devices. Hence, our low-intrusive and low-cost solution could be used in an interactive system with a reduced response time of 2 s.
... Moreover, the literature Huang et al. (Huang et al. 2023) is only classified into four categories, while our proposed model is for intention recognition of five types of EEG activities. Particularly, the decoding effect of the model proposed is improved about 23.5% compared with the model proposed by Almoaril et al. (Alomari et al. 2014); The recognition rate is improved by 11.16% compared with the model proposed by Sita et al. (Sita and Nair 2013); The recognition result is improved by 3.9% compared with the model proposed by Hou et al. (Hou et al. 2020b) and the recognition rate is improved by 2.87% compared with the model proposed by Zhang et al. (Zhang et al. 2017). Through comparing the model proposed by Chen et al. (Chen et al. 2018), the result of the model proposed in this study is improved by 0.54%; The model proposed in this study is improved by 2.08% compared with the algorithm model proposed by Zhang et al. (Zhang et al. 2019). ...
Currently, electroencephalogram (EEG)-based motor imagery (MI) signals have been received extensive attention, which can assist disabled subjects to control wheelchair, automatic driving and other activities. However, EEG signals are easily affected by some factors, such as muscle movements, wireless devices, power line, etc., resulting in the low signal-to-noise ratios and the worse recognition results on EEG decoding. Therefore, it is crucial to develop a stable model for decoding MI-EEG signals. To address this issue and further improve the decoding performance for MI tasks, a hybrid structure combining convolutional neural networks and bidirectional long short-term memory (BLSTM) model, namely CBLSTM, is developed in this study to handle the various EEG-based MI tasks. Besides, the attention mechanism (AM) model is further adopted to adaptively assign the weight of EEG vital features and enhance the expression which beneficial to classification for MI tasks. First of all, the spatial features and the time series features are extracted by CBLSTM from preprocessed MI-EEG data, respectively. Meanwhile, more effective features information can be mined by the AM model, and the softmax function is utilized to recognize intention categories. Ultimately, the numerical results illustrate that the model presented achieves an average accuracy of 98.40% on the public physioNet dataset and faster training process for decoding MI tasks, which is superior to some other advanced models. Ablation experiment performed also verifies the effectiveness and feasibility of the developed model. Moreover, the established network model provides a good basis for the application of brain-computer interface in rehabilitation medicine.
... In 2017, a LSTM model was proposed [38] to obtain intent recognition considering all the experimental MI conditions of 10 subjects. A hyper-parameter selection through orthogonal array application was also performed obtaining a final average accuracy of 95.53%. ...
The Motor Imagery (MI) electroencephalography (EEG) based Brain Computer Interfaces (BCIs) allow the direct communication between humans and machines by exploiting the neural pathways connected to motor imagination. Therefore, these systems open the possibility of developing applications that could span from the medical field to the entertainment industry. In this context, Artificial Intelligence (AI) approaches become of fundamental importance especially when wanting to provide a correct and coherent feedback to BCI users. Moreover, publicly available datasets in the field of MI EEG-based BCIs have been widely exploited to test new techniques from the AI domain. In this work, AI approaches applied to datasets collected in different years and with different devices but with coherent experimental paradigms are investigated with the aim of providing a concise yet sufficiently comprehensive survey on the evolution and influence of AI techniques on MI EEG-based BCI data.
... • Neurorehabilitation: recently, promising papers have been published studying the benefit of applying BCI in neurorehabilitation, with the aim to increase the effects of physiotherapy in patient with sever motor impairment [2,5,35,40,43]. • Health: like assisted living [14], appliance control [22,62] and biomedical engineering field [1,10,17,34,44,56,58]. ...
Due to its non-invasiveness and easiness to implement, EEG signals decoding are in base of most based brain computer interfaces (BCI) studies. Given the non-stationary nature of these signals, a preprocessing phase is needed. An interesting idea to perform the preprocessing is the use of spatial covariance matrices. In the last years, spatial covariance matrices based preprocessing was extensively used in electroencephalography (EEG) signal processing and spatial filtering for Motor imagery (MI) BCI. Spatial covariance matrices lie in the Riemannian manifold of Symmetric Positive-Definite (SPD) matrices, therefore, the use of Riemannian geometry is attracting a lot of attention and showing to be simple, robust, and providing good performance. This paper explores the idea of enhancing the information provided to the classifier by the combination of different covariance matrices projections from their native Riemannian space to multiple class-depending tangent spaces. We demonstrate that this new approach provides a significant improvement in model accuracy.
... Another approach may be through intent recognition [10], which can be used for smart applications such as turning off the led or controlling a robot. LSTM [?] tuned using implementation of Orthogonal Array experiment, they were able to achieve considerably higher accuracy. ...
The neurons in the brain produces electric signals and a collective firing of these electric signals gives rise to brainwaves. These brainwave signals are captured using EEG (Electroencephalogram) devices as micro voltages. These sequence of signals captured by EEG sensors have embedded features in them that can be used for classification. The signals can be used as an alternative input for people suffering from severe locomotive disorder.Classification of different colors can be mapped for many functions like directional movement. In this paper, raw EEG signals from NeuroSky Mindwave headset (a single electrode EEG sensor) have been classified with an attention based Deep Learning Network. Attention based LSTM Networks have been implemented for classification of two different colors and four different colors. An accuracy of 93.5\% was obtained for classification of two colors and an accuracy of 65.75\% was obtained for classifcation of four signals using the mentioned attention based LSTM network.
... In 2017, a LSTM model was proposed [43] to obtain intent recognition considering all the experimental MI conditions of 10 subjects. A hyper-parameter selection through orthogonal array application was also performed obtaining a final average accuracy of 95.53%. ...
The Motor Imagery (MI) electroencephalography (EEG) based Brain Computer Interfaces (BCIs) allow the direct communication between humans and machines by exploiting the neural pathways connected to motor imagination. Therefore, these systems open the possibility of developing applications that could span from the medical field to the entertainment industry. In this context, Artificial Intelligence (AI) approaches become of fundamental importance especially when wanting to provide a correct and coherent feedback to BCI users. Moreover, publicly available datasets in the field of MI EEG-based BCIs have been widely exploited to test new techniques from the AI domain. In this work, AI approaches applied to datasets collected in different years and with different devices but with coherent experimental paradigms are investigated with the aim of providing a concise yet sufficiently comprehensive survey on the evolution and influence of AI techniques on MI EEG-based BCI data.
... The proposed unified deep learning framework aims to interpret the subjects' intent and decode it into the corresponding commands which are discernible for the IoT devices. Based on our previous study [5], [18], for each single brain signal sample, the self-similarity is always higher than the cross-similarity, which means that the intra-intent cohesion of the samples is stronger than the inter-intent cohesion. In this paper, we propose a weighted average spatial Long Short-Term Memory (WAS-LSTM) to exploit the latent correlation between signal dimensions. ...
Transformation of technologies to operate in Internet of Things (IoT) will increase their connectivity for sharing and collecting data with minimal human intervention, as well as to use advantages of cloud computing. The paper presents a novel approach for integrating EEG brain computer interface (BCI) with a humanoid robot for communications in the IoT. A transformation of both devices into IoT things allow seamless communication for data exchange with other IoT devices or services and more efficient computations in the web or cloud. Node-RED is the chosen programming tool providing a gateway to IoT, by which the EEG-based Emotiv EPOC+BCI device and the humanoid robot Pepper communicate. There is a library of input nodes in Node-RED for EPOC+ device, by which it interfaces services and devices in Internet, however such nodes do not exists for Pepper. Two approaches for transformation of the robot into IoT device have been designed and tested. In the first, MQTT protocol has been deployed on the robot side that can use local or remote MQTT broker. In the second approach, there is no need to install specific publish-subscribe client software for interactions with the broker. Instead, continuous python process in Node-RED accesses the Pepper internal memory for storing and retrieving key-value pairs. The devices share and collect data based on publish-subscribe model embedded in Node-RED. Original flows were designed and developed for BCI control of the robot in the IoT. The control is based on EEG data featuring, classifying and translating into machine commands that interface the robot APIs from Node-RED.
... Wrong parameter tuning can cause a model inaccuracy. In fact, the parameter adjustment fluctuates the accuracy of the classification from 32.2% to 92.6% [20]. Instead of using Trial and Error, many techniques were used such as Grid search and PSO. ...
Water is a vital resource. It supports a multitude of industries, civilizations, and agriculture. However, climatic conditions impact water availability, particularly in desert areas where the temperature is high, and rain is scarce. Therefore, it is crucial to forecast water demand to provide it to sectors either on regular or emergency days. The study aims to develop an accurate model to forecast daily water demand under the impact of climatic conditions. This forecasting is known as a multivariate time series because it uses both the historical data of water demand and climatic conditions to forecast the future. Focusing on the collected data of Jeddah city, Saudi Arabia in the period between 2004 and 2018, we develop a hybrid approach that uses Artificial Neural Networks (ANN) for forecasting and Particle Swarm Optimization algorithm (PSO) for tuning ANNs’ hyperparameters. Based on the Root Mean Square Error (RMSE) metric, results show that the (PSO-ANN) is an accurate model for multivariate time series forecasting. Also, the first day is the most difficult day for prediction (highest error rate), while the second day is the easiest to predict (lowest error rate). Finally, correlation analysis shows that the dew point is the most climatic factor affecting water demand.
... Accuracy variety depends on setting the hyperparameters. According to paper [1], accuracy can be between 32.2% and 92.6%. Therefore, the correct tuning of hyper-parameters is often a problem. ...
... However, BCIs for augmenting human functions raise important ethical issues and debates [163] that have led to them receiving less attention than applications that improve quality of life for people with disabilities, which require little justification. MI BCIs have also been used to control unmanned ground or aerial vehicles [121], [164], [165], as well as the environment in smart homes [166]. The researchers in [164] conducted experiments with healthy users to control a robotic drone in three-dimensional space. ...
The brain-computer interface (BCI) is an emerging technology that has the potential to revolutionize the world, with numerous applications ranging from healthcare to human augmentation. Electroencephalogram (EEG) motor imagery (MI) is among the most common BCI paradigms that has been used extensively in smart healthcare applications such as post-stroke rehabilitation and mobile assistive robots. In recent years, the contribution of deep learning (DL) has had a phenomenal impact on MI-EEG-based BCI. In this work, we systematically review the DL-based research for MI-EEG classification from the past ten years. This article first explains the procedure for selecting the studies and then gives an overview of BCI, EEG, and MI systems. The DL-based techniques applied in MI classification are then analyzed and discussed from four main perspectives: pre-processing, input formulation, deep learning architecture, and performance evaluation. In the discussion section, three major questions about DL-based MI classification are addressed: (1) Is pre-processing required for DL-based techniques? (2) What input formulations are best for DL-based techniques? (3) What are the current trends in DL-based techniques? Moreover, this work summarizes MI-EEG-based applications, and extensively explores public MI-EEG datasets and gives an overall visualization of the performance attained for each dataset based on the reviewed articles. Finally, current challenges and future directions are discussed.
... Robot controls for wheelchair navigation, implemented in EEG based mobile robot control through adaptive brain-robot interface [13] was obtained through intelligent adaptive user interface(iAUI) which adapted to the situation. As EEG signals are noisy for altering, RNN was used. ...
People with poor or no muscle movement and the visually impaired face difficulties in their
daily lives while operating home appliances. What this research project aims to do is
provide a solution for those individuals, to enable them to get things done efficiently around
their houses or rooms without external assistance. The system being developed enables the
individual to control their devices via an alternative means of brainwaves, making it
accessible to anyone with a healthy brain. It can also be useful to anyone in their day-to-day
life. This report deals with the extraction and analysis of EEG Data on multiple classifiers
and its processing with BCI-based Headsets (Neurosky Mobile).
... They proposed a threelayer model to classify two classes of BCI competition, gaining a mean accuracy of 78.9%. Zhang et al. [8] proposed seven-layer DL model with orthogonal array experiment technique to optimize hyper parameters like the number of layers and nodes. They reached a mean accuracy of 93%. ...
Many techniques have been introduced to improve both brain-computer interface (BCI) steps: feature extraction and classification. One of the emerging trends in this field is the implementation of deep learning algorithms. There is a limited number of studies that investigated the application of deep learning techniques in electroencephalography (EEG) feature extraction and classification. This work is intended to apply deep learning for both stages: feature extraction and classification. This paper proposes a modified convolutional neural network (CNN) feature extractorclassifier algorithm to recognize four different EEG motor imagery (MI). In addition, a four-class linear discriminant analysis (LDR) classifier model was built and compared to the proposed CNN model. The paper showed very good results with 92.8% accuracy for one EEG four-class MI set and 85.7% for another set. The results showed that the proposed CNN model outperforms multi-class linear discriminant analysis with an accuracy increase of 28.6% and 17.9% for both MI sets, respectively. Moreover, it has been shown that majority voting for five repetitions introduced an accuracy advantage of 15% and 17.2% for both EEG sets, compared with single trials. This confirms that increasing the number of trials for the same MI gesture improves the recognition accuracy
... This was briefly discussed in [1], as an example, in turn-based interactions relatively small discount factors (i.e., 0.7 ď γ ď 0.95) are more common, whereas for the frame-based interactions with rather long trajectories, higher discount factors seem to be more suitable (i.e., γ ě 0.99). In deep networks, the selection of different hyper-parameters affects the accuracy of the algorithm [118]. This also applies to DRL, Lathuilière et al. [86] presented several experiments to evaluate the impact of some of the principal parameters of their deep network structure. ...
This article surveys reinforcement learning approaches in social robotics. Reinforcement learning is a framework for decision-making problems in which an agent interacts through trial-and-error with its environment to discover an optimal behavior. Since interaction is a key component in both reinforcement learning and social robotics, it can be a well-suited approach for real-world interactions with physically embodied social robots. The scope of the paper is focused particularly on studies that include social physical robots and real-world human-robot interactions with users. We present a thorough analysis of reinforcement learning approaches in social robotics. In addition to a survey, we categorize existent reinforcement learning approaches based on the used method and the design of the reward mechanisms. Moreover, since communication capability is a prominent feature of social robots, we discuss and group the papers based on the communication medium used for reward formulation. Considering the importance of designing the reward function, we also provide a categorization of the papers based on the nature of the reward. This categorization includes three major themes: interactive reinforcement learning, intrinsically motivated methods, and task performance-driven methods. The benefits and challenges of reinforcement learning in social robotics, evaluation methods of the papers regarding whether or not they use subjective and algorithmic measures, a discussion in the view of real-world reinforcement learning challenges and proposed solutions, the points that remain to be explored, including the approaches that have thus far received less attention is also given in the paper. Thus, this paper aims to become a starting point for researchers interested in using and applying reinforcement learning methods in this particular research field.
... This was briefly discussed in [1], as an example, in turn-based interactions relatively small discount factors (i.e., 0.7 ≤ ≤ 0.95) are more common, whereas for the frame-based interactions with rather long trajectories, higher discount factors seem to be more suitable (i.e., ≥ 0.99). In deep networks, the selection of different hyper-parameters affects the accuracy of the algorithm [118]. ...
This article surveys reinforcement learning (RL) approaches in social robotics. RL is a
framework for decision-making problems in which an agent interacts through trial-and-error
with its environment to discover an optimal behavior. Since interaction is a key component in
both RL and social robotics, it can be a well-suited approach for real-world interactions with
physically embodied social robots. The scope of the paper is focused particularly on studies that
include social physical robots and real-world human-robot interactions with users. In addition to a
survey, we categorize existent RL approaches based on the design of the reward mechanisms.
This categorization includes three major themes: interactive reinforcement learning, intrinsically
motivated methods, and task performance driven methods. Thus, this paper aims to become a
starting point for researchers interested to use and apply reinforcement learning methods in this
particular research field.
... While latest approaches already achieve accurate and robust speech recognition [3,6] to identify user intents [7,8,9] using semantic parsing [10,11], the mapping to advanced data operations (AI algorithms) is limited -and their extensibility is challenging, requiring a high effort. Most often in this context, intents are mapped to SQL query language of databases. ...
Advances in voice-controlled assistants paved the way into the consumer market. For professional or industrial use, the capabilities of such assistants are too limited or too time-consuming to implement due to the higher complexity of data, possible AI-based operations, and requests. In the light of these deficits, this paper presents Insightful Assistant---a pipeline concept based on a novel operation graph representation resulting from the intents detected. Using a predefined set of semantically annotated (executable) functions, each node of the operation graph is assigned to a function for execution. Besides basic operations, such functions can contain artificial intelligence (AI) based operations (e.g., anomaly detection). The result is then visualized to the user according to type and extracted user preferences in an automated way. We further collected a unique crowd-sourced set of 869 requests, each with four different variants expected visualization, for an industrial dataset. The evaluation of our proof-of-concept prototype on this dataset shows its feasibility: it achieves an accuracy of up to 95.0% (74.5%) for simple (complex) request detection with different variants and a top3-accuracy up to 95.4% for data-/user-adaptive visualization.
Brain-computer interface (BCI) is an emerging technology which provides a road to control communication and external devices. Electroencephalogram (EEG)-based motor imagery (MI) tasks recognition has important research significance for stroke, disability and others in BCI fields. However, enhancing the classification performance for decoding MI-related EEG signals presents a significant challenge, primarily due to the variability across different subjects and the presence of irrelevant channels. To address this issue, a novel hybrid structure is developed in this study to classify the MI tasks via deep separable convolution network (DSCNN) and bidirectional long short-term memory (BLSTM). First, the collected time-series EEG signals are initially processed into a matrix grid. Subsequently, data segments formed using a sliding window strategy are inputted into proposed DSCNN model for feature extraction (FE) across various dimensions. And, the spatial-temporal features extracted are then fed into the BLSTM network, which further refines vital time-series features to identify five distinct types of MI-related tasks. Ultimately, the evaluation results of our method demonstrate that the developed model achieves a 98.09% accuracy rate on the EEGMMIDB physiological datasets over a 4-second period for MI tasks by adopting full channels, outperforming other existing studies. Besides, the results of the five evaluation indexes of Recall, Precision, Test-auc, and F1-score also achieve 97.76%, 97.98%, 98.63% and 97.86%, respectively. Moreover, a Gradient-class Activation Mapping (GRAD-CAM) visualization technique is adopted to select the vital EEG channels and reduce the irrelevant information. As a result, we also obtained a satisfying outcome of 94.52% accuracy with 36 channels selected using the Grad-CAM approach. Our study not only provides an optimal trade-off between recognition rate and number of channels with half the number of channels reduced, but also it can also advances practical application research in the field of BCI rehabilitation medicine, effectively.
Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.
Objective: Brain Computer Interface (BCI) systems have been developed to identify and classify brain signals and integrate them into a control system. Even though many different methods and models have been developed for the brain signals classification, the majority of these studies have emerged as specialized models. In addition, preprocessing and signal preprocessing methods which are largely based on human knowledge and experience have been used extensively for classification models. These methods degrade the performance of real-time BCI systems and require great time and effort to design and implement the right method. Approach: In order to eliminate these disadvantages, we developed a generalized and robust CNN model called as No-Filter EEG (NF-EEG) to classify multi class motor imagery brain signals with raw data and without applying any signal preprocessing methods. In an attempt to increase the speed and success of this developed model, input reshaping has been made and various data augmentation methods have been applied to the data. Main results: Compared to many other state-of-the-art models, NF-EEG outperformed leading state-of-the-art models in two most used motor imagery datasets and achieved 93.56% in the two-class BCI-IV-2A dataset and 88.40% in the two-class BCI-IV-2B dataset and 81.05% accuracy in the classification of four-class BCI-IV-2A dataset. Significance: This proposed method has emerged as a generalized model without signal preprocessing and it greatly reduces the time and effort required for preparation for classification, prevents human-induced errors on the data, presents very effective input reshaping, and also increases the classification accuracy.
The interplay of brain signals and the Internet of Things (IoT) is an emerging area. The processing of electroencephalographic (EEG) signals is mainly divided into two categories: (1) signal analysis for diagnosis and (2) brain-computer interfaces that can control devices through brain signals. Some applications of EEG are early detection of seizures, wheelchair control, and smart homes. Processing EEG is a complex task that requires expertise in fields such as neuroscience and computer science. There are frameworks to facilitate EEG processing, such as MNE-Python. Utilizing distributed and parallel processing is crucial in dealing with large-scale EEG data or when requiring scalable models. We have looked into platforms like WeBrain and a Spark-based framework to understand their capabilities for the issue. Based on our findings, we suggest a cloud framework. It supports the main features of previous works. It includes EEG processing capability using common machine learning (ML) and deep learning (DL) approaches based on the user’s required action paradigm. Since it is possible to run some trained models on limited resources devices in fog computing, an IoT device with caching capability is proposed to send and receive data. Collaboration and data sharing among users are made possible in this system.
The challenge to automatically design machine learning-based models for datasets with varying dimensions (features) remains open, especially in the context of unstructured and noisy data. Bayesian neural networks employ Markov chain Monte Carlo (MCMC) and variational inference methods for training (sampling) model parameters. However, the progress of MCMC methods in deep learning has been slow due to high computational requirements and uninformative priors of model parameters. Reversible jump MCMC allows sampling of model parameters of variable lengths; hence, it has the potential to train Bayesian neural networks effectively. In this paper, we implement reversible jump MCMC sampling for training dynamic Bayesian neural networks that feature cascaded neural networks with dynamic hidden and input neurons. Our dynamic Bayesian neural network can model unstructured data of varying dimensions and provide uncertainty quantification in model predictions. We apply the methodology to selected regression and classification problems from the literature. The results show that our method provides an effective approach for the dynamic exploration of models while featuring uncertainty quantification that not only caters to model parameters but also extends to model topology. This opens up the road for uncertainty quantification in dynamic deep learning models for dynamic and unstructured data streams.
A significant environmental problem in Vietnam and around the world is litter and garbage left on the street. The illegal dumping of garbage can have negative consequences on the environment and the quality of human life, as can all forms of pollution, in addition to having a large financial impact on communities. Thus, in order to reduce the impact, we require an automatic litter detecting method. In this study, we propose a new method for spotting illegal trash disposal in surveillance footage captured by real-world cameras. The illegal littering is recognized by a deep neural network. An ordered series of frames makes up a video. The order of the frames carries the temporal information, and each frame contains spatial information. To model both of these features, convolutional layers are used for spatial processing, and instead of using recurrent layers for obtaining temporal information, we take advantage of transformer encoder for encoding sequential data by evaluating each element in the sequence’s relevance. Besides, CNNs is also used to extract important features from images, hence lowering input size without compromising performance. This leads to reduce computational requirements for the transformer. Through testing on actual recordings of various dumping operations, we show that the proposed strategy is effective. Specifically, the validation outcomes from the testing data’s prediction reveal an accurate value of 92.5%, and this solution can be implemented in a real-time monitoring system.
Accurate and stable carbon price forecasts serve as a reference for assessing the stability of the carbon market and play a vital role in enhancing investment and operational decisions. However, realizing this goal is still a significant challenge, and researchers usually ignore multi-step-ahead and interval forecasting due to the non-linear and non-stationary characteristics of carbon price series and its complex fluctuation features. In this study, a novel hybrid model for accurately predicting carbon prices is proposed. The proposed model combines multi-step-ahead and interval carbon price forecasting based on the Hampel identifier (HI), time-varying filtering-based empirical mode decomposition (TVFEMD), and transformer model. First, HI identifies and corrects outliers in carbon price. Second, TVFEMD decomposes carbon price into several intrinsic mode functions (imfs) to reduce the non-linear and non-stationarity of carbon price to obtain more regular features in series. Next, these imfs are reconstructed by sample entropy (SE). Subsequently, the orthogonal array tuning method is used to optimize the transformer model’s hyperparameters to obtain the optimal model structure. Finally, after hyperparameter optimization and quantile loss function, the transformer is used to perform multi-step-ahead and interval forecasting on each part of the reconstruction, and the final prediction result is obtained by summing them up. Five pilot carbon trading markets in China were selected as experimental objects to verify the proposed model’s prediction performance. Various benchmark models and evaluation indicators were selected for comparison and analysis. Experimental results show that the proposed HI-TVFEMD-transformer hybrid model achieves an average MAE of 0.6546, 1.3992, 1.6287, and 2.2601 for one-step, three-step, five-step, and ten-step-ahead forecasting, respectively, which significantly outperforms other models. Furthermore, interval forecasts almost always have a PICI above 0.95 at a confidence interval of 0.1, thereby indicating the effectiveness of the hybrid model in describing the uncertainty in the forecasts. Therefore, the proposed hybrid model is a reliable carbon price forecasting tool that can provide a dependable reference for policymakers and investors.
Brain-computer interface (BCI) is a new communication and control technology established between human or animal brains and computer or other electronic equipment that does not rely on conventional brain information output pathways. The non-invasive BCI technology collects EEG signals from the cerebral cortex through signal acquisition equipment and processes them into signals recognized by the computer. The signals are preprocessed to extract signal features used for pattern recognition and finally are transformed into specific instructions for controlling external types of equipment. Therefore, the robustness of EEG signal representation is essential for intention recognition. Herein, we convert EEG signals into the image sequence and utilize the Local Relation Networks model to extract high-resolution feature information and demonstrate its advantages in the motor imagery (MI) classification task. The proposed method, MIIRvLR-Net, can effectively eliminate signal noise and improve the signal-to-noise ratio to reduce information loss. Extensive experiments using publicly available EEG datasets have proved that the proposed method achieved better performance than the state-of-the-art methods.
Recently, the rapid development of Artificial Intelligence (AI) applied in the Medical Internet of Things (MIoT) for the diagnosis of diseases such as epilepsy based on the investigation of electroencephalography (EEG) signals. Thanks to AI-based deep learning models, the procedure of epileptic seizure detection can be performed professionally in Smart Healthcare. However, the security issues for protecting sensitive medical EEG signals from disclosure and unauthorized operations from severe attacks over open networks. Therefore, there is a serious need for providing an effective method for encrypted EEG classification and prediction. In this paper, a new and efficient encrypted EEG data classification and recognition system using Chaotic Baker Map and Arnold Transform algorithms with Convolutional Neural Networks (CNNs). In this system, the channel's EEG time series is first converted into a 2D spectrogram image and then encrypted using Chaotic Baker Map and Arnold Transform algorithms, and finally fed to CNNs-based Transfer Learning (TL) models. From the experimental results, the proposed approach is validated and evaluated on a public CHB-MIT dataset and the googlenet with encrypted EEG images provides satisfactory performance by outperforming the models of other CNN like Alexnet, Resnet50, and squeezenet.
Recently, it is the key of brain-computer interface (BCI) technology to extract electroencephalogram (EEG) data features effectively and classify them accurately. As a challenging research topic in the field of BCI, the decoding technology of MI classification based on EEG can provide a reliable and convenient way of information interaction for patients with spinal cord injury, disability and stroke. However, the recorded EEG signal is easily interfered by other signals, which leads to its low signal-to-noise ratio. This paper proposes a model based on deep learning network to decode EEG-based MI actions, combining deep separation convolution network (DSCNN) and bidirectional long short-term memory (BLSTM) neural network. Firstly, DSCNN with two layers of logical convolution is used to extract the spatio-temporal feature information of EEG-based MI tasks, and then the spatial feature information is further extracted through the ordinary convolution network, which is connected into a one-dimensional vector by one layer, and then transmitted to BLSTM to further extract the temporal feature information. Finally, the EEG-based MI task is decoded by the softmax function. A prominent decoding rate is obtained to evaluate the performance of the model with the open physiological EEGMMIDB datasets, which is superior to other advanced models. The research of EEG-based MI algorithm model proposed can effectively promote the application of brain-computer interface technology in medical field.KeywordsBrain-computer interface (BCI)Electroencephalogram (EEG)DSCNNMotor imageryBLSTM
Context
Bug reports contain information that can be used by researchers and practitioners to better understand the bug fixing process and to enable the estimation of the effort necessary to fix bugs. In general, estimation models are built using the data (e.g., fixing time, severity, number of comments, number of attachments, and number of patches) present in the reports of fixed bugs (i.e., the report final’s state). However, we claim that this approach is not reliable in a real setting. Effort estimation is necessary for bug fix scheduling and team allocation tasks, which happens closer to the bug report opening than its closing. At that moment, the data available in the bug report is less informative than the data used to build the model, which may lead to an unrealistic estimation.
Objective
We propose a new approach to estimate bug-fixing time, i.e., the time span between the moment the bug was first reported until the bug is considered fixed. We consider not only the final state of the bug report to create our estimation model but all the previous available states, different from some previous studies that do not consider the reports’ updates. The concept of bug report evolution is used to create a dataset containing all investigated report states.
Method
First, we verify how often the bug reports and their fields are updated. Next, we evaluate our approach using different machine learning methods as a classification problem, with distinct output configurations, and class balancing techniques. The experimental analysis is performed with data from the JIRA issue tracking system of ten open-source projects. By leveraging the best models (considering all possible configurations) for the different states of the evolution of a bug report, we can assess whether there are significant differences in the models’ estimation ability due to the report’s state.
Results
We gathered evidence that the reports’ fields are updated often, which characterizes the reports’ evolution, impacting the building of bug-fixing estimation models. The models’ evaluation shows promising results 0.44 up to 0.85, precision values from 0.34 up to 0.74 and recall values from 0.62 up to 0.99, depending on the project.
Conclusions
Our experiments show that field updates have a meaningful impact on the models’ performance. Furthermore, we present a new approach to deal with the bug report evolution by considering each report version as an independent report. Finally, we also make available our dataset to the community.
Recently, Brain-Computer Interface (BCI) technology has been applied more and more in the field of clinical rehabilitation, which provides an effective way of communication for patients with brain disability and stroke. Because electroencephalogram (EEG) signals are extremely complex and contain many redundant signals, the recognition effect of BCI system based on EEG is not very well. The purpose of this paper is to solve the problem of brain intention classification in brain-computer interface system, which combines Deep Separation Convolutional Neural Network (DSCNN) and Gate Recurrent Unit (GRU) network to classify the motor imagination task of EEG. Firstly, one-dimensional timing EEG signals are transformed into two-dimensional array, and the temporal and spatial features of EEG signals are extracted by separate convolution. Then, these EEG signals containing spatio-temporal features are convolved once to extract spatial features, and the temporal features is extracted by GRU. Finally, the experiments shows that the final intention recognition accuracy reach 97.76% via the open physiological motor imagery data set EEGMMIDB, which is superior to some advanced research methods for motor imagery task recognition at present and helpful to restore the rehabilitation ability of patients with brain injury.
The capacity of machine objects to communicate autonomously is seen as the future of the Internet of Things (IoT), but machine-to-machine communication (M2M) is also gaining traction. In everyday life, security, transportation, industry, and healthcare all employ this paradigm. Smart devices have the ability to detect, handle, store, and analyze data, resulting in major network issues such as security and reliability. There are numerous vulnerabilities linked with IoT devices, according to security experts. Prior to performing any activities, it is necessary to identify and classify the device. Device identification and classification in M2M for secure telerobotic surgery are presented in this study. Telerobotics is an important aspect of the telemedicine industry. The major purpose is to provide remote medical care, which eliminates the requirement for both doctors and patients to be in the same location. This paper aims to propose a security and energy-efficient protocol for telerobotic surgeries, which is the primary concern at present. For secure telerobotic surgery, the author presents an Efficient Device type Detection and Classification (EDDC) protocol for device identification and classification in M2M communication. The periodic trust score is calculated using three factors from each sensor node. It demonstrates that the EDDC protocol is more effective and secure in detecting and categorizing rogue devices.
The brain–computer interface (BCI) control technology that utilizes motor imagery to perform the desired action instead of manual operation will be widely used in smart environments. However, most of the research lacks robust feature representation of multi-channel EEG series, resulting in low intention recognition accuracy. This article proposes an EEG2Image based Denoised-ConvNets (called EID) to enhance feature representation of the intention recognition task. Specifically, we perform signal decomposition, slicing, and image mapping to decrease the noise from the irrelevant frequency bands. After that, we construct the Denoised-ConvNets structure to learn the colorspace and spatial variations of image objects without cropping new training images precisely. Toward further utilizing the color and spatial transformation layers, the colorspace and colored area of image objects have been enhanced and enlarged, respectively. In the multi-classification scenario, extensive experiments on publicly available EEG datasets confirm that the proposed method has better performance than state-of-the-art methods.
The motor imagery brain-computer interface uses the human brain intention to achieve better control. The main technical problems are feature representation and classification of signal features for specific thinking activities. Inspired by the structure and function of the human brain, we construct a neural computing model to explore the critical issues in the representation and real-time recognition of the state of specific thinking activities. In consideration of the physiological structure and the information processing process of the brain, we construct a multi-scale cascaded Conv-GRU model and extract high-resolution feature information from the dual spatio-temporal dimension, effectively removing signal noise, improving the signal-to-noise ratio, and reducing information loss. Extensive experiments demonstrate that our model has a low dependence on training data size and outperforms state-of-the-art multi-intention recognition methods.
A large number of the population around the world suffers from various disabilities. Disabilities affect not only children but also adults of different professions. Smart technology can assist the disabled population and lead to a comfortable life in an enhanced living environment (ELE). In this paper, we propose an effective voice pathology assessment system that works in a smart home framework. The proposed system takes input from various sensors, and processes the acquired voice signals and electroglottography (EGG) signals. Co-occurrence matrices in different directions and neighborhoods from the spectrograms of these signals were obtained. Several features such as energy, entropy, contrast, and homogeneity from these matrices were calculated and fed into a Gaussian mixture model-based classifier. Experiments were performed with a publicly available database, namely, the Saarbrucken voice database. The results demonstrate the feasibility of the proposed system in light of its high accuracy and speed. The proposed system can be extended to assess other disabilities in an ELE.
Despite the active research into, and the development of, human activity recognition over the decades, existing techniques still have several limitations, in particular, poor performance due to insufficient ground-truth data and little support of intra-class variability of activities (i.e., the same activity may be performed in different ways by different individuals, or even by the same individuals with different time frames). Aiming to tackle these two issues, in this paper, we present a robust activity recognition approach by extracting the intrinsic shared structures from activities to handle intra-class variability, and the approach is embedded into a semi-supervised learning framework by utilizing the learned correlations from both labeled and easily-obtained unlabeled data simultaneously. We use l2,1 minimization on both loss function and regularizations to effectively resist outliers in noisy sensor data and improve recognition accuracy by discerning underlying commonalities from activities. Extensive experimental evaluations on four community-contributed public datasets indicate that with little training samples, our proposed approach outperforms a set of classical supervised learning methods as well as those recently proposed semi-supervised approaches.
Traumatic Brain Injury (TBI) is recognized as an important cause of death and disabilities after an accident. The availability a tool for the early diagnosis of brain dysfunctions could greatly improve the quality of life of people affected by TBI and even prevent deaths. The contribution of the paper is a process including several methods for the automatic processing of electroencephalography (EEG) data, in order to provide a fast and reliable diagnosis of TBI. Integrated in a portable decision support system called EmerEEG, the TBI diagnosis is obtained using discriminant analysis based on quantitative EEG (qEEG) features extracted from data recordings after the automatic removal of artifacts. The proposed algorithm computes the TBI diagnosis on the basis of a model extracted from clinically-labelled EEG records. The system evaluations have confirmed the speed and reliability of the processing algorithms as well as the system's ability to deliver accurate diagnosis. The developed algorithms have achieved 79.1% accuracy in removing artifacts, and 87.85% accuracy in TBI diagnosis. Therefore, the developed system enables a short response time in emergency situations and provides a tool the emergency services could base their decision upon, thus preventing possibly miss-diagnosed injuries.
Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.
The main idea of the current work is to use a wireless Electroencephalography (EEG) headset as a remote control for the mouse cursor of a personal computer. The proposed system uses EEG signals as a communication link between brains and computers. Signal records obtained from the PhysioNet EEG dataset were analyzed using the Coif lets wavelets and many features were extracted using different amplitude estimators for the wavelet coefficients. The extracted features were inputted into machine learning algorithms to generate the decision rules required for our application. The suggested real time implementation of the system was tested and very good performance was achieved. This system could be helpful for disabled people as they can control computer applications via the imagination of fists and feet movements in addition to closing eyes for a short period of time.
This paper presents a flexible standalone, low cost smart home system, which is based on the Android app
communicating with the micro-web server providing more than the switching functionalities. The Arduino
Ethernet is used to eliminate the use of a personal computer (PC) keeping the cost of the overall system to a
minimum while voice activation is incorporated for switching functionalities. Devices such as light
switches, power plugs, temperature sensors, humidity sensors, current sensors, intrusion detection sensors,
smoke/gas sensors and sirens have been integrated in the system to demonstrate the feasibility and
effectiveness of the proposed smart home system. The smart home app is tested and it is able successfully
perform the smart home operations such as switching functionalities, automatic environmental control and
intrusion detection, in the later case where an email is generated and the siren goes on.
People with motor and neurological impairments have little control over parts of their bodies, so they have great difficulty in walking. The development of solutions based on assistive technology dedicated to people with severe motor disabilities can provide accessibility and mobility, the intelligent wheelchair is an example of this type of technology. However, its use without proper training can be dangerous, a wheelchair simulator games can be a good tool for training people with severe disabilities. The EEG signals can be used as a source of information that allows communication between the brain and an intelligent wheelchair. This research aimed to develop a computer model to categorize electroencephalogram signals and control a virtual wheelchair using motor imagery of the left and right wrists, both wrists and both feet. Signs of electroencephalogram were acquired through the eegmmidb database — EEG Motor Movement/Imagery Dataset, captured by the BCI2000 system, and electroencephalogram signal samples from 10 individuals were used to validate the model. The techniques used are promising, making possible its use in three-dimensional simulation environments for intelligent wheelchair controlled by a brain-computer interface.
Objective:
Signal classification is an important issue in brain computer interface (BCI) systems. Deep learning approaches have been used successfully in many recent studies to learn features and classify different types of data. However, the number of studies that employ these approaches on BCI applications is very limited. In this study we aim to use deep learning methods to improve classification performance of EEG motor imagery signals.
Approach:
In this study we investigate convolutional neural networks (CNN) and stacked autoencoders (SAE) to classify EEG Motor Imagery signals. A new form of input is introduced to combine time, frequency and location information extracted from EEG signal and it is used in CNN having one 1D convolutional and one max-pooling layers. We also proposed a new deep network by combining CNN and SAE. In this network, the features that are extracted in CNN are classified through the deep network SAE.
Main results:
The classification performance obtained by the proposed method on BCI competition IV dataset 2b in terms of kappa value is 0.547. Our approach yields 9% improvement over the winner algorithm of the competition.
Significance:
Our results show that deep learning methods provide better classification performance compared to other state of art approaches. These methods can be applied successfully to BCI systems where the amount of data is large due to daily recording.
Classification of imagery motor tasks is the main challenge of analysing electroencephalography (EEG) data from a brain-computer interfaces (BCI) system. The noise and artifacts recorded by BCI system frequently corrupt imagery motor EEG data and reduce classification accuracy. In this paper, wavelet denoising algorithm is proposed to reduce noise from motor imagery EEG data and a power spectral density (PSD) feature selection method is used to improve classification accuracy. Experimental results show that the classification accuracy of the proposed method is significantly improved compared to the same PSD feature selection method without wavelet denoising. This result also certainly indicated that wavelet denoising algorithm successfully purified motor imagery EEG data and made classifying features more prominent.
Background:
Transcranial direct current stimulation (tDCS) is a promising treatment for major depressive disorder (MDD). Standard tDCS treatment involves numerous sessions running over a few weeks. However, not all participants respond to this type of treatment. This study aims to investigate the feasibility of identifying MDD patients that respond to tDCS treatment based on resting-state electroencephalography (EEG) recorded prior to treatment commencing.
Methods:
We used machine learning to predict improvement in mood and cognition during tDCS treatment from baseline EEG power spectra. Ten participants with a current diagnosis of MDD were included. Power spectral density was assessed in five frequency bands: delta (0.5-4Hz), theta (4-8Hz), alpha (8-12Hz), beta (13-30Hz) and gamma (30-100Hz). Improvements in mood and cognition were assessed using the Montgomery-Åsberg Depression Rating Scale and Symbol Digit Modalities Test, respectively. We trained the classifiers using three algorithms (support vector machine, extreme learning machine and linear discriminant analysis) and a leave-one-out cross-validation approach.
Results:
Mood labels were accurately predicted in 8 out of 10 participants using EEG channels FC4-AF8 (accuracy=76%, p=0.034). Cognition labels were accurately predicted in 10 out of 10 participants using channels pair CPz-CP2 (accuracy=92%, p=0.004).
Limitations:
Due to the limited number of participants (n=10), the presented results mainly aim to serve as a proof of concept.
Conclusions:
These finding demonstrate the feasibility of using machine learning to identify patients that will respond to tDCS treatment. These promising results warrant a larger study to determine the clinical utility of this approach.
The processing of electroencephalograms (EEGs) is a growing field where mature speech processing techniques are able to rapidly progress development and understanding of the associated neuroscience. I-vectors and Joint Factor Analysis (JFA), along with their foundational universal background models (UBMs) have progressed to a level of understanding that makes them prime for transition to the EEG community. To prove the capability of these techniques they are tested against two contrasting EEG data sets, PhysioNet's EEG Motor Movement/Imagery Dataset and the Temple University Hospital EEG Corpus, to highlight the effectiveness of the techniques with minimal domain knowledge modifications. The initial results, presented as equal error rates as low as 20%, support the development of these techniques as a viable approach to addressing subject verification within and across subjects.
Brain-computer interfaces (BCI) are devices that enable communication between a computer and humans by using brain activity as input signals. Brain imaging technology used in a BCI system is usually electroencephalography (EEG). In order to properly interpret brain activity, acquired signals from the brain have to be classified correctly. In this paper EEG signals are transformed by means of discrete wavelet transform. Thus the obtained signal features are used as inputs for a neural network classifier that should separate five different sets of EEG signals representing various mental tasks. Mean classification accuracy for the recognition of all five tasks was 90.75% and mean classification accuracy for the recognition of two tasks (baseline and any other mental task) was 99.87%. The same procedure was also used on the motor imagery dataset. A mean classification accuracy of 68.21% suggests alternative methods of feature extraction for motor imagery tasks.
In order to convenience of the disabled people living more convenient, using EEG signal is a good way, this paper design a smart home system based on the results of EEG signal studies, through this system can carry out the identify authentication in the room, can by EEG signal to control power switch. In order to reduce the noise signal interference, the high pass and low pass were used to cut extra frequencies, and in order to prominent the feature signal, the power spectrum method was used to convent the time domain signal to frequency domain, and then fisher distance was used to extraction the feature. All EEG signal was acquired by Neuroscan. In the simulated environment, identity authentication is using the visual evoked potential, and control switching is using motor imagery. The results showed, the subjects in identity authentication, an average of three times certification can identify subjects, switch control experiment, the accurate control of switching frequency to 86%.
Long Short-Term Memory (LSTM) is a specific recurrent neural network (RNN) architecture that was designed to model temporal sequences and their long-range dependencies more accurately than conventional RNNs. In this paper, we explore LSTM RNN architectures for large scale acoustic modeling in speech recognition. We recently showed that LSTM RNNs are more effective than DNNs and conventional RNNs for acoustic modeling, considering moderately-sized models trained on a single machine. Here, we introduce the first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines. We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance. This architecture makes more effective use of model parameters than the others considered, converges quickly, and outperforms a deep feed forward neural network having an order of magnitude more parameters.
Effectively extracting EEG data features is the key point in Brain Computer Interface technology. In this paper, aiming at classifying EEG data based on Motor Imagery task, Deep Learning (DL) algorithm was applied. For the classification of left and right hand motor imagery, firstly, based on certain single channel, a weak classifier was trained by deep belief net (DBN); then borrow the idea of Ada-boost algorithm to combine the trained weak classifiers as a more powerful one. During the process of constructing DBN structure, many RBMs (Restrict Boltzmann Machine) are stacked on top of each other by setting the hidden layer of the bottom layer RBM as the visible layer of the next RBM, and Contrastive Divergence (CD) algorithm was also exploited to train multilayered DBN effectively. The performance of the proposed DBN was tested with different combinations of hidden units and hidden layers on multiple subjects, the experimental results showed that the proposed method performs better with 8 hidden layers. The recognition accuracy results were compared with Support vector machine (SVM) and DBN classifier demonstrated better performance in all tested cases. There was an improvement of 4 – 6% for certain cases.
This brief presents a low-power, flexible, and multichannel electroencephalography (EEG) feature extractor and classifier for the purpose of personalized seizure detection. Various features and classifiers were explored with the goal of maximizing detection accuracy while minimizing power, area, and latency. Additionally, algorithmic and hardware optimizations were identified to further improve performance. The classifiers studied include $k$-nearest neighbor, support vector machines, naïve Bayes, and logistic regression (LR) . All feature and classifier pairs were able to obtain F1 scores over 80% and onset sensitivity of 100% when tested on ten patients. A fully flexible hardware system was implemented that offers parameters for the number of EEG channels, the number of features, the classifier type, and various word width resolutions. Five seizure detection processors with different classifiers have been fully placed and routed on a Virtex-5 field-programmable gate array and been compared. It was found that five features per channel with LR proved to be the best solution for the application of personalized seizure detection. LR had the best average F1 score of 91%, the smallest area and power footprint, and the lowest latency. The ASIC implementation of the same combination in 65-nm CMOS shows that the processor occupies 0.008 mm2 and dissipates 19 nJ at 484 Hz.
We introduce Adam, an algorithm for first-order gradient-based optimization
of stochastic objective functions. The method is straightforward to implement
and is based an adaptive estimates of lower-order moments of the gradients. The
method is computationally efficient, has little memory requirements and is well
suited for problems that are large in terms of data and/or parameters. The
method is also ap- propriate for non-stationary objectives and problems with
very noisy and/or sparse gradients. The method exhibits invariance to diagonal
rescaling of the gradients by adapting to the geometry of the objective
function. The hyper-parameters have intuitive interpretations and typically
require little tuning. Some connections to related algorithms, on which Adam
was inspired, are discussed. We also analyze the theoretical convergence
properties of the algorithm and provide a regret bound on the convergence rate
that is comparable to the best known results under the online convex
optimization framework. We demonstrate that Adam works well in practice when
experimentally compared to other stochastic optimization methods.
We present a simple regularization technique for Recurrent Neural Networks
(RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful
technique for regularizing neural networks, does not work well with RNNs and
LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show
that it substantially reduces overfitting on a variety of tasks. These tasks
include language modeling, speech recognition, and machine translation.
Predicting tDCS treatment outcomes of patients with major depressive disorder using automated EEG classification
- A M Al-Kaysi
- A Al-Ani
- C K Loo
- AM Al-Kaysi