Measuring working memory load with an electroencephalograph. Shot taken from experiment. dynamically adapt and support users’ goals [4]. Since we have traditionally interacted with computers through our physical bodies, most of these techniques have been based on observations of user actions and behavior (e.g. [23]). Less frequently, other techniques have utilized physiological signals as indicators of user state [14,21]. While these measures have been reasonably successful, they are rather indirect, especially when the user state in question is of a cognitive nature. Fortunately, advances in cognitive neuro- science and brain-sensing technologies provide us with the ability to interface more directly with the human brain. This is possible through the use of sensors that monitor the electrical and chemical changes within the brain that correspond with certain forms of thought. While using these technologies in HCI research has been previously articulated [12,28], we believe there is an opportunity to further explore practical issues with their use in HCI applications. In our work, we explore using one of these technologies, an electroencephalograph (EEG), to estimate or classify working memory load, or the cognitive effort dedicated to hold- ing information in the mind for short periods of time while performing a cognitive task [1]. Working memory has been shown to be a key component of cognitive load, and is a 

Context in source publication

Context 1

... reliable and unobtrusive measurement of working memory load could be used to evaluate the efficacy of interfaces and to provide real-time user-state information to adaptive systems. In this paper, we describe an experiment we conducted to explore some of the issues around using an electroencephalograph (EEG) for classifying working memory load. Within this experiment, we present our classification methodology, including a novel feature selection scheme that seems to alleviate the need for complex drift modeling and artifact rejection. We demonstrate classification accuracies of up to 99% for 2 memory load levels and up to 88% for 4 levels. We also present results suggesting that we can do this with shorter windows, much less training data, and a smaller number of EEG channels, than reported previously. Finally, we show results suggesting that the models we construct transfer across variants of the task, implying some level of generality. We believe these findings extend prior work and bring us a step closer to the use of such technologies in HCI research. Author Keywords: Brain-Computer Interface (BCI), electroencephalogram (EEG), cognitive load, memory load, machine-learning, feature selection, classification. ACM Classification Keywords : H.1.2 [User/Machine Systems]; H.5.2 [User Interfaces]: Input devices and strate- gies; B.4.2 [Input/Output Devices]: Channels and control- lers; J.3 [Life and Medical Sciences]. Human-computer interaction (HCI) researchers continually work on techniques that allow us to measure user states such as cognitive and memory workload, task engagement, surprise, satisfaction, or frustration. Such measures are useful not only for evaluating the efficacy of interfaces, but also for providing real-time information to systems that reasonable measure of how hard a user is working to solve a problem or use an interface. For example, working memory load has long been recognized in HCI to be an important indicator of potential errors as well as a predictive feature of procedural skill acquisition [3]. Given this evidence, interface designers often try to minimize the working memory load required to perform a task, and reliable real-time measures would benefit them greatly. While various researchers have worked on classifying working memory with EEG (e.g. [6,7,20,21,27]), previous work has typically relied on costly equipment and techniques that make it difficult for non-EEG-experts to replicate and use this work. Additionally, this work has often required experimenters to collect large amounts of classifier training data (sometimes on the order of days), a process that is often prohibitively expensive. While we believe that EEG is complementary to many of the other measures of memory and cognitive load, it is outside the scope of this paper to explore the detailed relationships between these measures. We leave this for future work. The contributions of this paper are three-fold: • First, we present our methodology within an experiment we ran to measure working memory load using only EEG signals. The innovation within this methodology is an automatic feature selection scheme that eliminates the need for procedures used in most previous work, such as complex device and physiological drift modeling as well as manual artifact rejection. • Second, using this methodology, we present classification results using machine learning techniques that replicate and extend prior work in the area. Specifically, we show classification accuracies of up to 99.0% between two load levels, and up to 88.0% between four levels, all with just 8 channels of EEG data. More importantly, we present results showing how classification accuracy varies with different temporal window sizes, amounts of training, and number of EEG channels. Specifically, the results suggest that our techniques allow us to attain accurate classification with less lag, much less training data, and simpler equipment. • Third, we show how our models work across variants of the memory task, providing encouraging evidence that it might be possible to develop canonical training tasks and to perform general classification of memory load. In this paper, we use an Electroencephalograph (EEG), a sensing technology that uses electrodes placed on the scalp to measure electrical potentials related to brain activity (see Figure 1). Each electrode typically consists of a wire leading to a conductive disk that is electrically connected to the scalp using conductive paste or gel. The EEG device re- cords the voltage at each of these electrodes relative to a reference point, which is often another electrode on the scalp. Because EEG is a non-invasive, passive measuring device, it is safe for extended and repeated use, a character- istic crucial for adoption in HCI research. Additionally, it does not require a highly skilled operator or medical procedure to use. For more information about electrical signals generated by the brain as well as EEG, see [5]. The signal provided by an EEG is, at best, a crude representation of brain activity due to the nature of the detector. Scalp electrodes are only sensitive to macroscopic and co- ordinated firing of large groups of neurons near the surface of the brain, and then only when they are directed along a perpendicular vector relative to the scalp. Additionally, because of the fluid, bone, and skin that separate the electrodes from the actual electrical activity, the already small signals are scattered and attenuated before reaching the electrodes. EEG data is typically analyzed by looking at the spectral power of the signal in a set of frequency bands, which have been observed to correspond with certain types of neural activity [5]. These frequency bands are commonly defined as 1-4 Hz (delta), 4-8 Hz (theta), 8-12 Hz (alpha), 12-20 Hz (beta-low), 20-30 Hz (beta-high), and >30 Hz (gamma). Early researchers observed the sensitivity of EEG to changes in mental effort. For example, Hans Berger [2] and others [11] report observing a decrease in the amplitude of the alpha (8-12 Hz) rhythm during mental arithmetic tasks Other researchers have shown that higher memory loads cause increases in theta (4-8 Hz) and low-beta (12-15 Hz) power in the frontal midline regions of the scalp [17], gamma (>30 Hz) oscillations [8], as well as inter-electrode correlation, coherence, cross phase, and cross power [24]. To test if alpha and theta bands were predictive of memory and cognitive loads in real world computing tasks, Smith et al. [27] compared EEG data when task difficulty was manipulated within a multi-attribute task battery (MATB) mul- titasking environment. They report successfully creating a user-specific index of task load, the average values of which increase with increasing task difficulty and differed significantly between the difficulty manipulations. Given this evidence of the existence of reliable indicators of memory load, researchers have attempted to build techniques that utilize these features to measure and classify memory load. Unfortunately, while these indicators may appear to be reliable when data is averaged over large time periods and many users, there is large variability within the signal for any given user at any given point in time. This makes using the features to classify memory loads an extremely difficult task. While it is reasonable to average the data when trying to make statements about the various rhythms, it is less useful when trying to classify user state in real time. For example, Jensen et al. found the increased theta power in only one of their ten subjects, and rather than an alpha decrease, they found that alpha power actually collected data from 8 users over three 6-8 hour sessions and present results showing ~95% classification accuracy between two levels of memory load [7]. They also showed relatively high cross-task and cross-session accuracies. However, subtle decisions made in their procedure leaves room for improvement. First, collecting 24 hours worth of training data from each user can be prohibitively high for some work. Second, they perform a Laplacian spatial en- hancement that requires accurate per-subject head measurements to filter noise from the signal. Third, they manu- ally inspect the data and throw out periods where there are artifacts in the data even after performing an automatic artifact rejection. This is tedious and requires expertise in read- ing the EEG signals. They report throwing away up to 20% of the data, which is not desirable in our targeted settings where data may be scarce. Furthermore, having to perform this manual step between training and classification has implications on real-time usability of the system. Finally, since their design interleaved different tasks, and used random hold out cross validation, they were training on data that was temporally fairly close to test data and we cannot be certain how well the models would generalize when applied to new data. In our work, we aim to replicate their high classification results and extend their work to further explore the space. We also set out to explore how various parameters such as temporal window size, amount of training data, and number of channels affect the classification. These factors are important to understand if EEG classification is to be used in HCI ...