Table 1 - uploaded by Ralf Rummer
Content may be subject to copyright.
Examples of ways in which an adaptive system might obtain information about causes of a user's resource limitations.
Source publication
One of the central questions addressed in the project READY was that of how a system can automatically recognize situationally determined resource limitations of its user—in particular, time pressure and cognitive load. This chapter summarizes most of the work done in READY on this topic, presenting as well some previously unpublished results. We f...
Context in source publication
Context 1
... evidence that suggests the presence of such a factor constitutes indirect evidence for the corresponding resource limitation. Table 1 gives some examples of the many possibilities. Requirement for fast response imposed by S itself (e.g., instruction by S to perform a given action quickly) S's access to its own processing history ...
Similar publications
Especially in mobile systems, one important part of the context of use involves psychological variables like cognitive load and time pressure. This paper looks at one possible way of capturing such aspects of context: the analysis of the features of the users' speech. In a replication and extension of an earlier study of our group, we created four...
Citations
... In collaboration load, speakers consider the mental effort of others and their actions, in order to predict their behaviour and take actions (Kolfschoten et al., 2012). Cognitive load has been measured in psycholinguistics utilising speech features such as pauses, articulation rate and disfluencies (Müller et al., 2001;Jameson et al., 2010;Womack et al., 2012). It has also been demonstrated that the more time a speaker takes to produce an utterance, the more cognitive resources are required (Schilperoord, 2002). ...
In face-to-face interaction, speakers establish common ground incrementally, the mutual belief of understanding. Instead of constructing “one-shot” complete utterances, speakers tend to package pieces of information in smaller fragments (what Clark calls “installments”). The aim of this paper was to investigate how speakers' fragmented construction of utterances affect the cognitive load of the conversational partners during utterance production and comprehension. In a collaborative furniture assembly, participants instructed each other how to build an IKEA stool. Pupil diameter was measured as an outcome of effort and cognitive processing in the collaborative task. Pupillometry data and eye-gaze behaviour indicated that more cognitive resources were required by speakers to construct fragmented rather than non-fragmented utterances. Such construction of utterances by audience design was associated with higher cognitive load for speakers. We also found that listeners' cognitive resources were decreased in each new speaker utterance, suggesting that speakers' efforts in the fragmented construction of utterances were successful to resolve ambiguities. The results indicated that speaking in fragments is beneficial for minimising collaboration load, however, adapting to listeners is a demanding task. We discuss implications for future empirical research on the design of task-oriented human-robot interactions, and how assistive social robots may benefit from the production of fragmented instructions.
... These include language-based and emotionbased behaviours. The language-based behaviours include hesitations, increased use of pauses, decreased articulation rate, decreased speech rate, self-corrections and several others [3,26,29,34]. The emotion-based behaviours include increased use of negative emotions, decreased use of positive emotions, and several other indicators [27,28]. ...
Cognitive load has been widely studied to help understand human performance. It is desirable to monitor user cognitive load in applications such as automation, robotics, and aerospace to achieve operational safety and to improve user experience. This can allow efficient workload management and can help to avoid or to reduce human error. However, tracking cognitive load in real time with high accuracy remains a challenge. Hence, we propose a framework to detect cognitive load by non-intrusively measuring physiological data from the eyes and heart. We exemplify and evaluate the framework where participants engage in a task that induces different levels of cognitive load. The framework uses a set of classifiers to accurately predict low, medium and high levels of cognitive load. The classifiers achieve high predictive accuracy. In particular, Random Forest and Naive Bayes performed best with accuracies of 91.66% and 85.83% respectively. Furthermore, we found that, while mean pupil diameter change for both right and left eye were the most prominent features, blinking rate also made a moderately important contribution to this highly accurate prediction of low, medium and high cognitive load. The existing results on accuracy considerably outperform prior approaches and demonstrate the applicability of our framework to detect cognitive load.
... Existing methods for measuring CL or ML are based on using subjectivequestionnaires [12], performance-based metrics such as response time, error-rate or accuracy [21], speech-based [5,16,17,30], physiological behavior-based methods based on measuring human organs [21]: 1) brain, through measuring electroencephalogram (EEG) or electrocardiogram (ECG), 2) heart, through measuring Heart-Rate (HR), or Heart-Rate Variability (HRV), 3) skin, through measuring Galvanic Skin Conductance (GSR), and 4) eyes, through measuring eye movements, Pupil Diameter (PD), or Blinking-Rate (BR) to measure CL or ML in humans during an experimental setting. With the advancements in the field of machine learning and artificial intelligence, CL can be estimated in real-time. ...
We present an experiment investigating the relationships between different physiological measures, including Mean Pupil Diameter Change, Blinking-Rate, Heart-Rate, and Heart-Rate Variability to inform the development of a measure to estimate Cognitive Load. Our experiment involved participants performing a task to spot correct or incorrect words and sentences which successfully induced Cognitive Load. Our results show that participants’ task performance predicts their subjective rating of Cognitive Load and that there was a decrease in participants’ performance with an increase in Cognitive Load. Furthermore, Mean Pupil Diameter Change was able to predict Blinking-Rate, and Heart-Rate was able to predict Heart-Rate Variability. This prediction is evidence that collecting data on physiological behaviours synchronously and analysing the trends can be an effective way of estimating Cognitive Load, and will help the future development of an online measure of Cognitive Load useful for responsive user interfaces.
... A perhaps remarkable gauge developed through the recent years is a real-time evaluation of speech features linked to MWL. Some of these potential features were initially features such as the number of sentence fragments and articulation (Berthold & Jameson, 1999) and high level features used in a Bayesian network (Jameson et al., 2009;Müller, Großmann-Hutter, Jameson, Rummer, & Wittig, 2001). ...
The industrial workplace is currently transforming into a knowledge driven and highly digitalized work environment yielding small batches of highly complex products. Coined 'Industry 4.0', this workplace is expected to put higher demands on the operator’s mental processing. In order to support the optimization of operator mental workload on the future shop floor, this interdisciplinary doctorate builds on seminal research in psychology and industrial design to help provide practitioners with a robust mental workload measure. In the conceptualization stage, a clear-cut conceptual definition of mental workload and a generic and implementable explanatory framework is presented. During the operationalization stage, first, the external validity of mobile pupillometry is explored in an assembly experiment. Secondly, a novel mental workload measure is designed and validated, based on observable assembly behavior such as 'freezing' of the hands and the amount of assembly part rotations made. The implementation stage, finally, gauges the wearability and applicability of these measures, to conclude with two studies on the employee acceptability of mental workload monitoring. Envisioning the technology’s future development the dissertation suggests that research should invest in long-term within-subjects triangulated field measurement, rooted in common theoretical and methodological grounds and supported by dynamic machine learning.
... Pupil dilation and blink rate do not require known focus regions; however, the metrics are greatly confounded by light and fatigue, respectively. Speech-response time and noise level are indicators of auditory workload [6,8,21], while speech workload can be assessed using speech-based measures (e.g., speech-response time, speech rate, and filler utterances [6,30]). Heart rate [18,32], respiration rate [33,47], skin temperature [39,40], and posture-based metrics (e.g., posture sway and posture magnitude [20,35,43]) correlate to physical workload. ...
High-stress environments, such as first-response or a NASA control room, require optimal task performance, as a single mistake may cause monetary loss or even the loss of human life. Robots can partner with humans in a collaborative or supervisory paradigm to augment the human’s abilities and increase task performance. Such teaming paradigms require the robot to appropriately interact with the human without decreasing either’s task performance. Workload is related to task performance; thus, a robot may use a human’s workload state to modify its interactions with the human. Assessing the human’s workload state may also allow for dynamic task (re-)allocation, as a robot can predict whether a task may overload the human and, if so, allocate it elsewhere. A diagnostic workload assessment algorithm that accurately estimates workload using results from two evaluations, one peer based and one supervisory based, is presented. The algorithm correctly classified workload at least 90% of the time when trained on data from the same human--robot teaming paradigm. This algorithm is an initial step toward robots that can adapt their interactions and intelligently (re-)allocate tasks.
... Speech-response time and noise level are indicators of auditory workload (Berthold & Jameson, 1999;Brenner, Doherty, & Shipp, 1994;Harris, 2011) and can be measured using wearable microphones. Wearable microphones can also collect speechbased measures (e.g., speech-response time, speech rate, and filler utterances; Berthold & Jameson, 1999;Jameson et al., 2010), which can be used to assess speech workload. Physical workload may be estimated using HR (Hankins & Wilson, 1998;Jorna, 1993), respiration rate (RR; Keller, Bless, Blomann, & Kleinbohl, 2011;Roscoe, 1992), skin-temperature (ST; Miyake et al., 2009;Mizuno et al., 2016), and posture-based metrics (e.g., posture sway and posture magnitude [PM]; Harriott, Zhang, & Adams, 2013;Lasley, Hamer, Dister, & Cohn, 1991;Paul, Kuijer, Visser, & Kemper, 1999), where such measures can be collected using a chest-strap or similar device. ...
Humans commanding and monitoring robots’ actions are used in various high-stress environments, such as the Predator or MQ-9 Reaper remotely piloted unmanned aerial vehicles. The presence of stress and potential costly mistakes in these environments places considerable demands and workload on the human supervisors, which can reduce task performance. Performance may be augmented by implementing an adaptive workload human–machine teaming system that is capable of adjusting based on a human’s workload state. Such a teaming system requires a human workload assessment algorithm capable of estimating workload along multiple dimensions. A multi-dimensional algorithm that estimates workload in a supervisory environment is presented. The algorithm performs well in emulated real-world environments and generalizes across similar workload conditions and populations. This algorithm is a critical component for developing an adaptive human–robot teaming system that can adapt its interactions and intelligently (re-)allocate tasks in dynamic domains.
... They explored the potential of different frame based features in English. Other studies [5,8] related to automatic load classification were conducted on utterance level features. They explored articulation rate, pause percentage, pause rate, onset delay, etc. with different learning techniques, such as Bayesian Network, Decision tree, Bayesian Network and Multilayer perceptron. ...
... Jameson et al. [8] explored the recognition of resource limitations, user's available time and working memory, based on speech, and statistically analyzed seven speech features (Number of syllables, articulation rate, silent pauses, filled pauses, hesitations, onset latency and disfluencies) under different time pressure, announcement and task (whether navigation or not) conditions to investigate the changes in these speech features. They also used Bayesian network to learn a user model to recognize the cognitive load of computer users based on speech. ...
... Pause length (pl) [8,12] could discriminate three load levels for English, and medium from non-medium for Chinese. Previous study showed an increase in pl when load was higher without time pressure, and decrease with time pressure when physical stress (moving around) is not present [8], which is consistent with our finding; There were significant difference between languages when load was high, which suggested that load difference would cause feature difference between languages. ...
Speech-based cognitive load modeling recently proposed in English have enabled objective, quantitative and unobtrusive evaluation of cognitive load without extra equipment. However, no evidence indicates that these techniques could be applied to speech data in other languages without modification. In this study, a modified Stroop Test and a Reading Span Task were conducted to collect speech data in English and Chinese respectively, from which twenty non-linguistic features were extracted to investigate whether they were language dependent. Some discriminating speech features were observed language dependent, which serves as an evidence that there is a necessity to adapt speech-based cognitive load detection techniques to diverse language contexts for a higher performance.
... Voice source characteristics, including the variation of primary open quotient, normalized amplitude quotient, and primary speed quotient, are also effective cues [7]. Voice quality features (creakiness, harmonicsto-noise ratio), glottal flow shape, speech phase (group delay, FM parameter), and lexical/linguistic information (e.g., the duration and number of pauses and fillers) were also reported as important cues for detecting cognitive load level [1,7,8]. ...
... Voice source characteristics, including the variation of primary open quotient, normalized amplitude quotient, and primary speed quotient, are also effective cues [7]. Voice quality features (creakiness, harmonicsto-noise ratio), glottal flow shape, speech phase (group delay, FM parameter), and lexical/linguistic information (e.g., the duration and number of pauses and fillers) were also reported as important cues for detecting cognitive load level [1,7,8]. ...
... Voice source characteristics, including the variation of primary open quotient, normalized amplitude quotient, and primary speed quotient, are also effective cues [7]. Voice quality features (creakiness, harmonicsto-noise ratio), glottal flow shape, speech phase (group delay, FM parameter), and lexical/linguistic information (e.g., the duration and number of pauses and fillers) were also reported as important cues for detecting cognitive load level [1,7,8]. ...
The goal in this work is to automatically classify speakers' level of cognitive load (low, medium, high) from a standard battery of reading tasks requiring varying levels of working memory. This is a challenging machine learning problem because of the inherent difficulty in defining/measuring cognitive load and due to intra-/inter-speaker differences in how their effects are man-ifested in behavioral cues. We experimented with a number of static and dynamic features extracted directly from the audio signal (prosodic, spectral, voice quality) and from automatic speech recognition hypotheses (lexical information, speaking rate). Our approach to classification addressed the wide vari-ability and heterogeneity through speaker normalization and by adopting an i-vector framework that affords a systematic way to factorize the multiple sources of variability.
