Mental stress grading based on fNIRS signals

Abstract

In this study, we propose functional near infrared spectroscopy (fNIRS) to objectively grade different levels of mental stress. The levels of stress were set based on arithmetic task difficulty, time pressure and negative feedback about peer performance. We examined the proposed approach on twelve human subjects using the Montreal Imaging Stress Task. The experiment results revealed a reduction in cortical activations at prefrontal cortex when stressed, and the differences in hemodynamic response between control condition and under stress were significant with mean p-values of 0.0023, 0.00015 and 0.0004 for arithmetic difficulty level one, two and three, respectively. We thus confirm the feasibility of fNIRS in grading mental stress.

Full text


Abstract— In this study, we propose functional near infrared
spectroscopy (fNIRS) to objectively grade different levels of
mental stress. The levels of stress were set based on the
difficulty of arithmetic task, time pressure and negative
feedback about peer performance. We examined the proposed
approach on twelve human subjects using the Montreal
Imaging Stress Task. The experiment results revealed a
reduction in cortical activation at prefrontal cortex when
stressed, and the differences in hemodynamic response between
control condition and under stress were significant for
arithmetic difficulty level one, two and three, respectively, (p =
0.0023, 0.0001 and 0.0004). The experiment results thus support
the suggestion of fNIRS in grading mental stress.
Keywords—Stress, fNIRS, neuroimaging
I. INTRODUCTION
Conventionally, questionnaires are used as a tool to
assess mental stress. However, such method is subjective
[1]. Stress has been reported to activate the hypothalamus-
pituitary-adrenocortical axis (HPA axis) and sympathetic
nervous system (SNS) causing an increase in cortisol
secretion in the adrenal cortex. The level of cortisol is
widely accepted as a biomarker of stress [2].
Besides cortisol, stress can be detected from human bio-
signals [3, 4]. Researchers have found a relationship
between salivary cortisol levels and physiological variable
changes such as heart rate variability (HRV), electrodermal
response (EDR) and blood pressure (BP) [4, 5]. Stress
causes a decrease in the high frequency components of heart
beat interval and an increase in the low frequency
components, respectively. Skin conductivity on the other
hand varies with the changes in skin moisture level revealing
the changes in SNS. It has been reported to increase during a
stressful task and can be acquired using a galvanic skin
response (GSR) sensor [6].
Changes in autonomous nervous system (ANS) can also
be represented by electroencephalography (EEG) signals [7].
EEG is one of the most studied non-invasive neuroimaging
modality that measures the electric potential of cortical
activation. EEG has the advantages of temporal resolution,
ease of use, and low set-up cost. Its signal components are
categorized by frequency bands; Delta (0.5-4 Hz), Theta (4-
8 Hz), Alpha (8-13 Hz) and Beta (14-30 Hz). Each of the
frequency band can be used as an indicator of one’s mental
state. However, EEG has some limitations as it has poor
*Research supported by Ministry of Education, Malaysia under the
HiCOE grant to Centre for Intelligent Signal and Imaging Research, UTP.
1Fares. Al-shargie and Tong Boon Tang are with the Centre for
Intelligent Signal and Imaging Research, Universiti Teknologi
PETRONAS, Malaysia. tongboon.tang@petronas.com.my
2Masashi Kiguchi is with Hitachi. Ltd., Research& Development Group,
350-0395,Japan.
spatial resolution and its signals are susceptive to noise. To
overcome these limitations, we proposed a new
neuroimaging modality to objectively grade the levels of
mental stress.
Functional Near-infrared Spectroscopy (fNIRS) is a non-
invasive brain imaging technology based on hemodynamic
responses to cortical activation [8]. It uses near-infrared light
in the wavelength range 650-900 nm and estimates the
changes in oxygenated and deoxygenated hemoglobin
concentrations (O2Hb and HHb) using modified Beer-
Lambert law [9]. fNIRS has several advantages over other
neuro-imaging modalities. Compared to EEG, it has better
spatial resolution and less affected by noise [10]. Compared
to functional magnetic resonance imaging (fMRI) and
positron emission topography (PET), fNIRS is portable,
cheaper and does not confine the subjects to lying position.
In this study, we aim to grade mental stress by measuring
the hemodynamic response at the prefrontal cortex (PFC).
PFC is the brain region responsible for regulating thoughts,
actions and emotions. It is also the most sensitive area in the
brain to detrimental effects of stress exposure [11].
According to previous fMRI studies [12, 13], solving
arithmetic tasks under time pressure induces mental stress on
the PFC. We developed three levels of arithmetic task
difficulty to induce stress on experiment participants.
Besides the task difficulty, time pressure and negative
feedback of peer performance were used as stressors in this
study. To the best of our knowledge, this is the first study to
quantify the levels of stress based on hemodynamic
responses to cortical activation.
II. METHODOLOGY
A. Subjects
Twelve male healthy, right-handed adults (aged 22 ± 4)
participated in this study. All participants were informed
prior to the experiment and gave written consent, in
accordance with the Declaration of Helsinki and ethics
approval granted by local ethics review committee at
Universiti Teknologi PETRONAS. None of the participants
had a history of psychiatric, neurological illness or
psychotropic drug use. To avoid any environmental stress, all
participants were seated in a comfortable chair in a room with
good air condition.
B. Stress stimuli
The experiment was developed based on the Montreal
Imaging Stress Task (MIST) [13]. In this study, the
arithmetic task was defined at three levels of difficulty,
where each level corresponded to one level of stress. The
task at level one (L1) involved 3-one digit integer (ranging
from 0 to 9) and used the operands of + or – (example 9+1-
Mental Stress Grading Based on fNIRS Signals
Fares Al-shargie,1 Tong Boon Tang,1 Masashi Kiguchi2
978-1-4577-0220-4/16/$31.00 ©2016 IEEE 5140

6). At level two (L2), the task involved 3 integers (ranging
from 0 to 99) with at least 2 two-digit integers using the
operands of +, –, and × (example 12×3-30). At level three
(L3), the task involved 4 integer numbers (ranging from 0 to
99) and the operands include +, –, ×, / and ÷ (example 7-
99/3+35). Besides the task difficulty, time pressure and
negative feedback about peer performance were
implemented to induce stress on the participants.
Participants were first trained at each level of task difficulty
and the average time for each individual in answering the
questions was recorded. This recorded time was then
reduced by 10% and used as time pressure on the
participants. Moreover, feedback of answering the questions
(“correct”, “incorrect” or “timeout”) and performance
indicators (one for the participant’s performance and one for
the averaged peer performance fixed at 90% accuracy) were
displayed on the computer monitor to further induce stress in
experiment participants.
C. Experiment procedure
Participants were instructed to avoid any head and body
movements and deep breathing during fNIRS measurements.
The experiment was performed in four successive sessions.
In first session, a brief introduction was given to each
participant to be familiar with the proposed tasks. In second
session, the participants were trained for five minutes at each
level of difficulty in the mental arithmetic (MA) task to
estimate average time taken to answer each question. In third
session (i.e. control session), the fNIRS cap was attached to
the participant’s head and fNIRS signals were recorded for a
total duration of 15 minutes while solving arithmetic
problems at three levels of difficulty without any time limit
per question. After fNIRS recording, a questionnaire was
filled by the participants as self-reporting about task loading
according to NASA-TLX rating scale [14]. The fourth
session (i.e. stress phase) was similarly as the control phase
where the fNIRS was recorded for 15 minutes but under
stress conditions (time limit and negative feedback). Similar
questionnaire about task loading was again completed. The
entire recording duration for each participant is ~ 1 hour.
Fig. 1 gives an overview of the experiment protocol and
the block design. Each block consists of 40 seconds of
mental arithmetic task and 30 seconds of rest. During the 40
seconds task, participants were shown mental arithmetic
tasks on a computer screen and had to solve them as fast as
they could (i.e. the control session) or within a given amount
of time (i.e. the stress session). During the 30 s rest,
participants needed to focus on a fixation cross with black
background to sustain their attention to the monitor display.
D. Functional near infrared spectroscopy
Relative concentrations of oxygenated and deoxygenated
hemoglobin were recorded with 16 optodes (8 sources and 8
detectors) of fNIRS system OT-R40 (Hitachi Medical,
Japan) using two infrared wavelengths (695 and 830 nm).
The sampling frequency was set to 10 Hz. The distance
between pairs of source and detector probes was set to 3.0
cm. Channel (Ch) was defined as the measurement area
between a pair of source-detector optodes. A total of 27
channels were measured in this study and PFC was the
region of interest. Fig. 2 shows the full configuration of
measurement channels.
E. fNIRS analysis
The fNIRS signals were transformed to concentration
changes of oxygenated, deoxygenated and total hemoglobin
using modified Beer-Lambert law [9]. In order to reduce the
noise and motion artifacts, fNIRS signals passed through
several pre-processing steps using the plug-in-based analysis
software Platform for Optical Topography Analysis Tool
(developed by Hitachi, CRL; run on MATLAB)[15]. The
process involved removing the motion artifacts [16],
filtering the signal in the range of 0.012 to 0.8 Hz using a 5th
order Butterworth filter, baseline correction, and moving
average. In baseline correction, we defined a period from 5s
prior to task condition to the end of each MA task as an
analysis block. Linear regression by least mean square
method was applied to remove any dc drift in fNIRS
recordings [16] before we averaged all the blocks and
performed statistical analysis on the changes of O2Hb.
Figure 1. Experiment protocol of mental stress study. The labels L1, L2 and L3 represent the levels of mental arithmetic task and MA stands for mental
arithmetic. Six recordings were performed in this experiment; three for control condition and three for stress condition. In each recording, there were four
blocks. In each block, mental arithmetic task was allocated for 40 s followed by 30 s rest.
5141

Figure 2. Measurement setup: (a) channel configuration and (b) probe
holder worn on the PFC area. Eight sources and eight detectors used in the
measurements corresponding to 27-channels over the PFC area.
III. RESULT AND DISCUSSION
The post-processed fNIRS measurements from the
twelve subjects were subsequently averaged channel-by-
channel for each difficulty level and experiment condition
(control/stress). The results obtained from level one (i.e.
control phase) demonstrated significant increase in O2Hb
due to MA task, as compared to baseline. Under stress
condition, however, there was a significant reduction in this
change in O2Hb. This revealed that stress might have
impaired the PFC and resulted in reduced cortical activities.
Fig. 3(a) and (b) illustrate the topographical map of O2Hb
under control and stress conditions, respectively. The
channel numbers were marked on the topography maps to
indicate their particular locations in the PFC area. Red
colour indicates high concentration level and blue colour
indicates less concentration level of O2Hb.
Similar results were obtained at level two and level three
of arithmetic task difficulty where a significant decline was
observed in the concentration change of oxygenated
hemoglobin under stress condition. Fig. 4 and Fig. 5
illustrate the topographical map of O2Hb under control and
stress conditions, at level two and level three, respectively.
In addition, we studied, at each level of mental stress, the
difference in hemodynamic response to MA task under the
control and the stress conditions, and their correlation with
mental stress level using two-sample t-test analysis. We
found a significant decrease in oxygenated hemoglobin
concentration from control condition to different stress
conditions with mean p-value of 0.0023, 0.0001 and 0.0004
for level one, level two and level three, respectively. In
addition, there was also a significant reduction in
hemodynamic response as the level of stress increased from
level one to higher levels, with p < 0.01.
We further examined the performance of each participant
in answering the arithmetic questions (accuracy score), and
how they were related to the hemodynamic response. Fig. 6
shows the relationship between the accuracy score by each
participant (twelve participants in total) and the mean
change in oxygenated hemoglobin concentration, at level 1
difficulty. We found a good agreement in their performance
when their cortical activation was affected by stress (R2 =
0.86963). When cross-checked with the results of self-
reporting about task loading questionnaires, NASA-TLX
rating scales showed no significant differences among the
three mental stress levels. Admittedly the sample size is
limited, the experiment results suggested that subjective
assessment using questionnaire might not be sensitive
enough as a tool for quantifying mental stress levels.
Right Left
(a)
(b)
Figure 3. Topographical map of oxygenated hemoglobin, (a) under control
condition and (b) under stress condition, at level 1 for average of 12
subjects. Red colour indicates high concentration levels of oxygenated
hemoglobin and blue colour indicates less activation or distortion.
Right Left
(a)
(b)
Figure 4. Topographical map of oxygenated hemoglobin, (a) under control
condition and (b) under stress condition, at level 2 for average of 12
subjects. Red colour indicates high concentration levels of oxygenated
hemoglobin and blue colour indicates less activation.
5142

Furthermore, we found the hemodynamic response to
mental stress were highly localized. Based on the repeated
measurements (three measurements for each level of mental
stress), we observed a high reduction in oxygenated
hemoglobin concentration on the right PFC in all the three
levels of mental stress. To confirm the right dominant of
PFC to mental stress, we calculated the laterality index at the
three levels of stress, where LIS = (right-left) / (right+left).
LIS <0 indicates high stress on the right PFC, while LIS >0
indicates the dominance of left PFC. The results showed
right PFC dominant with mean LIS value of -0.0821 and -
0.1428 at level one and level two of mental stress,
respectively, with p < 0.05. But, at level three, the result was
not significant, perhaps due to the excessive level of stress.
Right Left
(a)
(b)
Figure 5. Topographical map of oxygenated hemoglobin, (a) under control
condition and (b) under stress condition, at level 3 for average of 12
subjects. Red colour indicates high concentration levels of oxygenated
hemoglobin and blue colour indicates less activation.
Figure 6. The graph shows the experiment result where the mean change in
oxygenated hemoglobin concentration is positively correlated with
performance (accuracy score).
IV. CONCLUSION
In this study, we investigated if fNIRS signals could be
used in grading mental stress. Using time pressure and
negative feedback about peer performance as stressors, the
experiment results showed that hemodynamic response at
the PFC was significantly different among the levels of
arithmetic difficulty/stress. Using laterality index, we found
right PFC as the brain region more sensitive to mental stress.
The results from questionnaire approach (self-reporting
about task load) indicated that the engagement of
participants reduced when stressed, but were not significant
to discriminate different levels of stress. In short, our study
supports the suggestion of using fNIRS to objectively grade
the levels of mental stress.
REFERENCES
[1] T.-K. Liu, Y.-P. Chen, Z.-Y. Hou, C.-C. Wang, and J.-H. Chou,
"Noninvasive evaluation of mental stress using by a refined rough set
technique based on biomedical signals," Artificial intelligence in
medicine, vol. 61, pp. 97-103, 2014.
[2] D. H. Hellhammer, S. Wüst, and B. M. Kudielka, "Salivary cortisol as
a biomarker in stress research," Psychoneuroendocrinology, vol. 34,
pp. 163-171, 2009.
[3] A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion
recognition using bio-sensors: First steps towards an automatic
system," in ADS, 2004, pp. 36-48.
[4] J. Healey and R. W. Picard, "Detecting stress during real-world
driving tasks using physiological sensors," Intelligent Transportation
Systems, IEEE Transactions on, vol. 6, pp. 156-166, 2005.
[5] H. Ashton, R. Savage, J. W. Thompson, and D. Watson, "A method
for measuring human behavioural and physiological responses at
different stress levels in a driving simulator," British journal of
pharmacology, vol. 45, pp. 532-545, 1972.
[6] C. L. Lisetti and F. Nasoz, "Using noninvasive wearable computers to
recognize human emotions from physiological signals," EURASIP
journal on applied signal processing, vol. 2004, pp. 1672-1687, 2004.
[7] F. Al-shargie, T. Tang, N. Badruddin, and M. Kiguchi, "Mental Stress
Quantification Using EEG Signals," in Int. Conf. for Innovation in
Biomedical Engineering and Life Sciences, 2016, pp. 15-19.
[8] S. Cutini, S. B. Moroa, and S. Biscontib, "Functional near infrared
optical imaging in cognitive neuroscience: an introductory review,
Journal of Near Infrared Spectroscopy, vol. 20, pp. 75-92," 2012.
[9] A. Sassaroli and S. Fantini, "Comment on the modified Beer? Lambert
law for scattering media," Physics in Medicine and Biology, vol. 49, p.
N255, 2004.
[10] Y. Hoshi, "Towards the next generation of near-infrared
spectroscopy," Philosophical Transactions of the Royal Society of
London A: Mathematical, Physical and Engineering Sciences, vol.
369, pp. 4425-4439, 2011.
[11] A. F. Arnsten, "Stress signalling pathways that impair prefrontal
cortex structure and function," Nature Reviews Neuroscience, vol. 10,
pp. 410-422, 2009.
[12] C. D'Aguiar, "What stress does to your brain: a review of
neuroimaging studies," Canadian Journal of Psychiatry, vol. 54, p. 6,
2009.
[13] K. Dedovic, R. Renwick, N. K. Mahani, V. Engert, S. J. Lupien, and J.
C. Pruessner, "The Montreal Imaging Stress Task: using functional
imaging to investigate the effects of perceiving and processing
psychosocial stress in the human brain," Journal of Psychiatry and
Neuroscience, vol. 30, p. 319, 2005.
[14] S. G. Hart and L. E. Staveland, "Development of NASA-TLX (Task
Load Index): Results of empirical and theoretical research," Advances
in psychology, vol. 52, pp. 139-183, 1988.
[15] S. Sutoko, H. Sato, A. Maki, M. Kiguchi, Y. Hirabayashi, H.
Atsumori, et al., "Tutorial on platform for optical topography analysis
tools," Neurophotonics, vol. 3, pp. 010801-010801, 2016.
[16] H. Sato, N. Tanaka, M. Uchida, Y. Hirabayashi, M. Kanai, T. Ashida,
et al., "Wavelet analysis for detecting body-movement artifacts in
optical topography signals," Neuroimage, vol. 33, pp. 580-587, 2006.
5143