Content uploaded by Abdurrahman Özbeyaz
Author content
All content in this area was uploaded by Abdurrahman Özbeyaz on Mar 14, 2018
Content may be subject to copyright.
Signal, Image and Video Processing
https://doi.org/10.1007/s11760-018-1269-x
ORIGINAL PAPER
Familiar/unfamiliar face classification from EEG signals by utilizing
pairwise distant channels and distinctive time interval
Abdurrahman Özbeyaz1·Sami Arıca2
Received: 29 March 2017 / Revised: 27 February 2018 / Accepted: 5 March 2018
© Springer-Verlag London Ltd., part of Springer Nature 2018
Abstract
The aim of the study is to classify single trial electroencephalogram and to estimate active regions/locations on skull in
unfamiliar/familiar face recognition task. For this purpose, electroencephalographic signals were acquired from ten subjects in
different sessions. Sixty-one familiar and fifty-nine unfamiliar face stimuli were shown to the subjects in the experiments. Since
channel responses are different for familiar and unfamiliar classes, the channels discriminating the classes were investigated. To
do so, three distances and four similarity measures were employed to assess the most distant channel pairs between familiar and
unfamiliar classes for a 1-s time duration; 0.6 s from the stimulus to 1.6 s in a channel selection process. It is experimentally
observed that this time interval is maintaining the greatest distance between two categories. The electroencephalographic
signals were classified using the determined channels and time interval to measure accuracy. The best classification accuracy
was 81.30% and was obtained with the Pearson correlation as channel selection method. The most discriminative channel
pairs were selected from prefrontal regions.
Keywords Familiar/unfamiliar face stimuli ·Electroencephalogram ·Event-related potentials ·Channel selection ·
Classification
1 Introduction
Computer-based analysis of electroencephalographic signals
(EEGs) is important to pre-diagnose some brain diseases. It
may also be possible to design a brain–computer interface
(BCI) for individuals who cannot use body parts.
Regions related to face perception in skull can be local-
ized from EEG, and we can provide some information about
previously viewed faces by classifying EEGs responsive to
familiar and unfamiliar faces. Thus, classification process
provides information about some disorders originating in the
brain and about the criminal in a judicial investigation.
There are many studies about analysis of EEG of famil-
iar/unfamiliar face experiment. In most of these studies,
BAbdurrahman Özbeyaz
aozbeyaz@adiyaman.edu.tr
Sami Arıca
arica@cu.edu.tr
1Department of Electrical and Electronics Engineering, Faculty
of Engineering, University of Adiyaman, Adiyaman, Turkey
2Department of Electrical and Electronics Engineering,
Faculty of Engineering, Çukurova University, Adana, Turkey
evoked potentials elicited by representation of familiar and
unfamiliar face stimuli have been investigated, in general,
employing statistical analysis. For instance, Tanaka et al.
[1] studied event-related potentials (ERPs) corresponding
to preexisting acquired face familiarity. Sun et al. [2]used
a directed-lying task to explore the differentiation between
identification and classification processes involved in the
recognition of familiar faces by inspecting ERPs. Among
the studies of familiar/unfamiliar face experiments the use of
signal processing and machine learning approaches is quite
limited. In [3], the authors have classified EEG recorded in
the familiar and unfamiliar face recognition experiment by
applying machine learning. In the experiment subjects should
press a button to mark that the displayed face is familiar. A
custom wavelet is generated and used to obtain wavelet trans-
form of the EEG data. Features are extracted in the transform
domain. The classification and performances were com-
pared with custom and common wavelets in the paper. They
obtained 69.7% classification accuracy. The authors in [4]
have investigated familiar and unfamiliar face categorization
by utilizing channel selection and machine learning tech-
niques. In the study, most discriminative channels have been
decided by computing pairwise distance between evoked
123
Signal, Image and Video Processing
potentials of the channels of the two classes with different
type of distances. Next the decided channels have employed
for training and testing of a linear support vector machine
classifier. The highest success rate reported is 72.67% with
the Kullback Leibler distance measure.
Some brain machine interfacing (BCI) studies adopted
familiar/unfamiliar faces in the experimental paradigm in
order to strengthen ERPs generated by the stimuli of cat-
egories [5,6]. However, these BCI studies do not categorize
the familiar/unfamiliar face stimuli. The experience obtained
with the familiar/unfamiliar face classification may also be
beneficial for the BCI researches.
This study is an improved version of the work in [4], and
the employed data are left from [4]. In this study, EEG respon-
sive to familiar and unfamiliar face stimuli was recorded from
ten healthy persons in different sessions. Moreover, EEG
was analyzed to understand the decomposition time period
and channel locations for familiar and unfamiliar classes on
the skull. In our study, EEG was, respectively, processed
in preprocessing, feature extraction, channel selection and
classification stages. For extracting features from the data,
piecewise constant approximation of EEG was used. Seven
different distance measures were employed to select suit-
able channels. We computed distances between EEGs of
two-channel pairs corresponding to two categories (famil-
iar/unfamiliar). In this manner, all distances were computed
for all channel pairs. And the most distant channel pairs were
employed to classify a single trial EEG. Since we did not
employed ERPs for channel selection and classification, and
utilized different experimental setup, the methods used in the
study were completely different than the existing methods in
literature [3,4]. There are two main differences between [4]
and this work: EEG signal is employed directly instead of
evoked potentials to compute the discriminative channels
and, two classifiers are combined with error correcting codes.
Primarily, the choice of discriminative channels makes this
study alternating from [4]. In [4], ERPs of the channels for
each category are computed by averaging trials from the
training data. And distance is computed between the ERPs
of channel pairs of the two categories, and a distance matrix
(where row hold channels for one category, and column holds
channels for one class) is generated. The two-channel pairs
satisfying the biggest distance measures in the rank are cho-
sen. On the other hand, in this study, the distances between
single trials of the categories from the paired channels are
computed, and a distance matrix is realized. Next, the cumu-
lative sum of distance matrices belonging to each trial is
obtained from the training data. The two-channel pairs sat-
isfying the biggest cumulative distance measures in the rank
are chosen (it is described in more details in Sect. 3). This is
a completely different practice.
In the next section, collection of EEG data and the
experimental procedure are described. In Sect. 3, signal mod-
eling, channel selection and classification approaches are
described. Findings and discussions are given in Sect. 4.The
study is concluded in Sect. 5.
2 Materials
2.1 EEG recording system and data
The commercial Emotiv EPOC wireless EEG headset with 14
channels was utilized to capture the associated brain activity
during the period of the experiments. The device headset con-
tains AF3, AF4, F3, F4, F7, F8, FC5, FC6, P7, P8, T7, T8, O1
and O2 electrodes of standard and extended 10/20 electrode
placement system. It should be noted that central electrodes
are missing in the headset. EEG was acquired from these
electrode locations, and electrode type was Ag/AgCl. The
P3 and P4 channels were employed as CMS (Common Mode
Sense) and DRL (Driven Right Leg) references, respectively.
EEG was recorded at a 128 Hz sampling frequency rate with
the help of a 16-bit analog to digital converter. As a default,
a 50 Hz notch filter was applied to the EEG recorded from
device. After the EEG data were acquired for 14 subjects,
all data were examined visually to identify any corrupted
records.
Next, all data were primarily preprocessed. In this process,
a first-order 0.16 Hz digital high-pass Butterworth IIR filter
was applied in a forward direction to eliminate the baseline
of the data. And subsequently, to remove the background
signal (noise) from the EEG, a linear phase low-pass digital
FIR filter was used. The order of the filter was chosen as
10 and the bandwidth was determined to be 32 Hz, which
is assumed to contain most of the energy of the EEG. The
“butter” and “fir1” functions of MATLABsoftware were
implemented to design the high-pass and low-pass filters,
respectively. Lastly, each trial of the data was normalized by
subtracting its mean and scaling with its standard deviation.
The data can be accessed from the address given in [7].
2.2 Subjects
Tenhealthy volunteers, one female and nine males, from 22 to
52 years old, with an average age of 34 and standard deviation
of 9.87, took part in this study. All participants have normal
or corrected-to-normal vision and no history of neurological
or psychiatric disorders, and they are all right-handed. Four
wore glasses, and all subjects were non-medicated during
the experiment. All participants signed an informed consent.
The experiments were conducted in the computer department
of Adiyaman University, Turkey, and the experiments were
approved by a local ethics committee of the health sciences
institute of the university.
123
Signal, Image and Video Processing
2.3 Stimuli
One-hundred-and-twenty familiar and unfamiliar face images
were used in the study. The familiar group contained 61 well-
known personalities (eight female) which were collected
from the public domain of the Internet, and 59 unfamiliar
faces (five females) were obtained from a modeling agency.
The faces presented a frontal view and were alike in gen-
eral visual appearance and attractiveness. All images were
subjectively equalized for luminance and contrast. In some
cases, manual corrections were applied to remove strands
of hair, paraphernalia or extensive makeup. Although some
changes were made in low-level visual features, the appear-
ances of the facial images were natural and bare. The images
were 24-bit color and rescaled to 600×600 pixels. All pro-
cesses on the images were realized via Adobe Photoshop 8
(Adobe Systems Inc.). The images were displayed with cus-
tom software developed in Borland Delphi 7 (stimuli display
program) on a liquid crystal display (LCD) monitor.
2.4 Procedure
Subjects were seated in a comfortable chair in front of an
LCD monitor (1024 ×768 pixel resolution) at a distance
about 100 cm from a computer screen controlled by a laptop
computer. Before starting the experiment, participants read
and signed the informed consent form. Following electrode
placement, the purpose of the experiment was explained to
the subjects and they were requested to look at the screen
center carefully, and to hold their eyes and body still.
The stimulus presentation time interval has been preferred
to be between 800 and 1000 ms, and the inter-stimulus inter-
val has been about 1000 ms. The applied procedure in our
experiment was determined based on practices from exper-
iments in previous studies. Different from past studies, a
marker button was not used in our experiment since it may
create a variance in the EEG unrelated to the face recognition
task, which could then influence classification.
Participants had to decide whether a displayed face
stimulus was familiar or unfamiliar at the instant of the pre-
sentation. Since the participant may not recognize the faces
as presumed, all participants declared a face as familiar by
marking printed faces on a form at the end of the EEG record-
ing. These were used in determining the actual familiar and
unfamiliar classes for each subject, and signals correspond-
ing to the face stimuli images were extracted afterward. The
display order of the images on the form (fixed for all subjects)
and during the experiment was different, and we expect that
the familiar and familiar faces recognized during the exper-
iment and after the experiment coincide.
The time course of the experiment was organized as fol-
lows: At the beginning of the experiment a black and white
checker image is shown for 5 s, and then an “Experiment
start” warning is displayed for 60 s on the screen to indi-
cate start of the session. Following the stimulus is shown for
800 ms, and then a gray screen appeared for 1200 ms. Next, a
gray fixation is displayed for 2500 ms to enable the subjects to
prepare and to concentrate for the coming trial [4]. One epoch
lasted 4.5 s, and the experiment lasted about 10 min and 5 s
(2.5+0.8+1.2=4.5s,4.5×120 +65 =605 s =10 min
5 s). The inter-stimulus interval was constant and the same
for all subjects. During the experiments, familiar and unfa-
miliar face stimuli images were randomly exhibited on an
80-inch monitor. Moreover, because epochs belonging to
stimuli should be extracted in turn, the display order of the
face images was also saved for each subject in the course of
the experiment.
3 Methodology
The recorded EEG was analyzed in time and spatial domain
on the scalp, and the most discriminative channels and
time intervals were determined. All algorithms were imple-
mented by using MATLABrelease R2010a on a personal
computer (equipped with an IntelCoreTM i7-3520M
CPU@2.90 GHz and a 12 GB RAM).
The EEG was approximated with piecewise constant mod-
eling and served as a feature extraction method, and SVM
was utilized for classification. In the study, the data were
divided into three groups: The first group comprised 50% of
the data and was reserved for channel selection, the second
comprised 37.5% of the data and was used for training classi-
fiers, and the remaining 12.5% was kept for validation. From
the first group, the channel selection algorithm determined
the most distant channel pairs. The features from the second
and third groups of data were computed from the most dis-
criminative channel pairs learned from the first data group.
Two classifiers were employed for classification. One classi-
fier was used to classify features computed from the channels
of a familiar category, and the other was engaged to classify
data obtained from the channels of the unfamiliar category
of the most distant pairs. The decision was familiar if the two
classifiers concluded a familiar class; otherwise, the decision
was unfamiliar. A block diagram summarizing the algorithm
is given in Fig. 1.
3.1 Feature extraction
The time varying EEG was approximated by using a piece-
wise constant model (PCM) for each trial. The purpose of the
piecewise modeling is to track the changes in the dynamics
or the properties of a signal through time. In each piece, we
assume that the signal is stationary. The signal is decomposed
to equal parts (windows). The metric describing each part is
the mean value (sample mean). The time length of each trial
123
Signal, Image and Video Processing
Fig. 1 Block diagram of the study
and its sample sizes are 1000 ms and 128 samples, respec-
tively. The length of each segment is chosen for eight samples
which accounts for a 1/16 s (62.5 ms) time interval, and there-
fore, each trial consists of 16 windows. The sequence of the
approximation coefficients (mean values) acts as a feature
vector for one trial.
3.2 Channel selection
Distance and similarity measures were employed to decide
discriminative channels. We describe them in the following.
Kullback Leibler distance The Kullback Leibler (KL) dis-
tance, in other words, the symmetric Kullback Leibler
divergence is a measure of the distance between two proba-
bility distributions [8].
Traditionally, to compute KL distance, the value of the
signal at a fixed time is considered random variable. This ran-
dom variable is quantized to eight levels, and histogram of
quantization levels over ensemble is computed and normal-
ized with the ensemble size to obtain the probability density.
Alternatively, a time index of a trial is considered as a random
variable. The magnitude of a trial consisting of a sequence
of numbers is quantized to eight levels. Then the sequence is
scaled with its sum to the quantized time samples. The value
of the resultant sequence for a time index becomes its proba-
bility, and the series is accepted as a probability density. The
KL distance is computed with these probability densities of
the two categories and is named as alternative KL distance
(AKLD).
Mutual information Mutual information (MI) is a measure of
how much one random variable contains information about
another. It may also be interpreted an amount of depen-
dence between two groups (variables) [9]. For computing the
mutual information, the two random variables corresponding
to the two categories are quantized using a common quan-
tizer with eight quantization levels.
Fisher score The Fisher distance or score (FS) is the nor-
malized level of discrimination between (the means of) two
classes. Therefore, the top ranked features present higher
discrimination. The criterion is slightly modified to suit the
application. Because the means may be close to each other
or be small compared to variances, the mean absolute value
of the signals belonging to classes is employed instead, that
is:
FS =|ma1−ma2|
s2
1+s2
2
(1)
In this equation, ma1and ma2are means of the absolute
values and s1and s2are sample variances of categories 1 and
2, respectively. With this modification the difference of the
variances between classes is considered discriminative.
Pearson correlation coefficient and R squared The most
commonly used measure of association is Pearson’s product-
moment correlation coefficient, often denoted as r. This value
is a measure of linear trend between two variables. The value
of r will always lie between −1 and 1. A correlation coef-
ficient rof zero indicates that there is no linear association
between the variables, r=1 means there is a perfect pos-
itive linear relationship between the variables and r=−1
shows a perfect negative linear relationship between the vari-
ables. In the study, r=−1 shows that the two variables were
completely distant and r=1 indicates that the groups are
identical (closest). The r-squared measure is square of the
correlation coefficient and was also employed in this study.
T-Te st The ttest is a method to test a hypothesis. The
null hypothesis is that two independent random variables x
and ywhich are normally distributed have equal mean with
unknown variances. The pvalue is a compatibility measure
between the null hypothesis and data, and in case it is less
than a particular level (degree of freedom) the null hypothe-
sis is rejected. In this study, the degree of freedom is chosen
as 0.5.
The channel selection algorithm is abstracted in the fol-
lowing.
3.3 Channel selection algorithm
The channel selection algorithm is as follows:
1. Compute pairwise distance between unfamiliar and
familiar EEGs of channels for each time instants. The
distance matrix D(a,b,t)=distance between familiar
123
Signal, Image and Video Processing
EEG of channel aand unfamiliar EEG of channel bat
time t.
2. For each time instants decide threshold. Choose third
highest distance as threshold (THR). Next perform
threshold operation. B(a,b,t)=D(a,b,t)>THR
3. Sum B(a,b,t)along time tto get votes for each channel
pairs. C(a,b)=∞
tB(a,b,t)
4. (Similar to step two thresholds are decided). Choose third
highest vote count as threshold. Then achieve threshold-
ing operation. E(a,b)=C(a,b)>THR
Consequently, the positions of logical one in matrix E(a,b)
indicate the best channel pairs for discrimination of familiar
and unfamiliar classes.
3.4 Classification
A linear support vector machine (SVM) was employed for
classification. The SVM used was a hard margin classifier.
The SVM classifier provided by the Bioinformatics Toolbox
of MATLAB software was utilized. Half of the data were
used to select four channels pairs. The remaining half were
employed to compute the performance of the classifier. The
features were computed from these selected four channels
pairs. The features computed from the channels of the famil-
iar class of the pairs were concatenated to form a feature
vector (feature set 1), and the features obtained from the
channels of the unfamiliar class of the pairs were combined
to shape another feature vector (feature set 2). Initially, the
EEG data were separated into two sets at this stage. Seventy-
five percent of the data was designated for training, and the
reaming 25% of the data was assigned for testing. An SVM
machine (classifier 1) learned the relationship between the
data and labels of classes from feature set 1 of the training
data and the other SVM classifier (classifier 2) learned the
relationship between data and labels of classes from feature
set 2 of the training data. Classifiers 1 and 2 categorized
the test data of feature sets 1 and 2, respectively. If two
classifiers decided an unfamiliar class then the label was
assigned familiar, and otherwise unfamiliar. Thus, classifica-
tion success was calculated. The same process was repeated
twenty times. The training and testing sets were specified ran-
domly throughout. The average of the classification results
was accepted as classification success.
4 Findings and discussion
The 1 s time duration from the 0.6 s stimulus onset was used
for channel selection and classification. Because we were
experimentally observed the time interval [0.6, 1.6] s after
stimulus achieved highest classification performance. The
histogram matrix (counts) of selected channel pairs for all
Fig. 2 The histogram matrix belonging to the selected channel pairs
between familiar and unfamiliar classes
Fig. 3 The histogram matrix of favored regions
subjects, obtained using three distances and four similarity
measures individually, is given in Fig. 2.
The columns stand for the features of familiar class read
from the channels, while the rows stand for the features of
unfamiliar class acquired from the channels in Fig. 2.Inthe
figure, the matrix size is 14 ×14 and each element is the
count of how many times a pair (row–column or familiar–
unfamiliar) is selected. This calculation was done by taking
into account all channel selection processes. Because the
selected pairs were dispersed among the subjects and meth-
ods, the distribution of the chosen pairs was examined in
terms of the scalp regions (frontal, temporal, parietal and
occipital). A histogram of the selected region-pairs is shown
in Fig. 3. The regional pairs, frontal–temporal, frontal–
occipital and frontal–frontal, were selected 28, 21 and 65
times, respectively, among 140 possible channel pairs. The
SVM classifier employed has been linear. It is also possible
to employ other kernels such as radial basis and polyno-
mial functions. The algorithm has also been run for these
two kernels. However, the results have not been better than
the results of the linear SVM. This should be because these
kernels do not fit correctly the separation boundary of the
classes.
A list of classification accuracies is given in Table 1.In
the table, the eleventh and twelfth rows are mean value and
standard deviation of the results. Rows thirteen and fourteen
are F1-Score and Matthews correlation coefficient (MCC)
123
Signal, Image and Video Processing
Table 1 Classification accuracies (%)
Sbj. KLD AKLD MI FS rr
2t-test
166.03 77.41 75.69 77.59 73.79 71.55 66.72
288.79 83.10 85.52 91.21 84.83 89.48 83.79
389.66 77.59 81.21 87.07 90.00 86.90 68.79
482.33 76.00 64.17 78.50 79.17 81.17 55.50
582.93 77.24 71.90 70.00 81.72 87.41 70.69
669.48 74.66 69.83 58.45 82.41 80.69 68.97
762.93 56.21 67.24 58.97 83.97 81.03 62.93
868.10 61.72 68.28 63.28 82.76 79.48 65.00
963.28 62.24 58.79 59.14 82.59 78.79 60.34
10 65.86 65.52 68.45 60.17 71.72 75.34 66.38
Avg. 73.94 71.17 71.11 70.44 81.30 81.19 66.91
SD 10.18 8.48 7.50 11.76 5.03 5.25 7.07
F10.67 0.64 0.64 0.64 0.78 0.78 0.58
MCC 0.53 0.45 0.45 0.43 0.64 0.64 0.37
values, respectively. The F1 score is a measure of a test’s
accuracy in statistical analysis of a binary classification.
MCC is a measure of the quality of binary classifications [10].
From the eleventh row of the table, it is deduced that the high-
est average classification accuracy of the subjects is 81.30%
with a standard deviation of 5.03 achieved by using Pear-
son correlation as channel selection method. Moreover, the
second average accuracy of the subjects is 81.19% with a
standard deviation of 5.25 achieved by using R-square as
channel selection method. The ranges of classification accu-
racies of the subjects obtained with the Pearson Correlation
and R-squared methods are 71.72–90.00 and 71.55–89.48%,
respectively. In addition to the accuracy, specificity (Qn),
which is the measure of number of detected unfamiliar face
stimuli among unfamiliar face stimuli, and sensitivity (Qd),
which is the measure of number of detected familiar face
stimuli among familiar face stimuli are reported in Table
2. The specificity of the classification obtained with Pear-
son correlation varies between 88.33 and 94.64%, while
the sensitivity varies between 45.77 and 85.36%. Moreover,
average of specificity and sensitivity were 92.44 and 69.92%,
respectively. Furthermore, the specificity of the classifica-
tion obtained with R-squared method varies between 86.67
and 98.33%, while the sensitivity varies between 52.69 and
86.33%. Besides, average of specificity and sensitivity were
92.08 and 70.04%, respectively.
The average classifications accuracies of Pearson correla-
tion and R-squared methods are observed to be nearly close
to each other, and these two methods give the best outcomes
in the ranking. The F1-Score and MCC approves the con-
clusion. And the results indicated that the classifier performs
better in classifying unfamiliar faces than in categorizing
familiar portraits.
It has been already expressed above that the Pearson
correlation coefficient and R-squared similarity measures
provided the best classification accuracy and outperformed
the other measures. It is also observed that the specificity
associated with subject-3 is the highest among all subjects
with R-squared value and is the second largest value if
Pearson correlation coefficient is employed. The divergent
channel pairs are O1–P7 and O1–O1 when the Pearson corre-
lation coefficient method is employed, and also the opposing
channel pairs are AF3–AF3 and AF3–T7 when R-squared
method is employed for this subject. The spatial locations of
these channels are displayed in Fig. 4.
Table 2 The specificity (Qn) and sensitivity (Qd) of the classification (%)
Sbj. KL AKL MI FS rr
2t-test
QnQdQnQdQnQdQnQdQnQdQnQdQnQd
190.67 39.64 93.67 60.00 89.00 61.43 91.33 62.86 90.00 56.43 86.67 55.36 86.33 45.71
296.43 81.67 94.64 72.33 95.36 76.33 94.64 88.00 93.21 77.00 96.79 82.67 96.07 72.33
398.00 80.71 96.67 57.14 96.00 65.36 92.67 81.07 94.33 85.36 98.33 74.64 83.00 53.57
493.33 71.33 90.67 61.33 84.67 43.67 85.67 71.33 88.33 70.00 92.00 70.33 77.67 33.33
592.14 74.33 91.07 64.33 84.64 60.00 85.00 56.00 88.57 75.33 88.57 86.33 86.79 55.67
689.29 51.00 87.86 62.33 83.21 57.33 80.00 38.33 94.64 71.00 95.36 67.00 88.21 51.00
781.79 45.33 68.93 44.33 87.50 48.33 73.93 45.00 94.64 74.00 90.36 72.33 84.29 43.00
883.93 53.33 79.29 45.33 88.93 49.00 78.93 48.67 93.93 72.33 91.79 68.00 85.71 45.67
979.29 48.33 84.64 41.33 82.14 37.00 80.71 39.00 93.93 72.00 87.14 71.00 77.86 44.00
10 89.38 36.92 89.06 36.54 90.63 41.15 85.31 29.23 92.81 45.77 93.75 52.69 86.88 41.15
Avg. 89.42 58.26 87.65 54.50 88.21 53.96 84.82 55.95 92.44 69.92 92.08 70.04 85.28 48.54
SD 5.82 16.20 7.87 11.15 4.54 11.61 6.28 18.51 2.37 10.54 3.79 9.91 5.00 10.01
123
Signal, Image and Video Processing
Fig. 4 Two most distant channel pairs for subject 3 with Pearson corre-
lation coefficient and R-squared channel selection methods are shown
in (a)and(b), respectively
5 Conclusions
The EEG data of a familiar/unfamiliar face recognition exper-
iment were analyzed in the time and spatial domain. EEG data
were obtained from ten subjects (nine male and one female)
who participated in the experiment. The experiment utilized
120 stimuli images in all sessions. The experiment setup did
not incorporated button press to mark face familiarity. Par-
ticipants had to decide upon the stimuli type (familiar or
unfamiliar) as soon as they saw them. Thus, any undesired
event artifacts were not allowed to occur on the EEG. The
recognized categories of the face stimuli were then declared
by subjects after the experiment and labels for EEG were
assigned accordingly.
In this study, time and spatial analyzes of the recorded
EEG were done according to seven different channel selec-
tion methods. While some of these measured similarities,
the others measured the distances between channel pairs
for familiar and unfamiliar classes. For each time instance,
the distance between two classes were measured over trials.
Accordingly, time intervals in which two classes differenti-
ated were also examined. The integrated distance over time
implied the most distant channel pairs. The 1 s time duration
from the 0.6 s stimulus onset was chosen to classify EEG
because, we have experimentally seen that the time inter-
val [0.6, 1.6] s after stimulus achieved highest classification
performance. It seems that 0.6 s delay is required for pro-
cessing and recognizing displayed image in the brain. This
interval does not overlap with the time course of latter (fol-
lowing) stimulus, and therefore, inter-stimulus interference
is avoided. It is also known that the P600 wave type is an
observed potential in familiarity experiments [11,12]. The
ideal time duration is in parallel to this finding. Continuing,
two-channel pairs were selected. One channel of a pair corre-
sponded to familiar and the other to unfamiliar. The channel
pairs were extracted from a histogram matrix obtained by
combining results of 20 iterations for each of the subjects.
According to the obtained results, the channels were mostly
selected from the frontal area of the scalp. This result matches
with the conclusion in [4]. It is well known that frontal and
temporal regions in the brain are active in familiar face selec-
tivity [13–15]. This experience supports the outcome. Using
the decided channel pairs, the features computed from the
channels of familiar and unfamiliar were classified using two
separate classifiers. The decision for the category was done
by looking at labels of these two classifiers. If the labels
of the two classifiers were familiar, category was familiar;
otherwise, it was unfamiliar. The best average classification
accuracy was observed 81.30% with Pearson correlation as
a channel selection method. This success is 8.63% more
than the rate obtained by the very similar study [4](see
Table 3). The R-squared similarity measure provided the
second topmost performance at 81.19%, which was very
close to the accuracy computed with the Pearson correla-
tion coefficient. For both similarity quantities, the rate of true
classification within an unfamiliar class was about 92% and
the rate of true classification within a familiar category was
around 70%. The statistical difference among channel selec-
tion methods was investigated by a one-way-ANOVA test.
The test returned a pval ue =0.0016 (<0.05) which indi-
cated that there was significant difference between channel
selection approaches. Statistically, there might be a promi-
Table 3 Comparisons of the
proposed method with other
relative studies in literature
Study Channel election/features Classifier Succ. (%)
[3] The wrapper method/four approximation
coefficients of custom wavelet transform
Fisher LDA 69.70
[4] The two-channel pairs satisfying the
biggest distance between the ERPs of
two categories in the rank/piecewise
constant approximation coefficients
Linear SVM 72.67
This work The two-channel pairs satisfying the
biggest cumulative distance between
trials of categories in the rank are
chosen/piecewise constant
approximation coefficients
Union of two linear
SVM classifiers
81.30
123
Signal, Image and Video Processing
nent channel selection technique (together with feature) in
the set of channel selection methods used. Additionally, sta-
tistical differences among channel selection methods were
investigated by computing a paired ttest. To a significance
level of 0.05, the accuracies obtained with Pearson correla-
tion and R-squared methods were significantly different from
the results obtained with the other five distances.
The absence of central electrodes and the limited number
of electrodes in the EEG device may be a drawback/limitation
of the study. However, the scalp regions which are shown [16]
to be responsible from the face recognition and cognition are
mostly covered.
The studies similar to the proposed method in the literature
are summarized in Table 3for the reader to make comparisons
conveniently. Consequently, it appears from the findings of
the proposed approach that the study is a candidate for use in
face recognition and medical applications such as diagnosis
of some brain diseases and in criminal identification, as long
as the measurements are repeated and proved with a larger
number of electrodes.
Acknowledgements This study was supported by The National Sci-
entific Research Council of Turkey (TUBITAK)—Grant No. 2211-C
and Adıyaman University’s Scientific Research Fund - Project Number:
TIPBAP/2012-0008. The experiments were approved by a local ethics
committee of the health sciences Institute of Adıyaman University.
Ethics Committee Decision Number is 2012/07-2.3. All participants
signed an informed consent.
References
1. Tanaka, J.W., Curran, T., Porterfield, A.L., Collins, D.: Activation
of preexisting and acquired face representations: the N250 event-
related potential as an index of face familiarity. J. Cogn. Neurosci.
18, 1488–1497 (2006)
2. Sun, D., Chan, C.C.H., Lee, T.M.C.: Identification and classifica-
tion of facial familiarity in directed lying: an ERP study. PLoS
ONE 7, e31250 (2012)
3. Çelik, U., Arıca, S.: Classification of evoked potentials of familiar
and unfamiliar face stimuli using multi-resolution approximation
based on excitatory post-synaptic potential waveform. Comput.
Electr. Eng. 39, 1571–1584 (2013)
4. Özbeyaz, A., Arıca, S.: Classification of EEG signals of familiar
and unfamiliar face stimuli exploiting most discriminative chan-
nels. Turk. J. Electr. Eng. Comput. Sci. 25, 3342–3354 (2017).
https://doi.org/10.3906/elk-1608-130
5. Kaufmann, T., Schulz, S.M., Grünzinger, C., Kübler, A.: Flashing
characters with famous faces improves ERP-based brain–computer
interface performance. J. Neural Eng. 8, 56016 (2011)
6. Jin, J., Allison, B.Z., Zhang, Y., Wang, X., Cichocki, A.: An ERP-
based BCI using an oddball paradigm with different faces and
reduced errors in critical functions. Int. J. Neural Syst. 24, 1450027
(2014)
7. Özbeyaz, A., Sami, A.: Biomedical data. https://sites.google.com/
view/biomedicaldata/home
8. Zhu, X.: Information Theory. In: Advanced Natural Language Pro-
cessing Lecture Notes. University of Wisconsin, Madison, p 3
(2010)
9. Cover,T.M., Thomas, J.A.: Elements of Information Theory.Wiley,
Hoboken (2006)
10. Matthews, B.W.: Comparison of the predicted and observed
secondary structure of T4 phage lysozyme. Biochim. Biophys.
Acta Protein Struct. 405, 442–451 (1975). https://doi.org/10.1016/
0005-2795(75)90109-9
11. Schweinberger, S.R., Pickering, E.C., Jentzsch, I., Burton, A.M.,
Kaufmann, J.M.: Event-related brain potential evidence for a
response of inferior temporal cortex to familiar face repetitions.
Cogn. Brain Res. 14, 398–409 (2002). https://doi.org/10.1016/
S0926-6410(02)00142-8
12. Gouvea, A.C., Phillips, C., Kazanina, N., Poeppel, D.: The linguis-
tic processes underlying the P600. Lang. Cogn. Processes 25(2),
149–188 (2010)
13. Rossion, B., Schiltz, C., Robaye, L., Pirenne, D., Crommelinck, M.:
How does the brain discriminate familiar and unfamiliar faces? A
PET study of face categorical perception. J. Cogn. Neurosci. 13,
1019–1034 (2001)
14. Haxby, J.V., Hoffman, E.A., Gobbini, M.I.: The distributed human
neural system for face perception. Trends Cogn. Sci. 4, 223–233
(2000)
15. Leveroni, C.L., Seidenberg, M., Mayer, A.R., Mead, L.A., Binder,
J.R., Rao, S.M.: Neural systems underlying the recognition of
familiar and newly learned faces. J. Neurosci. 20, 878–886 (2000)
16. Guyton Arthur, C., Jhon, E.: Cerebral cortex, intellectual func-
tions of the brain, learning and memory. In: William, S., Rebecca,
G. (eds.) Textbook of Medical Physiology, pp. 714–727. Elsevier,
Philadelphia (2006)
123
A preview of this full-text is provided by Springer Nature.
Content available from Signal Image and Video Processing
This content is subject to copyright. Terms and conditions apply.
