Content uploaded by Nur Keleşoğlu
Author content
All content in this area was uploaded by Nur Keleşoğlu on Feb 13, 2024
Content may be subject to copyright.
HierAct: a Hierarchical Model for Human Activity
Recognition in Game-Like Educational Applications
Nur Keles¸o˘
glu
Institute of Theoretical and Applied
Informatics
Polish Academy of Sciences
Gliwice, Poland
0000-0002-0306-7281
Katarzyna Filus
Institute of Theoretical and Applied
Informatics
Polish Academy of Sciences
Gliwice, Poland
0000-0003-1303-9230
Joanna Domanska
Institute of Theoretical and Applied
Informatics
Polish Academy of Sciences
Gliwice, Poland
0000-0002-1935-8358
Abstract—Constantly evolving landscape of modern education
makes the integration of technology crucial to enable more
interactive and immersive learning experiences. One of the key
problems in this domain is Human Activity Recognition (HAR),
which uses standard smartphone sensors to understand and rec-
ognize user movements. While HAR exhibits high potential in this
area, its persistent challenge is to differentiating between similar
activities, whose occurrences can often lead to misclassification
and reduced quality of applications. To tackle this problem, we
employ specialized classifiers and use them in a hierarchical
manner instead of using monolith multi-class models. We propose
HierAct - a Hierarchical model that solves the multi-class human
activity recognition problem. We also release a new HAR dataset
- EduAct - (https://github.com/iitis/HierAct-Dataset) focused on
activities that could be used to create game-like educational
applications.
Our results show that the performance of the proposed
model is substantially better than other state-of-the-art machine
learning models. Our work shows the potential of hierarchical
classifiers in HAR applications for education, by offering ed-
ucators and students a more accurate, reliable, and engaging
interactive experience.
Index Terms—Human Activity Recognition, Hierarchical
Model, Gaming Industry, Digital Education, Game-Like Appli-
cations
I. INTRODUCTION
The advancement of modern education is tied to technolog-
ical innovations because they offer new means to effectively
engage students by providing new interactive tools. Human
Activity Recognition (HAR) is one of the technologies that
can be used to create such tools. HAR uses data from different
sensors to detect and classify user movements. HAR is an
established solution in many different areas: from healthcare
systems [1]–[3], through identifying and classifying human
gestures [4] and movements [5] to self-driving cars [6], [7].
It is also used for analyzing sports performance [8] and the
gaming industry. Due to this wide range of applications, it
would be also good to utilize it more in the area of education
The research leading to these results received partial funding from the
National Centre for Research and Development under the project ”ExploreAR:
an innovative interaction game technology that uses new communication
mechanisms with the user and the environment, combining digital and real
components of augmented reality games” (POIR.01.01.01-00-0400/22)
to enable the creation of game-like interactive applications that
could engage and motivate students, especially those young
people that grow up in the digital world. For example, it has
been observed that integrating a game-like application into the
Spanish classroom increases the performance of students in
Spanish verb conjugations [9]. Additionally, game-based learn-
ing has positively developed students’ computational thinking
abilities [10].
Using HAR can provide us with a more immersive experi-
ence by classifying various user actions. HAR can describe the
movements of the user during the application use and provide
them with feedback. This technology plays a significant role
in the gaming industry. Especially in Virtual Reality (VR) and
Augmented Reality (AR) games, by reflecting the movements
of real-life players in their virtual characters, it enables more
engaging interactions.
It is also possible to incorporate this technology into digital
education and learning, as various learning and development
resources are fundamental to distance education [11]. Thus,
students can be further encouraged to learn with games, AR,
and VR technologies and to increase the time allocated for
education. Although the potential of this technology for edu-
cation seems vital, there are not many works in the literature
tackling this problem.
HAR can be categorized into three main approaches: the
vision-based (camera), the sensor-based, and the radio-based
[12]. Vision-based HAR processes images and videos [13],
radio-based HAR uses Bluetooth, WiFi, and ZigBee [14],
and Sensor-based HAR uses accelerometers, gyroscopes, and
magnetometer sensor data [15]. In this study, we focus on
sensor-based human activity recognition by the use of a
smartphone since they are low-cost and practical. Also, they
are available on a great majority of modern smartphones,
which are a target platform for budget-friendly educational
applications due to the accessibility to students.
Motivated by:
1) the fact that HAR exhibits enormous potential for educa-
tion and has proven success in many different domains,
2) incredible abilities of machine learning algorithms to
979-8-3503-2445-7/23/$31.00 ©2023 IEEE
recognize different activity types,
3) the challenging nature of the multi-classification prob-
lem in domains with highly similar activities with similar
sensor patterns,
4) the necessity to provide budget-friendly and accessible
solutions for the creation of interactive applications in
the area of education,
we propose a cost-efficient, smartphone-based solution for
Human Activity Recognition in education. Although existing
studies are successful in the field of different kinds of human
activity recognition, there are no satisfactory studies address-
ing the above problems for the game industry and education.
As traditional, monolith models do not differentiate well be-
tween closely-related activities, we propose to use specialized
classifiers in a hierarchical manner to obtain better accuracy
and geralization. We call our hierarchical model HierAct. We
compare HierAct with existing state-of-the-art models. We
also release a new HAR dataset - EduAct - whose unique
part is that it is focused on game-like educational applications
and the game industry in general. The dataset includes data
from a gyroscope and accelerometer for 8 different activities
by using a smartphone via an IoS application we developed.
This makes the contribution of this study as follows: (1) A
new multi-class human activity recognition dataset (EduAct),
(2) A novel hierarchical model for multi-class human activity
recognition (HierAct).
The rest of this paper is organized as follows: Section II
describes the relationship of this work to the state of the
art. In Section III, we present our novel HierAct model for
human activity recognition. In Section IV, we implement the
HierAct model with EduAct data. In Section V, we present
the experimental results. Finally, we discuss our results and
describe our future work in Section VI.
II. RELATED WO RKS
In this section, we contrast our novel proposed HierAct
model against the state of the art. Although there are many
studies conducted in HAR in many fields, the studies using
HAR for game-like applications in the field of education are
very limited. However, by using cost-efficient HAR in game-
like applications students can increase their interest in lessons,
and the potential to increase their success by increasing
interactivity is very high.
Now, we contrast our work with the previous sensor-
based human activity recognition works. There are studies
that use only accelerometer data [16]–[19] or accelerometer
and gyroscope data together [20]–[23] for human activity
recognition. The paper [24] describes a deep learning method
using smartphone inertial sensors (accelerometer and gyro-
scope both) for recognizing basic activities. The average
activity recognition rate of the method is 89.61% (accuracy
95.85%) for 12 different physical activities (including stand-
ing, sitting, lying down, walking, going up and down stairs).
In contrast, we proposed a HierAct model for human activity
recognition which is suitable for education and gaming. Also,
it is using by smartphones can be used to create budget-
friendly apps.
We compare our work against works that use hi-
erarchical models. One of the methods used for HAR is
the hidden Markov statistical probability model. Hierarchic
adaptations of this model are widely seen in the HAR field
such as the Hierarchical hidden Markov model (HMM) [25],
hierarchical continuous HHMM [26], two-stage continuous
HMM [27]. The paper [28] proposed a two-layer hierarchical
hidden Markov probabilistic model for activity recognition
from wireless sensor network data such as float sensors.
However, it is difficult to determine the relationship between
these models and similar activities. The paper [29] proposed
a Hierarchical Hidden Markov Model to recognize human
activity from accelerometer and gyroscope data. In the study,
two public datasets were used and results were compared with
Random forest and Long short-term memory models. The
accuracy of the proposed model in the study is around 84% and
94% for two datasets separately and is better than LSTM and
RF. The paper [30] proposed the hierarchical approach for real-
time sensor-based activity recognition of 26 classes. In contrast
to previous works, our proposed model uses the activity-
focused strategy to establish the hierarchy. Moreover, we use
specialized models based on small neural network models to
build our ensemble. However, it is difficult to determine the
relationship between these models and similar activities.
We describe previous game-like application studies for
education and their effects. Reference [31] mentions that
the VR and AR game-based interface applications used by
civil engineering students can motivate the learning process
of students. Reference [32] has shown that educational com-
puter games are effective in achieving success and providing
physical motivation by using the Digital Game-Based Learning
(DGBL) approach in high school computer science classes. In
another study using DGBL, two digital games with Chinese
language and art content were developed for 4th-grade students
[33]. Data were collected from wearable electroencephalo-
graphic sensors while the participants were playing games, and
when the students’ behaviors were examined in 5 patterns, it
was observed that the students in the DGBL groups showed
better performance than other students in learning the Chinese
language art. Reference [34], an experiment was conducted
using versatile gamification applications for the online English
course of 5th-grade students and it was compared with students
who learned with the traditional learning method. In the ex-
periment, it was observed that the test results of students using
gamification applications were higher than those of students
who learned with traditional presentation methods, and the
positive impact of gamification applications on education was
observed. In contrast to the above studies, the proposed model
HierAct can be used in the field of digital education in all kinds
of games involving human activity, it also supports remote
(online) education such as waving in the virtual classroom,
and it can be integrated with VR and AR.
We compare our hierarchical model with studies based
on HAR and focus on an area of the game industry.
For instance, reference [35] utilizes video frames from body-
played games (dance or sports games) and employs the Ad-
aboost machine learning model to perform real-time action
recognition. In another study, a 3D game environment based on
multiple depth sensor (Kinect) data that detect the user’s body
movements is presented and used to evaluate the student’s
performance and to encourage the learners [36]. In contrast,
the IMU sensor is available on all smartphones, its analysis is
faster than the one of 3D visual data, and also it is cheaper
than other sensors or cameras. That is why we decided to use
this kind of data in our study.
In summary, the HierAct model can be easily used on
smartphones and it is cost-effective, therefore it is expected to
be more widely used compared to other studies. Also, along
with the fact that not many studies are focused on HAR in
education while it exhibits high potential for this domain.
III. HIERARCHICAL MOD EL (H IE RACT) METHODOLOGY
In this section, we first describe the methodology of the
HierAct model for human activity recognition in Section III-A
then we explain the steps of the HierAct model in Sec-
tion III-B.
A. Methodology of the HierAct Model
The proposed Hierarchical model is a method used to
solve a multi-class human activity recognition problem, which
consists of at least a level (layer) and contains at least a binary
classifier (BC) in each level. The block diagram of a HierAct
model is shown in Figure 1.
Each horizontal BC array in the figure denoted a level l.
The Lis the total number of levels in the HierAct model
and l∈(1, ..., L). There can be a number of mBCs in each
level lexcept the first level (l= 1). The mis the number of
binary classifiers at each level and m∈(1, ..., M)(for the
initial value, when l= 1,m= 1). The number of levels and
the number of binary classifiers at each level can be changed
according to the data and the number of activities. As the
number of activities increases, more binary classification is
needed.
The probability of true prediction of BCl,m at level land
mth classifier is Pl,m, and the probability of true prediction
at each level is Plwhere l∈(1, ..., L). Then, we calculate the
probability of true prediction for each level Plin (1).
Pl=
M
X
m=1
Pl,m
nl,m
nl
(1)
where the nis the number of samples in the dataset, nlis
the total number of samples at level land nl,m is the number
of samples used in BCl,m.
We assume the number of samples nl,m is equal for each
binary classifier. Then, the probability of true prediction for
each level Plis shown in (2).
Pl=1
M
M
X
m=1
Pl,m (2)
Fig. 1. Block Diagram of the HierAct Model
Then, the probability of true prediction at the end of each
level ˜
Plis shown in (3) under the condition that the number
of samples is equal at each level l.
˜
Pl=
L
Y
l=1
Pl(3)
Thus, the probability of true prediction at the last level ˜
PL
is equal to the average probability of true prediction of the
HierAct model for all classes. It also equals the true prediction
performance of the model.
B. Implementation of the HierAct Model
The implementation of the proposed HierAct model consists
of three steps. The first step is to determine the number of lev-
els in the HierAct model and the number of binary classifiers
at each level. The second step is to build Binary Classifiers
and train them. The third step is to build the HierAct model
architecture and apply it to the activity recognition problem.
1) Determine the Number of Levels and Binary Classifiers
in the Hierarchical Model: First, we perform the Gradient
Boosting Classifier to train the model with the training dataset.
Then, we test the trained model with the training dataset and
generate the confusion matrix for the results. The values in the
confusion matrix provide information about the similarity of
activities to each other. So, we find vulnerable classes based
on the most similar class (based on max confusion). Then, we
place the activity class that is least similar to any other class in
the first level. Then, we place the second least similar class in
the second level, and we continue this process until all classes
are assigned to levels. If there are two classes with the same
similarity ratio, we place them on the same level l. Because
if the similarity values are close to each other, we do not give
priority to any of them. We evaluate both at the same level l.
The pseudocode explains how we determine the binary
classifiers and place them in the hierarchical model, as shown
in the Algorithm 1 given below.
Algorithm 1 Determining Binary Classifier
1: function MONOLITHCLASSIFICATION(dataset)
2: Perform monolith multi-class classification;
3: Return the resulting model;
4: function CONFUSIONMATRI X(model, dataset)
5: Use the model to predict the dataset and calculate the
confusion matrix;
6: Return the confusion matrix;
7: function SIMILARITYMATRIX(confusionM atrix)
8: Extract the similarity matrix from the confusion ma-
trix;
9: Return the similarity matrix;
10: function DETERMINECLA SSI FIE RS(similarityMatrix)
11: H←hierarchical model ▷empty at the beginning
12: S←a set of classes that represents classifiers
13: l←the number of levels
14: while S /∈ ∅ do:
15: if l= 1 then
16: leastSimiClass ←
ArgMin(similarityM atrix);
▷if His empty, we can add only one binary
classifier to the first level of H
17: Add the leastSimiClass to Hat level l= 1;
18: Delete the leastSimiClass from S;
19: else l≥2
20: leastSimiClasses ←
ArgMin(similarityM atrix);
▷if His not empty, we can add more than one
binary classifier to the level lof H
21: Add the leastSimiClasses to Hat level l;
22: Delete the leastSimiClasses from S;
23: Increase number of level lby 1;
24: Return H
25: // Main part of the code
26: model ←MONOLITHCLASSIFICATION(dataset)
27: confusionMatrix ←CONFUSIONMA-
TR IX(model, dataset)
28: similarityMatrix ←SIMILARITYMA-
TR IX(conf usionM atrix)
29: H←DETERMINECLA SSI FIE RS(similarityMatrix)
30: // Now, the Hconsists of a number of Llevels, with one
or more classifiers at each level l.
2) Implementation of Binary Classifiers: In this step, we
train a separate binary classifier for each activity class at each
level. First, the balanced dataset is set for each binary classifier.
Then, the Multi-Layer Perceptron (MLP) model is optimized
and recorded according to the training results of the binary
classifier.
3) Hierarchical Model : In this step, the architecture of the
HierAct model is created by using binary classifiers.
IV. HUMAN ACTIVITY RECOGNITION BY HIERACT
MOD EL FO R TH E EDUACT DATASET
In this section, we perform human activity recognition using
the proposed HierAct model with a real multi-class activity
dataset. In this study, one of the significant things is that we
gathered the data from a smartphone by game-like application
we developed. We call the HAR dataset as a EduAct. In
section IV-A and section IV-B, we explain the data and the
preprocessing stage for data respectively. In section IV-C,
we implement the HierAct model according to the EduAct
dataset.
A. EduAct Dataset
The data was gathered from the iPhone 14 Plus smart-
phone through a game-like application we developed. The
data contains time information, 3dimensions acceleration, and
gyroscope data in x, y, and z dimensions. Table I shows the
description of each column of the data gathered from the
smartphone. In the table, from 1st row to 7th row are the
features of the data set, and the last row contains the target
information.
TABLE I
DATA DESCRIPTION
Column Description
timestamp timestamp determined for all data
accelerometerxaccelerometer information in x direction
accelerometeryaccelerometer information in y direction
accelerometerzaccelerometer information in z direction
gyroscopexgyroscope information in x direction
gyroscopeygyroscope information in y direction
gyroscopezgyroscope information in z direction
activity type labeled activity information
Data were gathered for eight different activities. These
activities follow: (1) Neutral, (2) Shake shake, (3) Cranking
Front, (4) Cranking Back, (5) Waving Front, (6) Waving
Back, (7) Mixing Left and (8) Mixing right. These activities
are the behaviors that the player does with the smartphone
while playing the educational game in our scenario. These
activities were have been chosen due to their high suitability
for educational application. E.g. the shake shake activity can
be used in Interactive Geology Applications to to simulate
seismic activity or in mathematical applications connected to
randomness and probability. Waving, on the other hand, could
be used to mark the Virtual Classroom Participation. E.g. in
remote learning environments, students could use waving to
signal that they want to answer a question offering them a
digital equivalent to raising their hands. Cranking could find
applications in Interactive History Classes to progress through
a timeline and in parallel making them active participants of
the story.
To create a heterogeneous dataset, data were collected from
nine different people separately. Each person was told to
gather data in 6 different manners: (1) right hand + sitting,
(2) left hand + sitting, (3) right hand + standing, (4) left
hand + standing, (5) right hand + walking, (6) left hand
+ walking. The reason was to create a training dataset that
would make machine learning model’s training easier by
providing similar activity-of-interest conditions while making
sure that ’background’ activities are different. Thus, we have
the opportunity to develop a robust model that will recognize
the same activities performed by different people and adapt
to changes in the sensor data, and the generalization of our
model increase.
Data collected from seven different people for eight activi-
ties were used as the training dataset and there were approx-
imately 122000 samples. Data collected from two different
people (not used for training) were used to test the model.
The test dataset has almost 627000 samples. In addition, we
release the processed data via our GitHub repository at
https://github.com/iitis/HierAct-Dataset.
B. Data Preprocessing Stage
The time window has an effect on human activity
recognition [37]. In order to choose the most optimal time
window, first, we read the data with 12,25,50,100, and 200
samples respectively which are responses to 0.25,0.5,1,2,
and 4seconds. (Each new sample is measured approximately
every 0.02 seconds.) Then, we compared the train and test
performances of binary classifiers for each time window size.
Finally, we determined the optimal time window size as 100
samples. Then for each time window, we calculated the mean
and standard deviation of the accelerometer and gyroscope in
x,y, and z directions and we used those calculations as input for
the model. In total, we have 12 features (accelerometerxmean,
accelerometerymean , accelerometerzmean , gyroscopexmean ,
gyroscopeymean , gyroscopezmean , accelerometerxstd ,
accelerometerystd , accelerometerzstd , gyroscopexstd ,
gyroscopeystd , gyroscopeystd ) and a target (activity type) in
the datasets and there are 121104 samples in the train set and
626279 samples in the test set after the preprocessing stage.
In addition, we assigned a number for each activity in the
activity class. Targets are numbers between 1and 8.
C. Implementation of the HierAct Model for EduAct Dataset
We perform the Gradient Boosting model to classify the
8-classes human activity recognition dataset before building
the HierAct model. The parameters used in the model are as
follows: number of estimators = 100, learning rate = 0.8, max
depth = 4. While the training accuracy of the model is 0.9998,
the test accuracy of the model is 0.5672. The test accuracy
result is much lower than train accuracy due to the data
used in the test is numerous and different from the data used
for training (unused data in the training process). Thus, it is
difficult to build a robust and highly accurate model. We define
the problem here as a multi-class activity recognition with low
performance and poor generalization ability. We recommend
the HierAct model to solve this problem.
To build the HierAct model for multi-class activity recog-
nition, we perform the following steps sequentially that we
pointed out in Section III-B.
1) Step 1: Determine the Number of Levels and Binary
Classifiers in the Hierarchical Model: First, we trained a
gradient-boosting classifier model to classify activities. Then,
we consider the confusion matrix of the test result of the
gradient-boosting model to determine the binary classifiers and
the levels in the model. According to the results, the similarity
of each class to other classes and the similarity ratio on a scale
of 1 are as follows:
•shake - mixing right →0.0018
•neutral - mixing right →0.0020
•waving back - cranking back →0.0034
•waving front - mixing left →0.0047
•mixing left - mixing right →0.0238
•mixing right - cranking back →0.0247
•cranking front - cranking back →0.0761
•cranking back - cranking front →0.0761
2) Step 2: Implementation of Binary Classifiers: According
to the list in Step 1, classifiers for shake, neutral, waving,
waving front, mixing, mixing left, and cranking front have
been created. When creating binary classifiers, each class was
labeled as 1, while all other classes (another activity) were
labeled as 0. For example, for the Shake binary classifier, all
samples belonging to shake activity in the dataset were labeled
as 1, and all other samples were labeled as 0. In this way, we
can distinguish Shake activity from all other activities.
Moreover, all binary classifiers and the HierAct model used
in this study are created using Python 3.10.9. We use Keras
and Scikit-learn libraries to implement the binary classifiers
and the rest of the methods in Python. We perform the
MLP machine learning model to create binary classifiers. In
addition, the optimizer parameter is “Adam”, the Activation
functions are “Tanh” (except the last layer) and “Sigmoid”
(only the last layer), and the number of neurons at the layers
respectively 12,18,24,36,24,12,6, and 1for all binary
classifiers. The number of epochs of Shake BC, Neutal BC,
Waving BC, Waving Front BC, Mixing BC, Mixing Left BC,
and Cranking Front BC is 40,30,20,20,40,30, and 30
respectively. Also, the batch size of each BC is 128, except
the Shake BC which is equal to 64.
3) Step 3: Hierarchical Model: According to the list in Step
1, the shake activity has the lowest similarity with the mixing
right activity with a similarity ratio of 0.0018. Therefore, we
place the shake activity classifier in the first level. Next, we
place the neutral activity in the second level. In the third
level, we place the waving activity, in the fourth level, the
waving front activity and mixing activity, in the fifth level,
the mixing left binary classifier, and cranking front activity.
Since the similarity ratio between cranking front and cranking
back activities is equal, we place the cranking front in the last
level and create the model.
In our HierAct model, we determine the number of level
Lis 5 and there is 1,1,1,2,2 number of binary classifiers at
each level lrespectively according to the steps 1 & 2. There
are seven binary classifiers to build a Hiearchical model for
the EduAct dataset. Now, we determined the HierAct model
architecture which is shown in Figure 2.
Fig. 2. Block Diagram of the HierAct Model for EduAct Dataset
In this figure, the shake classifier at the first level classifies
whether the activity is “Shake” or other activities. Then, if
it is decided that the activity is not a shake, the second level
classifies whether the activity is “Neural” or not. If it is decided
that the activity is not neutral, at the third level it is checked
whether the activity belongs to any “Waving” class. In the
third level, if it is decided that the activity is waving, in the
fourth level it is classified which waving type (waving front
or back) the activity belongs to, but if in the third level it is
decided that the activity is not waving, in the fourth level it is
classified whether the activity belongs to the “Mixing” class or
not. Then, at the fourth level, if it is determined that the activity
is “Mixing”, at the fifth level, it is classified which mixing type
(mixing left or right) the activity belongs to, but if at the fourth
level, it is determined that the activity is not mixing, this time
the activity belongs to the remaining “Cranking” class. For
this reason, at the fifth level, it is defined that the activity
belongs to the cranking front or back class with the cranking
front binary classifier.
Thus, every incoming input enters the HierAct model,
passes through binary classifiers, and determines which class
it belongs to. For example, the first input data enters the
shake classifier. If the output of the shake classifier is 1, it
is determined to be a Shake activity. If the output data is 0,
the input data enters the neutral classifier, and so on. After
determining which class the incoming input is in, the same
process is repeated for the next input data.
The logic in this model is that classes are eliminated by
determining which class the activity does not belong to, and
the process continues until the class to which it belongs is
determined. In this way, we confirm that the activity does not
belong to the previous class and we determine the class to
which it belongs. Thus, we present a more robust model than
other activity recognition models.
V. RE SULTS
In this section, we present our experimental results in
three ways. First, we present the performance of the binary
classifiers that we used in the HierAct model. Second, we
present the performance of the HierAct model against other
machine learning models. Third, we present training and
execution times for each model.
A. Performance of the Binary Classifiers in the HierAct model
Table II presents the recall, precision, specificity, accuracy,
and F1 score of each binary classifier that is used in the
HierAct model. The results are presented for the training and
testing datasets separately. The first column shows the name of
the binary classifiers. For example, the Shake binary classifier
is trained to classify shake activity. The columns between
the second and sixth show the recall, precision, specificity,
accuracy, and F1 score of the binary classifiers results for
the training dataset, and the column between the seventh and
eleventh shows the recall, precision, specificity, accuracy, and
F1 score of the binary classifiers results for the test dataset.
Precision also called positive predictive value (PPV) is the
ratio between true positives and all positive predictions. Recall
also called the sensitivity and true positive rate (TPR) is the
ratio between true positives and actual positives. Specificity
also known as true negative rate (TNR) is the ratio between
true negatives and actual negatives. Accuracy is the ratio
between the true predictions and the total number of predic-
tions. The accuracy is usually shown as a percentage and it
is one of the metrics for evaluating the performance of the
classification model. Thus, we used this metric to compare
our model with other models. In addition, the probability of
the true predictions at the last layer ˜
PLthat we mention in
TABLE II
THE PE RFOR MAN CE OF EA CH BINA RY CLAS SIFI ER
Name of the Train Dataset Test Dataset
Classifiers Recall Precision Specificity Accuracy F1 Recall Precision Specificity Accuracy F1
1.Shake 0.9937 0.9996 0.9996 0.9967 0.9967 0.9812 0.9846 0.9847 0.9829 0.9829
2.Neutral 0.9906 0.9979 0.9979 0.9942 0.9942 0.9635 0.9930 0.9931 0.9783 0.9780
3.Waving 0.9983 0.9999 0.9999 0.9991 0.9991 0.9641 0.9791 0.9794 0.9717 0.9715
4.Waving Front 0.9695 0.9840 0.9842 0.9768 0.9767 0.9483 0.9884 0.9889 0.9686 0.9679
5.Mixing 0.9282 0.9717 0.9724 0.9503 0.9492 0.8270 0.6496 0.5549 0.6908 0.7276
6.Mixing Left 0.8277 0.9377 0.9450 0.8863 0.8930 0.8842 0.7742 0.7442 0.8132 0.8256
7.Cranking Front 0.8671 0.9208 0.9329 0.9017 0.8931 0.4405 0.6331 0.7447 0.5926 0.5195
Section III is equal to the accuracy of the model. The F1
score is the harmonic mean of the Recall and Precision. It is
used to see the effect of both Precision and Recall value.
The recall, precision, specificity, accuracy, and f1 score of
the binary classifiers shake, neutral, waving, and waving front
is above 0.94 for both the training and test datasets in the t
able II. Besides, the performance metrics of shake, neutral,
and waving binary classifiers for the training dataset are
almost 1. This result also shows us that these activities (shake,
neutral, and waving binary) can be easily classified from other
activities. All performance metrics of mixing binary classifiers
are above 0.92 for the training dataset. The performance
decreases slightly for the test dataset. Especially when the
specificity is considered, the (1-0.5549) ratio shows that the
mixing classifier perceives non-mixing activities as mixing. All
performances for the Train dataset in mixing left and cranking
front classifiers are above 0.82. Performances decrease slightly
for the test dataset.
Fig. 3. Graph of F1 Score of Binary Classifiers
The F1 scores of both the train and test data sets are shown
in detail in Figure 3. Dark blue bars show the F1 score of the
training dataset and light blue bars represent the F1 score of
the test dataset. As can be seen from the figure, the F1 score of
the training data set is higher than that of the test data set for all
binary classifiers. In addition, the F1 score of Shake, Neutral,
Waving, and Waving Front is higher than 0.96 for both data
sets. We can say that the ability to recognize positive cases
(activities in this study) is higher for these binary classifiers
than for the rest of the binary classifiers.
In summary, the performances of shake, neutral, waving,
and even waving front classifiers provide high performance
for both data sets. For this reason, these activities are easier to
classify than other activities. However, it is possible to say that
mixing and cranking activities interfere with other activities.
B. Performance Comparison of HierAct Model and Other
Machine Learning Models
Now, we present the performance of the proposed HierAct
model in both the train and test datasets with respect to the
accuracy in Table III. Then, we compare the performance of
our HierAct model against other machine learning models in
Table IV.
TABLE III
PERFORMANCE OF HIERACT M ODEL AT E ACH LEV EL
Number of
Level
Train Accuracy
Until end of
The Level
Test Accuracy
Until end of
The Level
1 0.9979 0.9813
2 0.9934 0.9700
3 0.9929 0.9501
4 0.9567 0.8491
5 0.8665 0.7359
Table III presents the accuracy performance of the HierAct
model at the end of each level of the model. The first column
shows the levels of the model. The second and third columns
display the accuracy of the model on the training and test
datasets, respectively, after a specific level. For example, the
first row of the table shows the train and test accuracy at the
end of the first level of the model (when l= 1) or the second
row shows train and test accuracy at the end of the second
level of the model (when l= 2). The last row shows the train
and test accuracy when the level l= 5 which also represents
the accuracy of the model.
Train accuracy is 0.9979,0.9934,0.9929,0.9567, and
0.8665 from the first level to the last level. The test accuracy
is 0.9813,0.9700,0.9501,0.8491,0.7359 respectively. It is
observed that accuracy decreases in both data sets towards the
last level of the HierAct model. The main reason for this is that
the performance of binary classifiers at the last levels is lower
than the binary classifiers at the upper levels. Another reason
is as the number of levels increases, the number of activities
that classify also increases proportionally. For example, in the
1st level, classification is performed for two classes: shake
activity and other activities. In the 2nd level, classification is
performed for three classes: shake activity, neutral activity, and
other activities. In the 3rd level, classification is performed for
four classes: shake activity, neutral activity, waving activity,
and other activities. In the 4th level, classification is performed
for six classes: shake activity, neutral activity, waving back,
waving front, mixing, and other activities. In the last level
(5th level), classification is conducted for all activities in the
dataset.
According to the HierAct model, the accuracy decreases
from the first level to the last level because the accuracy at each
level is multiplied to obtain the overall accuracy according to
Eq.3.
For this reason, a decrease in accuracy as the number of
levels increases is normal. Additionally, the accuracy value
at the last level (shown in bold in the table) represents
the accuracy of the HierAct model for the test dataset. The
significant thing is for this value to have better performance
when compared to other models.
Table IV presents the performance of the HierAct model
and other machine learning models in terms of accuracy for
both train and test datasets. The other models we used for
comparison are as follows: (1) MLP Classifier, (2) Random
Forest Classifier, (3) Decision Tree, (4) LightGBM Classifier,
(5) CatBoost Classifier, (6) Support Vector Machine (SVM),
(7) XGBoost, (8) Gradient Boosting Classifier, (9) Naive
Bayes. A grid search was performed for each machine learning
model to determine the best test accuracy of the models. Thus,
the most optimal parameters were determined. Then, each
machine learning model was trained with the training set for
window sizes 25,50, and 100 and tested with the test set.
However, the time window size was chosen as 100 samples
for these models to compare with the HiearAct model.
The decision tree model achieves the highest training per-
formance with a score of 1.0, but its test accuracy is only
0.6027. Also, the training accuracy of the MLP Classifier,
Random Forest, LightGBM, CatBoost Classifier, Support Vec-
tor Machine, XGBoost, and Gradient Boosting Classifier is
higher than 0.84; however, test accuracy drops significantly.
Naive Bayes has the lowest train and test accuracy and is the
least successful classification model for this data set among all
models. This indicates that these models may not generalize
well and are not robust for the EduAct dataset.
TABLE IV
PERFORMANCES COMPARISON AGAINST TO OTHER ML MODELS
Models Train
Accuracy
Test
Accuracy
HierAct
Model 0.8665 0.7359
MLP Classifier 0.9965 0.6171
Random Forest 0.9118 0.6027
Decision Tree 1.0 0.6027
LightGBM
Classifier 0.9880 0.5994
CatBoost
Classifier 0.8787 0.5947
Support Vector
Machine 0.8441 0.5919
XGBoost 0.9968 0.5907
Gradient Boosting
Classifier 0.9998 0.5672
Naive Bayes 0.6111 0.3953
The performance of the HierAct model on the training
dataset which is 0.8665 is not the best performance when
compared with the others. However, it achieves the highest
performance with 0.7359 on the test dataset in terms of
accuracy. This indicates that the HierAct model is more robust
and has a higher generalization ability compared to the other
models.
C. Comparison of Training and Execution Time
Now, we compare the training and execution (test) time of
the HierAct model against the other machine learning model
for the different time windows. The comparison results are
presented in Table V.
The first column of the table shows the names of the
models. Training time and execution time of each model are
shown from the second to fourth column and from the fifth to
seventh column according to the number of samples for the
window size. The table provides training and testing durations
in seconds. Additionally, the durations in the table are the time
spent for training and testing all samples in the dataset (not
for a single sample).
It is observed that as the number of samples for the time
window increases, the processing time increases both in the
training and test datasets for the MLP Classifier, Random
Forest, Decision Tree, LightGBM Classifier, Support Vector
Machine, XGBoost, and Gradient Boosting Classifier machine
learning models. On the other hand, training time increases and
execution time decreases while the time window size is from
25 to 100. We say that training and execution times for the
Naive Bayes model are almost constant. It is only observed
that the training time of the HierAct model decreases as the
number of samples in the time window increases.
The total training time of the HierAct model is the sum
of the training time of binary classifiers that are used in the
model. Additionally, the training times of Shake BC, Neutal
BC, Waving BC, Waving Front BC, Mixing BC, Mixing
Left BC, and Cranking Front BC are as follows: 21.62,
10.76,12.65,7.25,21.26,9.83,10.70 seconds, respectively
for window time of 100 samples.
The training time of the HierAct model is 100.64,94.71,
94.07 respect to the time window size 25,50, and 100. The
training time of the model is longer than the training time
of the Random Forest, Decision Tree, LightGBM Classifier,
CatBoost Classifier, XGBoost, and Naive Bayes machine
learning models.
The execution time of the HierAct model is 5.13,5.30, and
5.36 respectively. The execution time of the HierAct model is
longer than other machine learning models except for the gra-
dient boosting classifier and support vector machine. Although
this process time is for almost 627000 samples, execution time
is 8.54 x10−6second per sample. This execution time for a
sample is acceptable since each new sample enters the model
every 0.02 seconds.
VI. CONCLUSION
Traditional machine learning methods are used for multi-
class human activity recognition. However, the true positive
rate of predictions for each class is not satisfied for every
class. However, when looking at binary classification, it is
seen that the performance of each binary classification is better
than multiple classification. Our motivation is to construct a
hierarchical model for multiple classes of human activity using
binary classifications.
We proposed a novel Hierarchical model -HierAct- con-
sisting of multiple levels and binary classifiers that solve a
multi-class human activity recognition problem. Each binary
classifier is trained exclusively to classify one activity. Our
binary classifiers use an MLP machine-learning model to
classify target activity. Then, we build the architecture of the
hierarchical model consisting of BCs.
When we compared the performance of other machine
learning models with the performance of the HierAct model,
our model performed significantly better for the test dataset.
This indicates the robustness and high generalization capability
of our model. We believe that this development will make a
valuable contribution to the gaming and educational sector.
In addition, the proposed model has the potential of hierar-
chical classifiers in educational HAR applications, providing
educators and students with a reliable and engaging interactive
experience.
In our future work, we plan to develop a reinforcement
learning system that will control the output of each binary
classifier and confirm it with a certain probability value.
Thus, binary classifiers will use the previously defined activity
information to become more adaptive, gain specific features,
TABLE V
COMPARISON OF TRAINING AND EXECUTION TIME
OF T HE HIER ACT MO DEL A ND MAC HINE LEARNING MO DEL S
ACC ORDI NG TO T HE DIFF ERE NT TIME WINDOWS
Models Training Time (s) Execution Time (s)
Number of Samples for Time Window
25 50 100 25 50 100
HierAct
Model 100.64 94.71 94.07 5.13 5.30 5.36
MLP
Classifier 271.90 362.83 439.96 1.41 1.44 1.58
Random
Forest 20.60 21.82 22.38 1.61 1.58 1.74
Decision Tree 6.20 6.75 7.04 0.08 0.09 0.1
LightGBM
Classifier 2.39 2.42 2.50 4.25 4.38 4.51
CatBoost
Classifier 2.44 2.56 2.61 0.21 0.20 0.19
Support
Vector
Machine
101.07 113.68 137.88 3258 3669 3578
XGBoost 39.24 39.62 39.65 1.23 1.30 1.26
Gradient
Boosting
Classifier
3591 3746 3674 8.29 8.63 8.71
Naive Bayes 0.03 0.03 0.03 0.54 0.54 0.53
and enhance their performances, which will generally improve
the model’s overall performance. Thus, we can present more
advanced HAR solutions for game-like applications that can
be used in digital education.
REFERENCES
[1] F. Serpush, M. B. Menhaj, B. Masoumi, B. Karasfi et al., “Wearable
sensor-based human activity recognition in the smart healthcare system,”
Computational intelligence and neuroscience, vol. 2022, 2022.
[2] A. Subasi, M. Radhwan, R. Kurdi, and K. Khateeb, “IoT based mobile
healthcare system for human activity recognition,” in 2018 15th learning
and technology conference (L&T). IEEE, 2018, pp. 29–34.
[3] R. Liu, A. A. Ramli, H. Zhang, E. Henricson, and X. Liu, “An overview
of human activity recognition using wearable sensors: Healthcare and
artificial intelligence,” in International Conference on Internet of Things.
Springer, 2021, pp. 1–14.
[4] M. G. Komang, M. N. Surya, and A. N. Ratna, “Human activity
recognition using skeleton data and support vector machine,” in Journal
of Physics: Conference Series, vol. 1192. IOP Publishing, 2019, p.
012044.
[5] F. Moya Rueda, R. Grzeszick, G. A. Fink, S. Feldhorst, and
M. Ten Hompel, “Convolutional neural networks for human activity
recognition using body-worn sensors,” in Informatics, vol. 5, no. 2.
MDPI, 2018, p. 26.
[6] M. Tammvee and G. Anbarjafari, “Human activity recognition-based
path planning for autonomous vehicles,” Signal, image and video pro-
cessing, vol. 15, no. 4, pp. 809–816, 2021.
[7] C. Crispim-Junior, R. Guesdon, C. Jallais, F. Laroche, S. S.-L. Corvec,
and L. T. Rodet, “AutoExp: A multidisciplinary, multi-sensor frame-
work to evaluate human activities in self-driving cars,” arXiv preprint
arXiv:2306.03115, 2023.
[8] N. A. Rahmad, M. A. As’Ari, N. F. Ghazali, N. Shahar, and N. A. J.
Sufri, “A survey of video based action recognition in sports,” Indonesian
Journal of Electrical Engineering and Computer Science, vol. 11, no. 3,
pp. 987–993, 2018.
[9] D. A. Casta˜
neda and M.-H. Cho, “Use of a game-like application on
a mobile device to improve accuracy in conjugating Spanish verbs,”
Computer Assisted Language Learning, vol. 29, no. 7, pp. 1195–1204,
2016.
[10] S. Liu, C. Peng, and G. Srivastava, “What influences computational
thinking? A theoretical and empirical study based on the influence of
learning engagement on computational thinking in higher education,”
Computer Applications in Engineering Education, 2023.
[11] C.-y. Li, Q. Zhao, N. Herencsar, and G. Srivastava, “The design of mo-
bile distance online education resource sharing from the perspective of
man-machine cooperation,” Mobile Networks and Applications, vol. 26,
no. 5, pp. 2141–2152, 2021.
[12] Z. Wang, Z. Yang, and T. Dong, “A review of wearable technologies for
elderly care that can accurately track indoor position, recognize physical
activities and monitor vital signs in real time,” Sensors, vol. 17, no. 2,
p. 341, 2017.
[13] D. R. Beddiar, B. Nini, M. Sabokrou, and A. Hadid, “Vision-based hu-
man activity recognition: a survey,” Multimedia Tools and Applications,
vol. 79, no. 41-42, pp. 30 509–30 555, 2020.
[14] Z. He and X. Bai, “A wearable wireless body area network for
human activity recognition,” in 2014 Sixth International Conference on
Ubiquitous and Future Networks (ICUFN). IEEE, 2014, pp. 115–119.
[15] V. Bijalwan, V. B. Semwal, and V. Gupta, “Wearable sensor-based
pattern mining for human activity recognition: Deep learning approach,”
Industrial Robot: the international journal of robotics research and
application, vol. 49, no. 1, pp. 21–33, 2022.
[16] A. Bayat, M. Pomplun, and D. A. Tran, “A study on human activ-
ity recognition using accelerometer data from smartphones,” Procedia
Computer Science, vol. 34, pp. 450–457, 2014.
[17] P. Casale, O. Pujol, and P. Radeva, “Human activity recognition from
accelerometer data using a wearable device,” in Pattern Recognition and
Image Analysis: 5th Iberian Conference, IbPRIA 2011, Las Palmas de
Gran Canaria, Spain, June 8-10, 2011. Proceedings 5. Springer, 2011,
pp. 289–296.
[18] A. Ignatov, “Real-time human activity recognition from accelerometer
data using Convolutional Neural Networks,” Applied Soft Computing,
vol. 62, pp. 915–922, 2018.
[19] Y. Chen and Y. Xue, “A deep learning approach to human activity
recognition based on single accelerometer,” in 2015 IEEE international
conference on systems, man, and cybernetics. IEEE, 2015, pp. 1488–
1492.
[20] S. Ha and S. Choi, “Convolutional neural networks for human activity
recognition using multiple accelerometer and gyroscope sensors,” in
2016 international joint conference on neural networks (IJCNN). IEEE,
2016, pp. 381–388.
[21] M. Webber and R. F. Rojas, “Human activity recognition with ac-
celerometer and gyroscope: A data fusion approach,” IEEE Sensors
Journal, vol. 21, no. 15, pp. 16 979–16 989, 2021.
[22] P. K. Shukla, A. Vijayvargiya, R. Kumar et al., “Human activity recog-
nition using accelerometer and gyroscope data from smartphones,” in
2020 International Conference on Emerging Trends in Communication,
Control and Computing (ICONC3). IEEE, 2020, pp. 1–6.
[23] A. Jain and V. Kanhangad, “Human activity classification in smartphones
using accelerometer and gyroscope sensors,” IEEE Sensors Journal,
vol. 18, no. 3, pp. 1169–1177, 2017.
[24] M. M. Hassan, M. Z. Uddin, A. Mohamed, and A. Almogren, “A robust
human activity recognition system using smartphone sensors and deep
learning,” Future Generation Computer Systems, vol. 81, pp. 307–313,
2018.
[25] P. Asghari, E. Soleimani, and E. Nazerfard, “Online human activity
recognition employing hierarchical hidden Markov models,” Journal of
Ambient Intelligence and Humanized Computing, vol. 11, pp. 1141–
1152, 2020.
[26] C. A. Ronao and S.-B. Cho, “Recognizing human activities from smart-
phone sensors using hierarchical continuous hidden Markov models,”
International Journal of Distributed Sensor Networks, vol. 13, no. 1, p.
1550147716683687, 2017.
[27] ——, “Human activity recognition using smartphone sensors with two-
stage continuous hidden Markov models,” in 2014 10th international
conference on natural computation (ICNC). IEEE, 2014, pp. 681–686.
[28] T. L. van Kasteren, G. Englebienne, and B. J. Kr¨
ose, “Hierarchical
activity recognition using automatically clustered actions,” in Ambi-
ent Intelligence: Second International Joint Conference on AmI 2011,
Amsterdam, The Netherlands, November 16-18, 2011. Proceedings 2.
Springer, 2011, pp. 82–91.
[29] S. Ashry, W. Gomaa, M. G. Abdu-Aguye, and N. El-borae, “Improved
IMU-based human activity recognition using hierarchical hmm dis-
similarity,” in Proceedings of the 17th International Conference on
Informatics in Control, Automation and Robotics, vol. 1, 2020, pp. 702–
709.
[30] L. Wang, T. Gu, X. Tao, and J. Lu, “A hierarchical approach to real-
time activity recognition in body sensor networks,” Pervasive and Mobile
Computing, vol. 8, no. 1, pp. 115–130, 2012.
[31] F. M. Dinis, A. S. Guimar˜
aes, B. R. Carvalho, and J. P. P. Martins, “Vir-
tual and augmented reality game-based applications to civil engineering
education,” in 2017 IEEE Global Engineering Education Conference
(EDUCON). IEEE, 2017, pp. 1683–1688.
[32] M. Papastergiou, “Digital game-based learning in high school computer
science education: Impact on educational effectiveness and student
motivation,” Computers & education, vol. 52, no. 1, pp. 1–12, 2009.
[33] Y.-T. Yu and M. Tsuei, “The effects of digital game-based learning
on children’s Chinese language learning, attention and self-efficacy,”
Interactive Learning Environments, pp. 1–20, 2022.
[34] A. C¸ INAR, Y. Eris¸en, and M. C¸ elik¨
oz, “A mixed-method research on
the effectiveness of using gamification elements in an online English
course,” International Journal of Educational Research Review, vol. 7,
no. 4, pp. 280–291, 2022.
[35] V. Bloom, D. Makris, and V. Argyriou, “G3D: A gaming action
dataset and real time action recognition evaluation framework,” in 2012
IEEE Computer Society Conference on Computer Vision and Pattern
Recognition Workshops. IEEE, 2012, pp. 7–12.
[36] A. Kitsikidis, K. Dimitropoulos, D. U ˘
gurca, C. Bayc¸ay, E. Yilmaz,
F. Tsalakanidou, S. Douka, and N. Grammalidis, “A game-like applica-
tion for dance learning using a natural human computer interface,” in
Universal Access in Human-Computer Interaction. Access to Learning,
Health and Well-Being: 9th International Conference, UAHCI 2015,
Held as Part of HCI International 2015, Los Angeles, CA, USA, August
2-7, 2015, Proceedings, Part III 9. Springer, 2015, pp. 472–482.
[37] O. Banos, J.-M. Galvez, M. Damas, H. Pomares, and I. Rojas, “Window
size impact in human activity recognition,” Sensors, vol. 14, no. 4, pp.
6474–6499, 2014.
