Content uploaded by Andrey Saltan
Author content
All content in this area was uploaded by Andrey Saltan on Mar 10, 2021
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=tmes20
International Journal of Mathematical Education in
Science and Technology
ISSN: 0020-739X (Print) 1464-5211 (Online) Journal homepage: http://www.tandfonline.com/loi/tmes20
Likert-scale questionnaires as an educational tool
in teaching discrete mathematics
O. A. Ivanov, V. V. Ivanova & A. A. Saltan
To cite this article: O. A. Ivanov, V. V. Ivanova & A. A. Saltan (2018): Likert-scale questionnaires
as an educational tool in teaching discrete mathematics, International Journal of Mathematical
Education in Science and Technology, DOI: 10.1080/0020739X.2017.1423121
To link to this article: https://doi.org/10.1080/0020739X.2017.1423121
Published online: 12 Jan 2018.
Submit your article to this journal
Article views: 28
View related articles
View Crossmark data
INTERNATIONAL JOURNAL OF MATHEMATICALEDUCATION IN SCIENCE AND TECHNOLOGY,
https://doi.org/./X..
CLASSROOM NOTE
Likert-scale questionnaires as an educational tool in teaching
discrete mathematics
O. A. Ivanova,V.V.Ivanova
bandA.A.Saltan
b
aDepartment of General Mathematics and Informatics, St. Petersburg State University, St. Petersburg, Russia;
bDepartment of Information Systems in Economics, St. Petersburg State University, St. Petersburg, Russia
ARTICLE HISTORY
Received October
KEYWORDS
Active learning; Likert-scale
questionnaires; discrete
mathematics; student
performance
ABSTRACT
In this paper, we report on the results of an experiment in teach-
ing discrete mathematics to students majoring in business informat-
ics. We supplemented our problem-based approach to teaching the
course with a set of Likert-scale surveys or questionnaires that helped
improve the students’performance. On the one hand, these surveys
gave us feedback and, on the other, encouraged the students to
reect on the subject-matter. The experiment was quite successful, as
the grades obtained by the students on the exam were signicantly
higher than usual. Here, we describe the structure of the surveys and
the method of evaluation of the experimental results.
1. Introduction
A course in discrete mathematics is in fact of an interdisciplinary nature. It develops the
analytical, logical, and algorithmic thinking necessary for successful study of many other
disciplines [1,2]. Although such a course is essential for students majoring in business
informatics, like other interdisciplinary courses, it appears to be quite challenging for
them [3].
Approaches to designing the structure of interdisciplinary courses and teaching them
in such a way as to both make them more attractive to students and improve their perfor-
mance have received much attention in research and practice over the last decade. Various
approaches assuming the involvement of students in the learning process with the pur-
pose to obtain long-lasting and meaningful learning are nowadays known as active learn-
ing. Active learning suggests students’ engagement in such high-order activities as analysis,
synthesis, and evaluation [4]. There is good evidence that the implementation of an active
learning approach improves students’ performance without noticeable loss in academic
knowledge (see, e.g. [5–7]). The types of active learning interventions considered vary
widely. As shown in [8], in teaching physics, the implementation of the technique called
‘interactive lecture demonstrations’ yields signicant improvements in learning outcomes.
On the other hand, there is evidence that the incorporation of a programming practicum
in a discrete mathematics course also promotes more active learning [9].
CONTACT O. A. Iv ano v o.a.ivanov@spbu.ru
© Informa UK Limited, trading as Taylor& Francis Group
2CLASSROOM NOTE
In our discrete mathematics course as formerly taught, we had incorporated elements
oftheactivelearningapproachfromtheverybeginning,andthelecturesweresupple-
mented with problem-solving sessions and a programming practicum. However, the stu-
dents’ results (in particular, their grades) were far from what one might desire.
The starting point of our experiment came from an observation made in [2]. The stu-
dents were asked to specify on a so-called Likert scale (ranging from strongly agree to
strongly disagree) their level of agreement or disagreement with the following statement:
‘The reason why it is hard to decipher a message encoded by the RSA method lies in the
factthatthecomplexityofndingtheinversetoagivenelementofZkdepends exponen-
tially on the number k’.
Sincethesuggestedreasonisinfactwrong,thecorrectreactiontothisstatementwould
be to check ‘strongly disagree’ on the Likert scale. Nevertheless, more than half of the stu-
dentsagreedwiththestatement.Suchareactionshowsclearlythatthestudentshadgrasped
neither the central idea of the RSA encryption algorithm (the diculty of factoring ‘very
large’ numbers that are products of two primes), nor the algorithm for nding the inverse
for a given element in the ring Zk. In fact, the complexity of nding a−1∈Zkdepends lin-
early even on a.
Thus, the reaction of the students to the given statement provides an indication of the
level of understanding (or rather misunderstanding) of the topic under study. The nov-
eltyofourproposedapproachwastosupplementthestandardapproachtolearning,that
is, one including home and in-class assignments as well as a mid-term colloquium, with
Likert-based surveys. These surveys were followed by a discussion of the main topics and
were intended to ‘close the knowledge loop’ in the sense of providing both students and
lecturer with information regarding the level of understanding of the topic. In this paper,
we describe the results of such an experiment conducted with freshmen majoring in busi-
ness informatics. Our main goal was to improve our students’ conceptual understanding of
discrete mathematics.
2. Description of the experiment
2.1. Design and scheduling of the experiment
The experiment aimed to investigate the extent to which the implementation of Likert-scale
surveys followed by detailed discussion can contribute to a better understanding of discrete
mathematics.
The experiment was carried out in the Spring of 2017 with 22 students majoring in Busi-
ness Informatics as participants. Four surveys were conducted: the rst three were oriented
towards theory, and the fourth to the implementation of algorithms using CAS Wolfram
Mathematica. In each of the four questionnaires provided, we asked the students to express
their attitude toward the statements on the following Likert scale
strongly agree agree hard to say disagree strongly disagree
The rst three questionnaires each consisted of 20 statements and the fourth question-
naire of 14. Each questionnaire contained an equal number of true and false statements (or
‘wrong-worded statements’, so-called).
INTERNATIONAL JOURNAL OF MATHEMATICAL EDUCATION IN SCIENCE AND TECHNOLOGY 3
Here is a detailed list of topics mentioned in the questionnaires:
Survey 1 (March): sequences and recurrence relations; sets and counting; combinatorics.
Survey 2 (April): modular arithmetic and some applications; the main algorithms relat-
ing to binary heaps and binary search trees; sorting algorithms.
Survey3(thebeginningofMay):themainalgorithmsrelatingtographsandtrees;greedy
algorithms relating to weighted graphs; relations on sets.
Survey4(theendofMay):thesyntaxandsemanticsofoperatorsandtheimplementation
of the main algorithms of discrete mathematics in CAS Mathematica.
A nal survey (similar to those described in [2]) was conducted just before the exami-
nation.Discussionoftheresultsofeachofthesurveystookplaceafewdayslater.
Surveys 1–3 lasted 15 minutes each, and their discussions 15–20 minutes each. Survey 4
required a little more time (about 30 minutes), and its discussion took also 15–20 minutes.
As will be shown below, the surveys covered a wider range of concepts, ideas, and
algorithms of the discrete mathematics course than did the regular quizzes. Also, since
the restricted time forced the students to answer the questions without hesitation, their
responses reected their understanding of the course. A student’s results on the surveys
did not aect his/her nal grade for the course, so he/she was motivated to provide answers
deemed by them to be correct in the light of their knowledge and understanding.
2.2. Statements examples
Here are the rst 10 statements of the rst questionnaire and the rst 5 statements of the
last questionnaire.
Statements of questionnaire 1 (Herestatements2,3,6,8,and9arethetrueones.)
(1) The magnitude of a Fibonacci number is a quadratic function of its index.
(2) Thesetofallarithmeticprogressionsisthesameasthesetofallsequencessatisfying
a second-order linear recurrence relation.
(3) The number of combinations is a special case of the number of permutations with
repetition allowed.
(4) The sum of the squares of the rst nnatural numbers is a quadratic polynomial in
n.
(5) If the height of a tower of Hanoi is doubled, then the number of moves required to
transfer the entire stack to another peg is approximately four times greater.
(6) If nite sets Aand Bhave the same number of elements, then the number of injec-
tions A→Bcoincides with the number of surjections A→B.
(7) In every set, the number of all 4-element subsets is greater then the number of all
3-element subsets.
(8) If a set has more than 3 elements, then the number of permutations on its elements
is greater than the number of its subsets.
(9) The number of subsets of a nite set Acoincides with the number of all maps from
Ato a 2-element set.
(10) If 13 out of 25 schoolmates like to read books and 17 adore computer games, then
exactly 5 of them like both to read books and play on the computer.
4CLASSROOM NOTE
Statements of questionnaire 4 (Here statements 1, 2, and 3 are true.)
(1) The procedure described below removes duplicates from a given list while preserv-
ing the order of the last occurrences of the items in the list.
(2) Though the complexities of the bubble-sort algorithm and the insertion sort algo-
rithm are both of the order O(n2), ‘bubble-sort’ is, on average, more time-consuming
than ‘insertion sort’.
(3) The procedure given below constructs the adjacency matrix of a graph from the
adjacencylistofthegraph:
(4) Execution of the following code yields a list of the rst ten Fibonacci numbers:
(5) The execution of the operator digits[13] results in the list {1,1,0,1} con-
sisting of the binary digits of the number 13.
2.3. Discussion of the above statements
We rst consider the statements of questionnaire 1 in order.
Those agreeing with statement 1 either do not knowBinet’s formula or fail to understand
its connection with the statement. That formula implies that Fn≈1
√51+√5
2n
,sothatFn
depends exponentially on n.
Of course, statement 2 is true since, by the denition of an arithmetic progression, we
have xn+1−xn=xn−xn−1,whencexn+1=2xn−xn−1.Itisclearthatasequencesatises
the latter recurrence relation if and only if the sequence is an arithmetic progression.
Assume that an n-element set contains kelements of type 1 and n−kelements of type 2.
Then, the formula for the number of permutations with repetitions takes the form n!
k!(n−k)!,
which is the number of k-combinations from a given n-element set. Note that, of course,
INTERNATIONAL JOURNAL OF MATHEMATICAL EDUCATION IN SCIENCE AND TECHNOLOGY 5
it is not actually necessary to use specic formulas for these numbers. One can use instead
the following purely combinatorial argument: There are n
kways to choose kplaces for
objects of type 1 in a permutation of length n, with repetition. Students who remembered
the proof of the formula for the number of permutations with repetition should easily be
able to respond correctly to Statement 3.
Weprovedinclassthatthesumofsquaresoftherstnnatural numbers is a cubic poly-
nomial in n. However, the fact that statement 4 is false is also clear from general consider-
ations.
Statement 5 is completely wrong. Indeed, by the well-known formula, we need 24−1=
15 moves for a tower of 4 discs, while we need 28−1=255 moves for a tower with 8 discs.
Students agreeing with Statement 5 have failed to understand that the number of moves in
the Hanoi problem depends exponentially on the number of discs.
Statement 6 is true because if two nite sets Aand Bhavethesamenumberofelements,
then a map f:A→Bis an injection if and only if fis surjection.
It would seem that the number of 4-element subsets of a set is greater than the number
of its 3-element subsets. However, this is true only if the set contains more than 8 elements.
Otherwise statement 7 is not true, since a 4-element set has only one 4-element subset, but
four 3-element subsets. Thus, Statement 7 is false.
Statement 8 is the only statement where any calculation is required. If a set has 4 ele-
ments, then the number of permutations of its elements is 4! =24, while the number of
its subsets is 24=16. One then argues by induction as follows. If n!>2n,then(n+1)! =
(n+1) ·n!>2·n!>2·2n=2n+1.Hence,Statement8istruesince,byassumption,the
number of elements in the given set is greater than 3.
Statement 9 is true because there is a one-to-one correspondence between the set
of all subsets of a given set Aand the set of all maps from Ato the 2-element set
{0, 1}.
Statement 10 would be true under the assumption that every student likes to engage in
at least one of the two activities. Note that if we reword the statement by writing ‘at least
ve’ instead of ‘exactly ve’, we get a true statement. Of course, Statement 10 is related to
the inclusion–exclusion formula.
Note that the above 10 statements cover most of the topics included in Survey 1.
We now turn to the statements of questionnaire 4.
Statement 1 is true. The code does remove duplicates from a given list while preserving
theorderofthelastoccurrencesoftheitemsinthelist.Forexample,applyingtheprocedure
nodupl to the list {1, 2, 3, 6, 6, 3, 5, 2, 1}, we obtain the list {6, 3, 5, 2, 1}. A similar procedure
was considered in class, the only dierence being that that procedure preserved the order
of the rst occurrences of the items in a list.
Statement 2 is also true because, on average, the bubble-sort algorithm requires twice
as many comparisons as the insertion sort. Moreover, the number of renumberings in the
bubble-sort depends quadratically on the length of the list, while for the insertion sort the
dependence is linear. The dierence in the execution speed was demonstrated to the stu-
dents in the computer lab.
Of course, Statement 3 is true. We deliberately changed the name of the procedure to
compel the students to think a little.
6CLASSROOM NOTE
Tab l e . Students’response to Questionnaire .
Statement
Right reaction
Wrong reaction
Neutral reaction
Statement 4 is false. The given code generates a list consisting of 10 ones. This question
tests the students’ knowledge of the CAS Mathematica syntax. The fact is that the form
fib1[1_]=1of the assignment operator entails that each value returned by fib1 is 1.
Statement 5 is also false. Though the given code indeed generates a list of binary digits of
the number 13, the digits will be written in a dierent order: the execution of the operator
digits[13] results in the list {1, 0, 1, 1}. The students should understand that the digits
are calculated starting from the least signicant digit. Since the operator Append appends
thenextelementtotheendofthelist,thebinarydigitsof13willbewritteninreverseorder.
Thus, we see that these ve statements test not only for knowledge of the syntax, but also
for knowledge of procedures, theoretical results, and ability to read codes.
3. Experiment evaluation
3.1. Students’ response
The Likert scale has been widely used in various psychometric studies, including research
in mathematics teaching. For example, there are many papers devoted to determining the
relation between mathematics instructors’ beliefs and practices (see e.g. [10–12]andrefer-
ences therein). In such studies, it was, of course, necessary to take into account the dier-
ence between the answers ‘agree’ and ‘strongly agree’. In our analysis of responses, however,
we did not distinguish between agreement and strong agreement nor between disagree-
ment and strong disagreement. We were concerned exclusively with basic understanding
of the subject: students should agree with a correct statement and disagree with a wrong
one.Butwewantedtotakeintoaccountthefactthatstudentscomingfreshtouniversity
often lack self-condence so tend not to give categorical answers. Another goal of the sur-
veys was to increase the students’ interest in the subject under study, and, in this respect, the
Likert scale was useful since it increases the emotional component of the learning process.
In Table 1,wegivethenumericalbreakdownoftheresponsesofthe21studentstothe
statements of Questionnaire 1, including the numbers of ‘undecides’.
This table clearly provides much material for discussion. The poor overall response to
Statement 2 was easily explained by the fact that the idea of dening an arithmetic progres-
sion by means of a recurrence relation had not been introduced in class: it was thus left
to the students to understand on their own that an arithmetic progression could be given
via a second-order recurrence relation. On the other hand, the fact that less than a half of
thestudentscorrectlyunderstoodStatement6indicatesthattheyhadnotunderstoodin
depth the concepts of injection and surjection. The fact that only a slim majority of stu-
dents understood Statements 3 and 4 would seem to indicate that, at that time, they were
having trouble understanding the denition of permutations with repetitions and did not
INTERNATIONAL JOURNAL OF MATHEMATICAL EDUCATION IN SCIENCE AND TECHNOLOGY 7
Tab l e . Students’ response to Question-
naire .
Statement
Right reaction
Wrong reaction
Neutral reaction
Tab l e . Students’ response to
the last Questionnaire.
Statement
Right reaction
Wrong reaction
Neutral reaction
remembertheformulaforthesumofsquaresoftherstnpositive integers (and, more-
over, did not remember the basic idea for deriving formulas for sums of exponents of these
numbers).
The results of Questionnaire 4 are as shown in Table 2.
The fact that two-thirds of the students responded incorrectly to Statement 4 indicates
that they do not understand the syntactic structure of the denition of new operators in
CASMathematica.TheresponsestoStatement1demonstratethatlargenumberofstudents
are not able to understand a program code.
We discuss the pedagogical conclusions of our experiment in the next section; for now
we just give one example involving the bubble-sort algorithm.
It can be seen from Ta b l e 2 that when answering Questionnaire 4, only half of the stu-
dents understood that this algorithm is less ecient than the insertion sort algorithm. It is
interesting to consider the change between the time this Questionnaire was administered
and the nal exam. To this end, the following statements were included in the last (pre-
examination) survey:
Statement 1. The bubble-sort algorithm is one of the most eective methods for sorting
large lists.
Statement 2. Applying this algorithm to a list consisting of 10 elements, in a worst case
scenario, one has to perform slightly fewer than 100 comparisons and the same number of
transpositions of pairs of elements in the list.
Statement 3. Doubling the number of elements of a list increases the time for sorting this
list by the bubble-sort algorithm approximately by the factor 4.
Table 3 indicates that by the end of the experiment, the majority of students had under-
stood that the complexity of the bubble-sort algorithm depends quadratically on the length
of the list, that the greatest number of comparisons needed for sorting a list consisting of
nelements by this algorithm is approximately equal, not to n2,button2/2, and that this
algorithmisinnowaythemosteective.
3.2. Student’s performance
In evaluating the results of our project, we used both objective and subjective data. The
objective data consisted in the distribution of students’ scores on the discrete mathematics
exam, as compared with the distribution of scores of students who sat the exam in 2016. (We
8CLASSROOM NOTE
Tab l e . Distribution of students’examination scores.
– – – – – – Mean
Spring % % % % % % .
Spring % % % % % % .
Tab l e . Average examination scores
in Calculus and Discrete Mathematics
courses.
Spring Spring
Calculus . .
Discrete math . .
Tab l e . Students’feedback on the experiment.
Didn’t affect at all Didn’t affect Don’t know Affected Strongly affected
were unable to use exam results from previous years because of dierences in the course
material.) Table 4 shows the percentage distribution of students’ scores across the indicated
intervalsaswellastheaveragescore.
One sees an obvious improvement in the results: the average examination scores
improvedby8.6percentagepoints,arelativeincreaseofmorethan17%.
We did n o t h av e t he o p p or tu ni ty t o fo r m a control group of students. Instead, as a next
best substitute, we compared our results with those obtained in the nal examination in
Calculus, taught in the traditional way. Tabl e 5 showstheaveragescoresobtainedbystu-
dents in that exam compared with their average score in discrete math.
(Theaveragescoreofstudentsenrolledayearearlierintheexaminationindiscrete
mathematics in Spring 2015 was 44.0.)
If we compare the ratio of the average scores obtained by students in the examinations in
discrete mathematics and Calculus in Spring 2016 and Spring 2017, we see that the dier-
ence between the average scores for 2017 and 2016 is denitely related to the methodology
used in teaching discrete mathematics in 2017. It follows from the above data that the ratio
increased by 12%, which is twice the value specied in [7].
The subjective data in question were the results of an anonymous survey conducted
after the examination. Students were asked to rate the extent to which their participation in
the experiment positively aected their understanding of the discrete mathematics course.
Table 6 presents the distribution of students’ answers. (Note that one student had dropped
out of the project).
Thus, the majority of students (86%) reported a positive impact of the methodology
used. Since this methodology has proven its eectiveness, we will introduce it into the edu-
cational process in the future, conducting similar surveys online.
4. Concluding remarks
We have here presented our rst experiment involving the use of a Likert-scale survey in a
discrete mathematics course. The eectiveness of this approach lies in the fact that it pro-
motes active learning among students. Discussion of their responses, both in small groups
and in the classroom, leads to a better understanding of the course material on their part.
INTERNATIONAL JOURNAL OF MATHEMATICAL EDUCATION IN SCIENCE AND TECHNOLOGY 9
At the same time, the fact that a student’s answer to a question formulated in an intriguing
way suddenly turns out to be incorrect may well increase his/her interest in the course and
desiretolearn.Informally,ourmodelofactivelearningmightbedepictedasfollows:
Understanding ⇒Response ⇒Reection & Discussion ⇒Understanding
In conclusion, we wish to note that the proposed pedagogical approach was also imple-
mented, though in a more limited fashion, in the teaching of a databases course for second-
year students. Within the course, only one survey (with subsequent discussion) was con-
ducted. However, more than 60% of students noted that even that single questionnaire con-
tributed signicantly towards their understanding of the course.
Acknowledgments
TheauthorsaredeeplygratefultoRobertG.Burns who kindly agreed to edit this manuscript.
Disclosure statement
No potential conict of interest was reported by the authors.
References
[1] Page RL. Software is discrete mathematics. 8th ACM SIGPLAN International Conference on
Functional Programming. Uppsala (Sweden): ACM Press; 2003. p. 79–86.
[2] Ivanov OA, Ivanova VV, Saltan AA. Discrete mathematics course supported by CAS Mathe-
matica. Int J Math Educ Sci Technol. 2017;48:953–963.
[3] Hämäläinen W. Problem-based learning of theoretical computer science. Proceedings of Fron-
tier in Education Conference. Savannah (GA): IEEE Press; 2004.
[4] Bonwel C, Eison J. Active learning: creating excitement in the classroom. Washington (DC):
School of Education and Human Development, The George Washington University; 1991.
(ASHE-ERIC higher education reports).
[5] Kim K, Sharma P, Land SM, et al. Eects of active learning on enhancing student critical think-
ing in an undergraduate general science course. Innov Higher Educ. 2013;38(3):223–235.
[6] Prince M. Does active learning work? A review of the research. J Eng Educ. 2004;93(3):223–
232.
[7] Freeman S, Eddy SL, McDonough M, et al. Active learning increases students performance in
science, engineering, and mathematics. Proc Natl Acad Sci. 2014;111:8410–8415.
[8] Georgiou H, Sharma MD. Does using active learning in thermodynamics lectures improve
students’ conceptual understanding and learning experience? Eur Phys J. 2015;36:015–020.
[9] McMaster K, Anderson N, Rague B. Discrete math with programming: better together. 38th
SIGCSE Technical Symposium on Computer Science Education; 2007;Covington,Kentucky,
USA: ACM Press. p. 100–104.
[10] Langrall CW, Thornton CA, Jones GA, et al. Enhanced pedagogical knowledge and reective
analysis in elementary mathematics teacher education. J Teach Educ. 1996;47:271–282.
[11] Stipec DJ, Givvin KB, Salmon JM, et al. Teachers’ beliefs and practices related to mathematics
instruction. Teach Teach Educ. 2001;17:213–226.
[12] Smith K. How teachers’ beliefs about mathematics aect students’ beliefs about mathematics
[honors theses. Paper 193]. Durham (NH): University of New Hampshire, University of New
Hampshire Scholars’ Repository; 2014.
