Effectiveness of Flipped Classroom Pedagogy in Programming Education: A Meta-Analysis

Full text

International Journal of Instruction April 2023 ● Vol.16, No.2
e-ISSN: 1308-1470 ● www.e-iji.net p-ISSN: 1694-609X
pp. 267-290
Citation: Almassri, M. A., & Zaharudin, R. (2023). Effectiveness of Flipped classroom pedagogy in
programming education: A meta-analysis. International Journal of Instruction, 16(2), 267-290.
https://doi.org/10.29333/iji.2023.16216a
Article submission code:
20220429182519
Received: 29/04/2022
Revision: 30/09/2022
Accepted: 24/10/2022
OnlineFirst: 10/01/2023
Effectiveness of Flipped Classroom Pedagogy in Programming Education:
A Meta-Analysis
Mirvat AH Almassri
Universiti Sains Malaysia, Malaysia, m_mervat@yahoo.com
Rozniza Zaharudin
Corresponding author, Universiti Sains Malaysia, Malaysia, roz@usm.my
The flipped classroom has generated considerable interest in programming
education in recent years. This meta-analysis aimed to assess the effectiveness of a
flipped classroom and traditional methods in teaching programming courses and
the impact on students’ performance, problem-solving abilities, and behavioural
outcomes, and to analyse the specific discipline, students’ type, students’ level, and
publication sources in the relevant studies. Articles published between 2010 and
2021 were searched carefully in six academic databases, comprising Web of
Science, IEEE Xplore Digital Library, ScienceDirect, NCBI Databases, and
Springer Link. Peer-reviewed articles written in English were selected and
screened according to the inclusion criteria. All the vital data were extracted and
stored in Microsoft Excel and meta-analysis was performed using the
Comprehensive Meta-analysis (CMA) software. A total of 101 articles were
retrieved while 27 of them met the inclusion criteria and were subjected to the
meta-analysis. Flipped classroom improved students’ achievement in programming
courses with statistically significant effect size (g = 0.56; p < 0.001, 95%
confidence interval; 0.33-0.79) compared to traditional teaching method. The
flipped classroom also favoured behavioural outcome (satisfaction) in
programming education. Programming subject areas had a significant moderating
impact on the effect sizes. Overall, evidence of publication bias was lacking in this
study. The findings and implications of implementing flipped classrooms in
programming education were highlighted. More studies are needed to elucidate the
effect of flipped classroom model on various dimensions of programming students’
learning outcomes to support comparative research in future.
Keywords: flipped classroom, student learning outcomes, lecture-based learning,
programming, meta-analysis
INTRODUCTION
Programming education remains one of the most difficult subjects for students at various
institutional levels (Sobral, 2021). Programming is not only challenging and complex,

268 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
students in science, engineering, and mathematics departments need to pass
programming courses at certain stages in their curriculum (Siti Rosminah & Ahmad
Zamzuri, 2012). To address this issue, several researchers have advocated for a radical
transformation in programming education (Threekunprapa & Yasri, 2020).
The challenges in comprehending computer programming can be classified into three
broad aspects: nature of the subject, student-related issues, and teaching-related issues
(Karaca & Ocak, 2017). In terms of the nature of the subject, there are two main aspects
in learning programming: programming strategies and programming knowledge (Davies
et al., 2013). Learning syntax and semantics of programming language are embedded in
programming knowledge, whereas the applications of such knowledge to innovate and
fabricate new programmes is regarded as programming strategies (Bayman & Mayer,
1988). Additionally, an algorithm-based solution can only be achieved when students
develop problem-solving skills, which is subsequently required for implementing a
computer programme. Thereafter, debugging codes and fixing syntax and semantic
errors are conditions to be met by students.
According to Hsu and Lin (2016), programming students are most times unaware of
their deficiencies and they rely mostly on reading textbooks and understanding language
syntax instead of practising to develop new programmes. Other researchers reported that
students are generally impatient to debug the errors in their codes, hence, they are
unable to create correct versions and error-free contents (Turan & Goktas, 2018).
Regarding teaching-related issues, instructors tend to spend time concentrating on
syntactic details instead of equipping students on how to create new programmes.
Meanwhile, traditional teaching methods or lecture-based instruction is only effective
for teaching language syntax, other vital aspects such as problem-solving, creation of
new programmes, debugging and fixing code bugs, and comprehending complete
programmes require more advanced teaching techniques (Davies et al., 2013).
A flipped classroom is a form of the blended learning platform (Atwa et al., 2022),
where students are exposed to and learn instructional content by watching video lectures
at the comfort of their homes, where lecturers or teachers provide personalised
interaction and guidance with other students, rather than in a typical classroom setting
(Al-zoubi, 2021). The definition by Lage (2000) simplified flipped or inverted
classroom as “Inverting the classroom signifies that events that traditionally occur within
the classroom now take place outside the classroom and vice versa”. Bishop and
Verleger (2013) posited that flipped classroom comprises two parts, namely, direct
computer-based individual instruction taking place outside the classroom and interactive
group learning activities within the classroom.
A flipped classroom model is more advantageous and suitable for teaching programming
courses by maximising the support provided for students learning (Turan & Goktas,
2018). This is achieved by freeing up class time for in-class activities, thereby helping to
shift the instructor’s role from teaching programming syntax to training students on
creating programming strategies. The broad components of learning programming:
strategies and knowledge, could be targeted by using the two phases of flipped class
model: out-class and in-class. Students could be taught programming language based on

Almassri & Zaharudin 269
International Journal of Instruction, April 2023 ● Vol.16, No.2
online recorded lectures with such knowledge serving as background and mandatory
programming skills, including programme comprehension, problem-solving, debugging
and correcting errors, building algorithm-based solutions, and writing new programmes
(Pattanaphanchai, 2019).
Several attempts have been made in implementing flipped classroom learning in
programming education and other related courses. For instance, flipped classrooms were
employed in computer science to teach introductory programming courses (Alhazbi,
2016; Antti et al., 2016; Elmaleh & Shankararaman, 2017; Marasco, Moshirpour, &
Moussavi, 2017) and advanced topics in software engineering (Paez, 2017). In these
studies, specific sessions within a course were flipped while some involved the entire
course. Nevertheless, contradicting outcomes were reported regarding the effectiveness,
as some researchers found positive learning impacts, whereas others reported either -
neutral or non-significant improvements when compared to traditional learning methods
(Elmaleh & Shankararaman, 2017).
Despite acknowledging the positive contributions of flipped classrooms in delivering
programming courses, only a few articles employed robust scientific techniques or study
designs to verify students’ learning (academic achievement and performance, problem-
solving abilities e.t.c.) and behavioural (i.e., motivation and satisfaction) outcomes. In
other words, robust experimental studies on the flipped classroom in programming
education are limited. A few systematic reviews have been performed to address flipped
classroom models in programming education, but meta-analysis was not conducted
probably due to data paucity and heterogeneity of studies. Nevertheless, a considerable
number of studies have been published recently, thus indicating the need and feasibility
to perform a meta-analysis to elucidate the effectiveness of flipped classroom approach
in programming education. This study aims to evaluate the effectiveness of flipped
classrooms in teaching programming courses in comparison to traditional or lecture-
based learning methods and the enhance on students’ performance, problem-solving
abilities, and behavioural outcomes and identify the subject area, student’s type,
students’ level and type of publication. The following research questions are addressed
in this article:
 What effects of the flipped classroom compared with the traditional lecture
(lecture-based learning) have been reported in programming education?
 How effective is using the flipped classroom in student achievement or
performance in programming courses?
 Is there a significant difference between the effect size in programming students’
achievement in relation to discipline, students’ type, students’ level, and
publication sources?
Theoretical Background
The first application of the flipped classroom model was in 2007 by a group of
chemistry teachers: Aaron Sams and Jonathan Bergmann. The main reason for applying
the model was to record video courses and make them available online. Thus, high

270 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
school students could watch the lesson at their convenience. Thereafter, the flipped
classroom model began to receive exceptional interest from several fields and students
in which courses were made available in downloadable format (Al Mulhim, 2020).
Besides, the technique was well-recognised as efficient in allocating the duration needed
to teach theoretical knowledge in combination with practical learning activities in the
traditional classroom (Bergmann & Sams, 2012).
Flipped classroom model is a form of blended learning model, whereby learning aligns
and considers the students’ learning levels and pace (Ayçiçek & Yelken, 2018).
Essentially, the responsibility to learn is transferred to the student. Hence, the term
“blended learning” is widely employed in describing flipped classroom model, which
comprises of the integration of traditional learning and technology. Yavuz et al. (2016)
posits that flipped learning entails the combination of traditional or face-to-face and
electronic teaching (online). Additionally, the model encourages problem-based,
inquiry-based, collaborative, and active learning theories. The weaknesses inherent in
the learning environment is removed to a certain degree, as documented in mobile
learning theory.
The social constructivist approach (SCA) has been used in elucidated the flipped
learning model. The SCA posits that social and culturally regulated experiences play
vital role in the structuring knowledge (Torun & Dargut, 2015). Furthermore, the
Bloom’s Taxonomy developed by Willian (2013) and Brame (2013) has been associated
with the flipped classroom model. This is based on the significance of “remembering”
and “understanding” the steps explained by the teacher via the theoretical knowledge
outside the classroom, meanwhile, concepts such as “Analysing”, “Applying”,
“Evaluating”, and “Creating” are delivered within the traditional classroom.
METHOD
Research Design
This meta-analysis research design applied to explore the effectiveness of flipped
classroom in teaching programming courses in comparison to traditional or lecture-
based learning methods. The study employed the following meta-analytic procedures:
retrieval of relevant articles, coding the articles features, estimating the effect sizes of
outcome measures in each study, and determining the moderating impacts of the study’s
features on the outcome measures.
Literature Search
The flipped classroom concept has been described using several terms in the context of
programming education, thus the researchers used all the related terms when searching
for relevant articles on the topic. The keywords used for the literature research included
language”, “computer science”, “computer programming”, “programming course”,
“introductory programming”, “novice programming”, “computer programming
education”.

Almassri & Zaharudin 271
International Journal of Instruction, April 2023 ● Vol.16, No.2
Articles published between 2010 and 2022 were searched carefully in seven academic
databases, comprising Web of Science, IEEE Xplore Digital Library, ScienceDirect,
NCBI Databases (PubMed and PMC), and Springer Link. Apart from the primary search
engines, references from the retrieved articles were also assessed if considered relevant
to the research topic.
Inclusion and Exclusion Criteria
A comprehensive set of inclusion and exclusion criteria were developed in this meta-
analysis. As shown in Table 1, all the articles included in this study were published
between 2010 and 2022, written in English, and focused on utilising flipped classroom
models in delivering programming courses in any discipline. Additionally, the studies
utilised either empirical, quasi-experimental, experimental, randomised control trials
(RCT), or longitudinal study designs in comparing flipped classroom and lecture-based
teaching methods. The student learning outcomes measures were either between or
within-subjects conditions. Articles providing detailed data (mean, standard deviations
(SD), sample size and corresponding inferential statistical test values such as t-value) to
compute the effect sizes were also considered. Studies were included if the student
learning or behavioural outcomes were clearly defined and described quantitatively for
the experimental or observational groups.
Table 1
Article inclusion and exclusion criteria
Criteria
Learning content
Involving flipped classroom in programming education
Language
English
Timeframe
Articles published between 2010 and 2021
Literature type
Peer-reviewed articles, dissertation/theses, conferences and proceedings
Research design
Experimental, quasi-experimental, RCT, observational (longitudinal and
prospective cohort)
Implementation
Flipped classroom
Accessibility
Full texts are available either as open access articles or via library repository
Research
outcomes/results
Basic statistical data to estimate the effect size (mean, standard deviation,
sample size, statistical test values)
Educational
outcomes/results
Well-described educational outcomes
Articles Identification and Selection
The identification and selection of articles in this study involved three phases. The titles
and abstracts were first screened to ensure they were related to flipped classrooms in
programming education. Several publications including journal articles, conferences and
proceedings. Meanwhile, reviews, online articles, and articles reporting flipped
classroom models in other disciplines besides programming were excluded. Thereafter,
all the relevant studies were imported into a Microsoft Excel spreadsheet. Duplicates in
various databases were removed and upon completing the data screening process, a total
of 101 articles were available for consideration (See Figure 1).

272 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
Figure 1
Study selection process and flow diagram using the PRISMA guideline
Coding features and procedures
A set of relevant variables such as subject areas, study duration, and types of publication
were used to code all the included articles. All the data available in computing the effect
size were extracted from each study. Attempts were made to code for the instructional
media, pre-class activities, and in-class activities but the data were inadequate to
facilitate a consistent coding system.
All the authors of this review participated in developing the coding scheme upon
reading a given number of articles that were randomly allocated. In instances where
opinions differed on how the variables should be coded, a discussion was held among
the authors to reach a consensus and resolve the differences. Although inter-rater
reliability (Kappa coefficient) was not conducted in this study, this would not affect the
meta-analysis results since the final coding process was performed consistently by three
authors of this review. Moreover, frequent discussions and communication were held to
reach a consensus on the coding process. Coding related to the quantitative data was
also examined for errors and corrected. The final coding comprised information on
subject areas (engineering, computer, and ‘others’) student groups (undergraduates or
post-graduates), and publication types (research articles and conference proceedings).
The following details were coded in the Microsoft Excel spreadsheet: year of
publication, author details, the title of the articles, type of publication, subject area or
discipline, study duration, and available data for effect size calculation.

Almassri & Zaharudin 273
International Journal of Instruction, April 2023 ● Vol.16, No.2
Extraction and Estimation of Effects Sizes
The effect sizes for the 27 articles included in the final analysis were estimated using the
Comprehensive Meta-analysis (CMA) version 3.0. Descriptive statistics of the data set
was performed using the Statistical Package for Social Science (IBM SPSS, Version
24). The effect sizes were extracted based on three domains namely, student overall
achievement or performance (final examination scores), problem-solving abilities, and
behavioural outcome (i.e., satisfaction). Notably, these domains were reported in a good
number of the included studies. However, other outcomes such as motivation,
competency acquisition, attentiveness, self-efficacy, and attitude were also reported in a
few articles.
Each of the studies reporting sufficient data for effect size estimation was analysed
following the suggestion by Lipsey and Wilson (2001). Only one effect size was
estimated for each study to prevent statistical dependence and bias in the overall results.
The averages in the CMA was employed to combine the effect size comparisons.
Several researchers have described various effect size comparison methods and
addressed the associated issues in the meta-analysis (Scammacca et al., 2014; Lpez-
Lpez et al. 2018). While the method employed in this study is considered divergent and
integrative, each method has its cons and pros as elaborated in the discussion session.
All the effect sizes were standardised in Hedge’s g before performing the meta-analysis.
The Hedge’s g is considered a standardised measure of effect size when dealing with
continuous data. Moreover, Borenstein et al. (2010) reported that Hedge’s g is more
effective than Cohen’s d when adjusting for bias relating to small sample size.
Both the fix and random model effects were compared in the meta-analysis. According
to Borenstein et al. (2010), random-effects models are best conducted when effect sizes
in reviewed studies differ from each other. Additionally, mixed-effects analysis in the
CMA software was employed to perform the post-hoc subgroup analyses. Effect sizes of
0.2 and below were considered small, values between 0.3 and 0.7 were considered
medium, and values of 0.8 and above were classified as large (Cohen, 1992). Visual
inspection of funnel plots, Orwin’s fail N test and fail-safe N procedure were used to
evaluate publication bias (Orwin, 1983).
FINDINGS
A meta-analysis was performed on 27 articles identified from the systematic literature
search and extracting process. These flipped classroom studies focused on the delivery
of programming courses and they were published mostly in conferences/proceedings (n
= 12) and journal articles (n = 15). Table 2 shows the summary of the articles in terms
of study designs, subject areas, students involved and their levels, main studied
variables, and articles reporting sufficient data for effect size estimation. The majority of
studies were quasi-experimental (17/27; 62.9%), followed by surveys (5/27; 18.5%),
observational (4/27; 14.8%) and only a single randomised control trial (RCT). As
expected, the main studied subject area was computer science (18/27; 66.7%) and 24
articles focused on undergraduates (88.9%). Meanwhile, 15 articles (55.6%) did not

274 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
state the students’ level. Overall, 18 studies provided sufficient data for effect size
estimation.
A higher number of articles (n = 18) reported student learning (performance/
achievement) and behavioural outcomes (satisfaction; n = 4). Other aspects investigated
in the reviewed studies included students’ problem-solving ability (Lin, 2019; Hsu &
Lin, 2016), attention, confidence (Chang et al., 2018), competencies acquisition
(Pattanaphanchai, 2019; Elmaleh & Shankararaman, 2019), learning motivation (Lin,
2019; Abdallah et al., 2020), and learning attitude (Lin, 2019).
Table 2
Descriptive analysis of the studies included in this meta-analysis
Variables
Number of studies (n)
%
Study designs
Quasi-experimental
17
62.9
Observational (prospective and longitudinal)
4
14.8
Survey
5
18.5
RCT
1
3.7
Subject areas
Computer science
18
66.7
Engineering
3
11.1
Computer science and engineering
3
11.1
Others
3
11.1
Student type
Undergraduates
24
88.9
Post-graduates
0
0.0
Unspecific
3
11.1
Student level
First-year
5
18.5
Second-year and above
7
25.9
NA
15
55.6
Main studied variables
Students’ performance/achievement
18
66.7
Satisfaction
4
14.8
Problem-solving abilities
3
11.1
Learning motivation
3
11.1
Competencies acquisition
2
7.4
Attitude
2
7.4
Attentiveness
1
3.7
Confidence
1
3.7
Self-efficacy
1
3.7
Articles reporting sufficient data for effect size estimation
Student performance
14
77.7
Satisfaction
2
16.1
Problem-solving abilities
2
16.1
Effectiveness of Flipped Classroom and Lecture-Based Teaching (Traditional
Method)
These results are divided into three main areas based on the areas investigated, namely,
student’s achievement or performance, problem-solving ability, and satisfaction with

Almassri & Zaharudin 275
International Journal of Instruction, April 2023 ● Vol.16, No.2
flipped classrooms. Other aspects in a few studies included attentiveness, confidence,
competencies acquisition, learning motivation, and learning attitude.
Students’ performance and achievement
A total of 14 articles compared the effects of flipped classrooms and lecture-based
methods on students’ achievement or performance in programming courses. Figure 2
presents the combined effect size of the 14 articles in terms of authors’ name, year of
publication, and statistic parameters such as the standard error, Hedge’s g, variance,
confidence interval, Z-value, and p-value. Each study contributed a specific effect size
that is indicated by the small boxes. Meanwhile, the confidence interval of the estimate
from each study is represented by the horizontal line that crosses each box. Upon
pooling all the studies combined with a confidence interval, the average effect size is
depicted by the diamond at the bottom of the plot.
Figure 2
Effect sizes of each study comparing student achievement/performance in programming
courses using flipped classroom and lecture-based learning
The forest plot revealed that 12 studies favoured the application of flipped classrooms
(experimental group) and their corresponding effect sizes were statistically significant
(Figure 2). Only one study was neither in favour of the flipped classroom nor lecture-
based method and the effect size was not statistically significant since the confidence
interval overlapped with zero (Cabi, 2018). The effect size differed between studies,
with those conducted by Chang et al. (2018), Elmaleh and Shankararaman (2017), and
McCord and Jeldes (2019) contributing the highest effect size, which could be attributed
to the large sample size. Nevertheless, sampling error might also be responsible for the
variation in effect sizes observed in the included studies.
Table 3 depicts the overall effect size of the fixed and random-effects model, with a g-
value of 0.41 and 0.56, respectively. These values are trivial to small effect size (Cohen,
1992), and they were both statistically significant at Z-values of 12.01 (CI 0.35-0.48)
and 4.45 (CI 0.32-0.81), respectively. The Q-statistic was checked to determine if the

276 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
studies included in the meta-analysis are homogeneous and characterised by common
effect sizes. In other words, the Q-statistic tests the null hypothesis regarding the studies
homogeneity (Borestein et al., 2010). The Q-value was 131.6 with a degree of freedom
(df) of 14 and was statistically significant at P < 0.001. Hence, the null hypothesis that
the actual effect size is identical in all of the studies was rejected. Meanwhile, a high
heterogeneity level of 90.1% (I-squared value) was detected (Higgins and Thompson,
2002). This result suggests that other moderators aside from the sampling error might be
responsible for this high heterogeneity (Borenstein et al., 2010). However, this is
unlikely as only one study found no significant difference in students’ performance in
programming courses between flipped classrooms and traditional teaching methods. In
other words, no study reported that students in the traditional room flipped classroom
performed worse than the lecture-based method.
Table 3
Overall effect size of the fixed and random-effects model for studies comparing student
achievement/performance in programming courses and satisfaction with using flipped
classroom and lecture-based or traditional teaching method
95% CI
Heterogeneity
Achievement/Performance
Model
K
ES
SE
Variance
Lower
Upper
Z
P
Q
Df (Q)
P
Fixed
14
0.41
0.03
0.001
0.35
0.48
12.01
0.000
131.62
13
0.000
Random
14
0.56
0.13
0.01
0.32
0.81
4.45
0.000
Satisfaction
Fixed
2
0.91
0.17
0.03
0.56
1.25
5.19
0.000
9.35
1
0.002
Random
2
1.22
0.64
0.41
-0.03
2.47
1.92
0.05
K = number of studies, SE = standard error, df = degree of freedom
Apart from the aforementioned studies, four other articles reported students’
performance using different study designs and methodologies. Wang et al. (2019) found
that the students obtaining good grades in the flipped classroom increased significantly
by 15% in two successive years, whereas no significant difference was detected in
students subjected to lecture-based methods. A similar study by Pattanaphanchai (2019)
reported an overall improvement in students’ examination scores in various
programming courses upon changing from traditional to flipped classroom pedagogy.
Meanwhile, the final exam scores of students subjected to flipped classrooms improved
significantly compared to the scores obtained in previous two years (Hayashi et al.,
2015).
Students’ satisfaction with flipped classrooms
Effect sizes were only estimated for two studies (Chang et al., 2018; Hsu & Lin, 2016)
comparing students’ satisfaction with flipped classroom and lecture-based methods in
delivering programming courses. As expected, the effect size contributed by Chang et al.
(2018) was much higher compared to Hsu and Lin (2016), which is clearly due to the
larger sample size. Both studies favoured students’ satisfaction with flipped classrooms
compared to lecture-based methods, and the effect sizes were statistically significant.
The overall effect size of the fixed and random-effects models are shown in Table 3,
with g values of 0.91 and 1.22, respectively. These values are also trivial to small effect

Almassri & Zaharudin 277
International Journal of Instruction, April 2023 ● Vol.16, No.2
size and both were statistically significant at Z-values of 5.19 (CI 0.56-1.26) and 1.92
(CI -0.02-2.47), respectively. The Q-value was 9.35 and P = 0.002. Likewise, the null
hypothesis that the true effect size was the same in all studies was rejected. The I-
squared value was 89.3%, which is also considered a high level of heterogeneity
(Higgins & Thompson, 2002).
Other students’ learning and behavioural outcomes
Apart from students’ achievement and satisfaction, other aspects investigated in the
reviewed studies included students’ problem-solving ability (Lin, 2019; Hsu & Lin,
2016), attention, confidence (Chang et al., 2018, competencies acquisition
(Pattanaphanchai, 2019; Elmaleh & Shankararaman, 2019), learning motivation (Lin,
2019; Abdallah, 2020; McCord and Jeldes, 2019), and learning attitude (Lin, 2019;
Taşpolat et al., 2021).
Students’ problem-solving ability was reported in two studies (Lin, 2019; Hsu & Lin,
2016) comparing the effects of flipped classrooms and lecture-based methods in
delivering programming courses. Both studies contributed similar effect sizes and
favoured the flipped classroom in improving students’ problem-solving ability in
programming courses. Other statistical parameters were not computed given that only
two studies were included in this analysis.
The application of flipped classrooms in delivering programming courses was found to
increase students’ attention and confidence compared to the traditional teaching method
(Chang et al., 2018). In terms of competencies acquisition, students’ knowledge and
expertise increased significantly (P < 0.05) following the implementation of flipped
classrooms with pre-and post-intervention scores (mean, SD) of 3.14±0.72 and
3.57±0.69 (Pattanaphanchai, 2019). Another study by Elmaleh and Shankararaman
(2019) reported that competencies acquisition among students in the flipped classroom
increased significantly by 27% compared to 20% in the lecture-based method. Likewise,
students were more motivated and demonstrated significantly higher attitude scores
towards flipped classrooms in teaching programming courses compared to the
traditional method (Lin, 2019; Abdallah et al., 2020; McCord & Jeldes, 2019).
Students’ level and subject areas
The analysis could not be broken down by student levels, as 24 studies were conducted
among undergraduate students while the remaining three articles were unspecific. In
terms of subject areas, 21 studies were conducted among only computer science students
(Elmaleh & Shankararaman, 2019; Chang et al., 2018a; 2018b; Abdallah, 2020; Hsu &
Lin, 2016; Jonsson, 2015; Loftsson & Matthiasdottir, 2021; Souza & Rodriguez, 2015;
Cabi, 2018; Wang et al., 2019; AlJarrah et al., 2018; Durak, 2019; Patrick, 2016; Indi,
2016; Hayashi et al., 2015; Zhuo & Qi, 2015; Puarungroj, 2015; Mithun & Evans, 2018;
Taşpolat et al., 2021; Amira et al., 2019; Ruiz de Miras et al., 2022), three studies
among Engineering students (Nikolic et al., 2019; Lin, 2019; McCord & Jeldes, 2019),
two studies among Computer and Engineering students (Karaca & Ocak, 2017; Alhazbi
et al., 2016) and one study involved students from several disciplines (Pattanaphanchai,
2019).

278 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
The effect sizes by subject area comparing student performance in flipped classrooms
and lecture-based methods are shown in Table 4. The flipped classroom was favoured
with corresponding positive effect sizes than the traditional method. Given that most
studies enrolled at least computer science students (k = 13), the subject area recorded
the highest population of participants (N = 2106) with the moderate effect size at g =
0.59 and statistically significant (p < 0.001). Meanwhile, the engineering subject area (k
= 2) involved a total of 1751 students, with a small effect size at g = 0.19, which was
also statistically significant. The subject areas (computer and engineering) were
subjected to a post-hoc test. The findings indicate that Engineering students benefit less
from the use of flipped classrooms in delivering programming courses compared to
Computer science students. Nevertheless, there is no statistical evidence to suggest that
the Engineering discipline will benefit significantly from the traditional method.
Table 4
Effect sizes of flipped classroom students’ achievement/performance based on subject
areas and publication sources
Effect Size and 95% CI
Heterogeneity
N
K
G
SE
Lower
Upper
Z
P
Q
Df (Q)
P
Subject area
Computer science
2106
13
0.59
0.13
0.34
0.85
4.58
0.000
98.60
13
0.000
Engineering
1751
2
0.19
0.05
0.06
0.28
3.68
0.000
0.18
1
0.67
Publication sources
Conference/proceeding
1110
8
0.67
0.19
0.29
1.06
3.40
0.001
88.32
7
0.000
Journal articles
2622
7
0.54
0.21
0.13
0.95
2.57
0.010
92.06
6
0.000
K = number of studies, SE = standard error, df = degree of freedom
Sources of Publication
Table 4 also illustrates the distribution of effect sizes according to publication sources,
specifically for articles comparing students’ achievement in flipped classrooms and
traditional teaching methods. The studies were equally divided into conference (k = 8)
and journal articles (k = 7) with a total of 1110 and 2622 programming students,
respectively. Overall, the effect size was 0.67 for conference and 0.54 for journal
articles and both were statistically significant. These effect sizes are considered
moderate to high. Both publication sources favoured the use of flipped classroom
models compared to lecture-based or traditional methods with high heterogeneity level
above 90.0%.
Publication Bias
Publication bias was investigated for each of the result sections that were eligible for
such analysis. The funnel plot generated from the first meta-analysis of studies
comparing the effects of flipped classrooms and lecture-based methods on students’
achievement or performance is shown in Figure 3. Meanwhile, Figures 4 and 5 depict
the funnel plots for publication bias based on publication sources, conference and
journal respectively. Upon visual inspection, all the funnel plots present an overall
symmetrical distribution around the weighted mean effect sizes. Sterne and Egger
(2001) described a funnel plot as a scatter plot of effect sizes computed from each study

Almassri & Zaharudin 279
International Journal of Instruction, April 2023 ● Vol.16, No.2
including a meta-analysis against a measure of study precision as quantified by the
standard error. Specifically, the vertical and horizontal axis in the diagram represents the
standard errors and the Hegdes’ g, respectively. The presence of a symmetric funnel plot
indicates that the meta-analysis lacks a publication bias (Duval and Tweedie, 2000).
Figure 3
Funnel plot of standard error by hedges’ g for the 14 articles comparing the effects
between flipped classroom and traditional teaching method on students’ performance in
programming courses
Figure 4
Funnel plot of standard error by hedges’ g for the conference articles (n = 8) comparing
the effects between flipped classroom and traditional teaching method on students’
performance in programming courses

280 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
Figure 5
Funnel plot of standard error by hedges’ g for the journal articles (n = 7) comparing the
effects between flipped classroom and traditional teaching method on students’
performance in programming courses
Table 5 shows the Classic fail-safe and Orwin’s fail-safe N tests. Resultantly, the overall
effect size detected in the current meta-analysis could only be nullified following the
addition of another 535 studies of programming students’ learning or performance
outcomes. Thus, the absence of publication bias is further confirmed based on the funnel
plots and fail-safe N tests results. However, these tests were not conducted for students’
satisfaction given that at least three studies are required for publication bias analysis.
Likewise, publication bias could not be performed for studies regarding the effects of
flipped classrooms on students’ problem-solving abilities.
Table 5
Classic fail-safe and Orwin’s fail-safe N tests to assess publication bias in studies
reporting the effects of flipped classroom on students’ performance/achievement in
programming courses
Classic fail-safe N
Achievement/performance
Z-value for observed studies
12.27
P-value for observed studies
0.00
Alpha
0.05
Tails
2.0
Z for Alpha
1.95
Number of observed studies
14.0
Number of missing studies that would bring p-value to > alpha
535
Orwin’s fail-safe N
Hedge’s g in observed studies
0.41
Criterion for a trivial std diff in means
0.00
Mean hedge’s g in missing studies
0.00

Almassri & Zaharudin 281
International Journal of Instruction, April 2023 ● Vol.16, No.2
DISCUSSION
This study entailed a systematic assessment and meta-analysis of the effects of flipped
classrooms on learning on students’ performance/achievement, learning satisfaction, and
problem-solving abilities in programming education. In order to discuss the effects
objectively, the main findings are elucidated in the following
The first research question addressed in this study is to compare the effects of flipped
classrooms and traditional teaching methods on students’ performance or achievement
in programming courses. Accordingly, 14 studies reporting students’ performance in the
selected articles considered flipped classrooms as the experimental group and traditional
lecture-based learning as the control group. Most of the studies were quasi-experimental
and only one was a randomised control trial. The computed effect size based on the
fixed and random effects was 0.41 and 0.56, respectively. These values were trivial to
small and moderate effect size (Cohen, 1992) and statistically significant (P < 0.05).
These effects represent the hinge point (> 0.4) of the average effects of educational
interventions that should be the aim of teachers and researchers (Hattie, 2012).
Nevertheless, these interpretative rules of effect sizes need to be elucidated in terms of
the underlying dependent variable assessed in this study, which is students’
performance. For instance, an effect size of 0.41 indicates that the average score of a
student in the flipped classroom is 0.41 standard deviations above the average student in
the traditional teaching method. In other words, 59% of the students in the flipped
classroom will score above the mean of the students in the traditional classroom. These
interpretations revealed that the effect on students’ performance seems small but very
meaningful in the context of programming education.
The effect sizes found in this study are consistent with other meta-analyses conducted
among higher education students (Spanjers et al., 2015; Chen et al., 2018; van Alten et
al., 2019). For instance, an insight into comparable meta-analyses to gauge the relative
size of effects was provided by Schneider and Preckel (2017). The researchers ranked a
total of 105 variables according to the strength of their association with higher education
students’ achievement and found that an effect size of 0.36 is comparable to other
interventions on the instruction variable technology such as blended learning (0.33,
52nd position) and intelligent tutoring systems (0.35, 47th position). Overall, the present
outcomes align with prior flipped classroom meta-analysis reporting small average
effect sizes that ranged from 0.19 to 0.47 on students’ learning outcomes (Spanjers et
al., 2015; Chen et al., 2018; Hew & Lo, 2018; Lo et al., 2017; van Alten et al., 2019).
Given that this study is the first attempt to perform a meta-analysis on flipped
classrooms in programming education, these findings could impact future research
positively.
Another important aspect in this meta-analysis was students’ satisfaction with the use of
flipped classrooms and traditional teaching methods in teaching programming courses.
Resultantly, a high effect size was detected following the meta-analysis of the relevant
articles. All three studies favoured students’ satisfaction with flipped classrooms
compared to lecture-based methods, and the effect sizes were statistically significant.
Nevertheless, this result should be interpreted with caution as it does not completely

282 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
mean that students were dissatisfied with traditional classrooms. The number of studies
included in this analysis is relatively small to make such a strong conclusion. The
present findings are inconsistent with the previous meta-analyses (Spaniers et al., 2015;
van Alten et al., 2019), where blended learning had a non-significant trivial effect size
on students’ satisfaction. However, these studies included articles from various
disciplines while the present study focused on programming courses. Given the large
heterogeneity between the studies included in this analysis, the impact of flipped
classrooms on programming students’ satisfaction requires further investigation. For
instance, Loftsson and Matthiasdottir (2021) reported that 47% and 33% were satisfied
and dissatisfied with flipped classrooms, respectively. Additionally, 60% of surveyed
students were satisfied with the teaching method but 50% of them felt that the course
lacked traditional lecturing. Therefore, the design and educational context of the flipped
classroom needs to be carefully planned before implementation for programming
courses.
Students’ problem-solving ability was reported in two studies (Lin, 2019; Hsu & Lin,
2016) comparing the effects of flipped classrooms and traditional methods in delivering
programming courses. Both studies contributed similar effect sizes and favoured the
flipped classroom. Other aspects that were reported in the reviewed studies included
students’ attention, confidence (Chang et al., 2018), competencies acquisition
(Pattanaphanchai, 2019; Elmaleh & Shankararaman, 2019), learning motivation (Lin,
2019; Abdallah, 2020), and learning attitude (Lin, 2019). These studies were not
sufficient to perform a meta-analysis, however, most of the findings also favoured the
application of flipped classrooms.
The third research question addressed in this study is the possible moderator effects of
students’ level, subject areas, and publication sources on the effectiveness of flipped
classrooms for programming courses. In all the meta-analyses, the significant
heterogeneity in effect sizes between studies was mainly attributed to random sampling
given the variation in sample sizes. Moreover, no study reported that students in the
flipped classroom performed worse than the lecture-based method in terms of
performance, learning outcome, satisfaction, and problem-solving ability.
Moderating effects were sparingly found in this study. The main moderating effect
detected in this study was the students’ discipline as the flipped classroom was more
effective for computer science students compared to those in engineering. This is in line
with the FTC meta-analysis by Cheng et al. (2018), who found that subject areas
significantly moderated their results. Nonetheless, the meta-analysis for students’
satisfaction had low power for proper moderator analysis. Thus, there is no strong
evidence to ascertain that moderator effects were absent.
Various subject areas or disciplines have been investigated in previous meta-analyses
comparing the flipped classroom to the traditional teaching method (Holdhusen, 2015;
Karabulut-Ilgu et al., 2018; van Alten et al., 2020). The present analysis involved only
two major disciplines: Computer science and Engineering, which is expected as these
subject areas entailed the introduction of undergraduates to various programming
courses. Resultantly, computer subjects benefitted more (higher effect size) from the

Almassri & Zaharudin 283
International Journal of Instruction, April 2023 ● Vol.16, No.2
implementation of flipped classrooms in delivering programming courses than
engineering subjects. This outcome aligns with a previous meta-analysis in which the
subject of Engineering underperformed compared to other disciplines following the
introduction of flipped classrooms (Cheng et al., 2019). However, this result needs to be
interpreted with caution as there is no evidence to suggest that flipped classrooms will
impact Engineering subjects negatively, and only two studies in this review focused on
Engineering students. Moreover, the Engineering field has been among the top
advocates of the flipped classroom model (Holdhusen, 2015). A previous meta-analysis
found a low number of studies reporting the use of flipped classroom models in
engineering subjects (Karabulut-Ilgu et al., 2018). Given that the present meta-analysis
focused on programming education, it is not surprising as only three relevant
engineering studies were identified in this study. Most of the articles identified in the
initial literature search lacked mean scores, SD, and sample sizes required for
performing a meta-analysis.
All the reviewed studies were conducted among undergraduate students. Hence, student
levels were not subjected to further analysis in this study. This result is unsurprising as it
aligns with a previous meta-analysis in which undergraduates accounted for the highest
percentage of students enrolled in studies comparing flipped classrooms and traditional
teaching methods (van Alten et al., 2020). Graduate students were not included in any of
the 28 reviewed articles, which might be due to the relatively low implementation of
lectures in graduate programming education compared to undergraduate students.
Moreover, graduate programming students usually analyse research works when outside
the class while the little time on lectures is mainly under the traditional lecturing
method.
Most of the publications that met the inclusion criteria were articles from conferences
and proceedings. Additionally, these articles demonstrated small to moderate effect sizes
favouring flipped classrooms over traditional methods in delivering programming
courses. Similar results were also detected in the journal articles comparing flipped
classrooms and the traditional method. Meanwhile, no significant difference was
observed between the effect sizes of publication sources (conferences/proceedings vs
journal articles). In general, no strong evidence of publication bias was observed in the
empirical studies on flipped classroom models included in the present meta-analysis.
Apart from the assessment of the funnel plot of studies included in the analysis, the
Classic fail-safe N test and Orwin’s fail-safe N test were also computed to determine if
any publication bias existed. Nevertheless, all the methods consistently showed that
evidence of publication bias was lacking in the meta-analysis.
LIMITATIONS
This meta-analysis focused on the effectiveness of flipped classroom model on various
student learning and behavioural outcomes in programming education. However, only a
few studies reported students’ behavioural outcomes, such as satisfaction and
motivation, hence, effect sizes were mainly estimated from a relatively high number of
studies reporting learning outcomes (academic performance, achievement, problem-
solving abilities and so on). Hence, findings from this study might inform policymakers

284 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
in the educational sector more about cognitive learning outcomes when deciding to
implement flipped classrooms for delivering programming courses.
Most of the studies did not mention the student level, study duration and frequencies of
the flipped classroom implementation. Thus, the potential moderating role of these
variables was not analysed in this study. Attempts were made to assess the moderating
role of the subject area but the number of studies, especially for the engineering
discipline, was very small. Given that flipped classroom has continued to receive
extensive interest among researchers, more evidence will be available across various
programming subjects in the future for a more robust analysis. Pedagogical
characteristics and study quality were also not coded in this meta-analysis. Most of the
analysed studies lack sufficient detail on flipped classroom implementations.
Meanwhile, the majority of studies were quasi-experimental as only a single RCT was
conducted among programming students.
Additionally, the articles included in this meta-analysis were mainly from
conferences/proceedings and journals. Given that the former publication sources are not
frequently subjected to rigorous review, more peer-reviewed journal articles are needed
to gain highly robust data to improve the present knowledge on flipped classroom
effectiveness in programming education.
CONCLUSION
This meta-analysis reviewed previous studies investigating the effectiveness of flipped
classrooms in programming education. Furthermore, the analysis focused on
comparative studies between the flipped classroom and traditional teaching methods in
delivering programming courses. Upon computing the overall effect, the flipped
classroom was favoured over traditional teaching methods in terms of students’ learning
outcomes, mainly achievement/performance and problem-solving ability, with a small to
moderate effect size. Similarly, the flipped classroom was favoured ahead of the
traditional method in terms of students’ satisfaction with methods of delivering
programming courses. Factors such as student type and type of publication had no
moderating effects on the results, however, the subject area seems to moderate the
effectiveness of flipped classrooms. This study is the first attempt to perform a meta-
analysis of flipped classroom implementation in programming education. Hence, these
findings may be helpful to researchers, educations and practitioners either when
designing or deciding to introduce flipped classroom pedagogy in delivering
programming courses. More research articles are needed to elucidate the impact of
flipped classroom models on various dimensions of programming students’ learning
outcomes, including performances, attitudes, satisfaction, self-efficacy and so on.
REFERENCES
Abdallah, M. (2020). Evaluating the Effectiveness of Flipped Teaching in a Mixed-
Ability CS1 Course. In Proceedings of 2020 ACM Conference on Innovation and
Technology in Computer Science Education (ITiCSE’20), Jun 15-19,2020, Trondheim,
Norway. ACM, New York, NY, USA, 7 pages.
https://doi.org/10.1145/3341525.3387395.

Almassri & Zaharudin 285
International Journal of Instruction, April 2023 ● Vol.16, No.2
Ahmed, A.A & Yousuf, M.I. (2016). Flipping introductory engineering design courses:
Evaluating their effectiveness, IEEE Global Engineering Education Conference,
EDUCON, 10-13, 234–239.
Al Mulhim, E. N. (2021). Flipped Learning, Self-Regulated Learning and Learning
Retention of Students with Internal/External Locus of Control. International Journal of
Instruction, 14(1), 827-846.
Alhazbi, S. (2016). Using flipped classroom approach to teach computer programming.
In 2016 IEEE International Conference on Teaching, Assessment, and Learning for
Engineering (TALE) (pp. 441-444). IEEE.
AlJarrah, A., Thomas, M.K., Shehab, M. (2018). Investigating temporal access in a
flipped classroom: procrastination persists. International Journal of Educational
Technology in Higher Education, 15:1
Al-Zoubi, A. M., & Suleiman, L. M. (2021). Flipped Classroom Strategy Based on
Critical Thinking Skills: Helping Fresh Female Students Acquiring Derivative
Concept. International Journal of Instruction, 14(2), 791-810.
Amira, T., Lamia, M., & Mohamed, H. (2019). Flipped classroom for algorithmic
teaching. In Proceedings of the 2nd International Conference on Networking,
Information Systems & Security (pp. 1-4).
Antti, K., Antti, H., Erno, V., & Jouni, I. (2016). The flipped classroom method:
Lessons learned from flipping two programming courses. In The 17th International
Conference on Computer Systems and Technologies, 1164, 423- 430, June 23.
Atwa, Z., Sulayeh, Y., Abdelhadi, A., Jazar, H. A., & Erigat, S. (2022). Flipped
classroom effects on grade 9 students’ critical thinking skills, psychological stress, and
academic achievement. International Journal of Instruction, 15(2), 737-750.
Ayçiçek, B., & Yanpar Yelken, T. (2018). The Effect of Flipped Classroom Model on
Students' Classroom Engagement in Teaching English. International journal of
instruction, 11(2), 385-398.
Bayman, P., & Mayer, R. E. (1988). Using conceptual models to teach BASIC computer
programming. Journal of Educational Psychology, 80(3), 291.
Bergmann, J., & Sams, A. (2012). Flip Your Classroom: Reach Every Student in Every
Class Every Day (pp. 120-190). Washington DC: International Society for Technology
in Education.
Bishop, J. L. & Verleger, M. A. (2013). The flipped classroom: A survey of the
research. In Proceedings of the American Society for Engineering Education Annual
Conference and Exposition (ASEE) (Atlanta, GA, 2013)
Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2010). A basic
introduction to fixed- effect and random-effects models for meta-analysis. Research
Synthesis Methods, 1(2), 97–111.

286 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
Brame, C. (2013). Flipping the classroom. Vanderbilt University Center for Teaching.
http://cft.vanderbilt.edu/guides-sub- pages/flipping-the-classroom/
Emine Cabi, (2018). The Impact of the Flipped Classroom Model on Students'
Academic Achievement. International Review of Research in Open and Distributed
Learning 19, 3.
Chang, Y. H., Song, A. C., & Fang, R. J. (2018a). The study of programming language
learning by applying flipped classroom. In 2018 1st IEEE International Conference on
Knowledge Innovation and Invention (ICKII) (pp. 286-289). IEEE.
Chang, Y., Song, A., Fang, & R. (2018b). Integrating ARCS Model of Motivation and
PBL in Flipped Classroom: a Case Study on a Programming Language. EURASIA
Journal of Mathematics, Science and Technology Education, 14(12), em1631
ISSN:1305-8223, 2018.
Chen, E. (2014). Flipped Classroom Model and Its Implementation in a Computer
Programming Course. Lrarlrdom, 2014. 2014:86.
Chen, K. S., Monrouxe, L., Lu, Y. H., Jenq, C. C., Chang, Y. J., & Chang, Y. C. (2018).
Academic outcomes of flipped classroom learning: A meta-analysis. Medical Education,
52(9), 910–924.
Cheng, L., Ritzhaupt, A.D., Antonenko, P. (2019). Effects of the flipped classroom
instructional strategy on students’ learning outcomes: a meta‐analysis. Education
Technology Research Development, 67:793–824
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.
Davies, R. S., Dean, D. L., & Ball, N. (2013). Flipping the classroom and instructional
technology integration in a college-level information systems spreadsheet course.
Educational Technology Research and Development, 61(4), 563–580. doi:10.1007/
s11423-013-9305-6
Durak, H.Y. (2020). Modeling Different Variables in Learning Basic Concepts of
Programming in Flipped Classrooms. Journal of Educational Computing Research
2020, Vol. 58(1) 160–199
Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot–based method of
testing and adjusting for publication Bias in meta-analysis. Biometrics 56, 455–463.
Elmaleh, J., & Shankararaman, V. (2017). Improving student learning in an introductory
programming course using flipped classroom and competency framework. Proceedings
of 2017 IEEE Global Engineering Education Conference, 49-55, June 7.
https://doi.org/10.1109/EDUCON.2017.7942823
Hattie, J. (2012). Visible learning for teachers: Maximising impact on learning. London
and New York: Routledge.
Hayashi, Y., Ken-Ichi, F., & Komatsugawa, H. (2015). Collaborative Learning in
Computer Programming Courses That Adopted the Flipped Classroom, International

Almassri & Zaharudin 287
International Journal of Instruction, April 2023 ● Vol.16, No.2
Conference on Learning and Teaching in Computing and Engineering (Latice 2015),
pp 209-212.
Hew, K. F., & Lo, C. K. (2018). Flipped classroom improves student learning in health
professions education: A meta-analysis. BMC Medical Education, 18(1), 1–12.
Higgins, J.P.T., & Thompson, S.G. (2002). Quantifying heterogeneity in a meta-
analysis. Statistics in Medicine, 21, 1539–1558.
Holdhusen, M. H. (2015). A “flipped” statics classroom. Paper presented at 2015 ASEE
annual conference & exposition, Seattle, WA.
Hsu, W.C., & Lin, H.C.K. (2016). Impact of Applying WebGL Technology to Develop
a Web Digital Game-Based Learning System for Computer Programming Course in
Flipped Class- room,” Proc. - 5th International Conference of Education Innovation
through Technology. EITT 2016, no. 2004, pp. 64– 69, 2017.
Indi, T.S., (2016). An Experience Report of Flipped Classroom Strategy Implementation
for Java Programming Course. IEEE 8th International Conference on Technology for
Education. 978-1-5090-6115-0/16
Jonsson, H. (2015). Using Flipped Classroom, Peer Discussion, and Just-in-time
Teaching to Increase Learning in a Programming Course. 2015, IEEE Frontiers in
Education Conference (FIE), in IEEE Frontiers in Education Conference (FIE), El
Paso, Oct. 2015.
Karabulut-Ilgu, A., Jaramillo Cherrez, N., & Jahren, C. T. (2018). A systematic review
of research on the flipped learning method in engineering education. British Journal of
Educational Technology, 49(3), 398–411.
Karaca, C., & Ocak, M.A. (2017). effects of Flipped Learning on University Students’
Academic Achievement in Algorithms and Programming Education. International
Online Journal of Educational Sciences, 2017, 9 (2), 527 - 543
Kazanidis, I., Pellas, N., Fotaris, P., & Tsinakos, A. (2019). Can the flipped classroom
model improve students’ academic performance and training satisfaction in Higher
Education instructional media design courses? British Journal of Educational
Technology, 50, 4.
Lage, M.J., Platt, G.J. & Treglia, M. (2000). Inverting the Classroom: A Gateway to
Creating an Inclusive Learning Environment. The Journal of Economic Education. 31, 1
30–43.
Lin, Y (2019). Impacts of a flipped classroom with a smart learning diagnosis system on
students' learning performance, perception, and problem solving ability in a software
engineering course. Computers in Human Behaviour 95, 187–196.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA:
Sage.

288 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
Lo, C. K., Hew, K. F., & Chen, G. (2017). Toward a set of design principles for
mathematics flipped classrooms: A synthesis of research in mathematics education.
Educational Research Review, 22, 50–73.
Loftsson, H., & Matthiasdottir, A., 2021. Using flipped classroom and team-based
learning in a first-semester programming course: An experience REPORT.
Lpez-Lpez, J. A., Page, M. J., Lipsey, M. W., & Higgins, J. P. (2018). Dealing with
effect size multiplicity in systematic reviews and meta-analyses. Research synthesis
methods, 9, 336–351.
Marasco, E. A., Moshirpour, M., & Moussavi, M. (2017). Flipping the foundation: A
multi-year flipped classroom study for a large-scale introductory programming course.
The 2017 ASEE Annual Conference & Exposition, Paper ID #20170, June 25-28.
Mithun, S., & Evans, N. (2018). Impact of the flipped classroom on students' learning
and retention in teaching programming. In 2018 ASEE Annual Conference &
Exposition.
McCord , R., & Jeldes, I. (2019): Engaging non-majors in MATLAB programming
through a flipped classroom approach, Computer Science Education, DOI:
10.1080/08993408.2019.1599645
Nikolic, S., Ros, M., & Hastie, D.B. (2018): Teaching programming in common first
year engineering: discipline insights applying a flipped learning problem-solving
approach, Australasian Journal of Engineering Education, DOI:
10.1080/22054952.2018.1507243
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of
Educational Statistics, 8(2), 157–159. Cheng, L., Ritzhaupt, A.D., Antonenko, P.
(2019). Effects of the flipped classroom instructional strategy on students’ learning
outcomes: a meta‐analysis. Education Technology Research Development, 67:793–824.
Paez, N.M. (2017). A Flipped Classroom Experience Teaching Software Engineering.
IEEE/ACM 1st International Workshop on Software Engineering Curricula for
Millennials (SECM). 978-1-5386-2795-2/17.
Pattanaphanchai, J. (2019). An Investigation of Students’ Learning Achievement and
Perception using Flipped Classroom in an Introductory Programming course: A Case
Study of Thailand Higher Education, Journal of University Teaching & Learning
Practice, 16(5), Available at:https://ro.uow.edu.au/jutlp/vol16/iss5/4
Patrick, S., (2016). Switching to Blend-Ed: Effects of Replacing the Textbook with the
Browser in an Introductory Computer Programming Course.
Peethambaran, M. K. P., Renumol, V. G., Murthy, S. (2018). Flipped Classroom
Strategy to Help Underachievers in Java Programming,” Proc. - 6th Int. Conf. Learn.
Teach. Comput. Eng. LaTiCE 2018, pp. 44–49.

Almassri & Zaharudin 289
International Journal of Instruction, April 2023 ● Vol.16, No.2
Puarungroj, W (2015). Inverting a Computer Programming Class with the Flipped
Classroom. The Twelfth International Conference on eLearning for Knowledge-Based
Society, 11-12 December 2015, Thailand.
Ruiz de Miras, J., Balsas‐Almagro, J. R., & García‐Fernández, Á. L. (2022). Using
flipped classroom and peer instruction methodologies to improve introductory computer
programming courses. Computer Applications in Engineering Education, 30(1), 133-
145.
Scammacca, N., Roberts, G., & Stuebing, K. K. (2014). Meta-analysis with complex
research designs: Dealing with dependence from multiple measures and multiple group
comparisons. Review of educational research, 84(3), 328–364.
Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher
education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565.
Siti Rosminah, M.D., & Ahmad Zamzuri, M.A. (2012). Difficulties in learning
Programming: Views of students. in 1st International Conference on Current Issues in
Education (ICCIE2012).
Sobral, S.R. (2021). Strategies on Teaching Introducing to Programming in Higher
Education. In: Rocha, Á., Adeli, H., Dzemyda, G., Moreira, F., Ramalho Correia, A.M.
(eds) Trends and Applications in Information Systems and Technologies. WorldCIST
2021. Advances in Intelligent Systems and Computing, vol 1367. Springer, Cham.
https://doi.org/10.1007/978-3-030-72660-7_14
Spanjers, I. A. E., Knings, K. D., Leppink, J., Verstegen, D. M. L., de Jong, N.,
Czabanowska, K., et al. (2015). The promised land of blended learning: Quizzes as a
moderator. Educational Research Review, 15, 59–74.
Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis:
Guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055.
Taşpolat, A., Özdamli, F., & Soykan, E. (2021). Programming language training with
the flipped classroom model. Sage Open, 11(2), 21582440211021403.
Threekunprapa, A., & Yasri, P. (2020). Unplugged Coding Using Flowblocks for
Promoting Computational Thinking and Programming among Secondary School
Students. International Journal of Instruction, 13(3), 207-222
Torun, F., & Dargut, T. (2015). Mobil renme ortamlarında ters yz sınıf modelinin
gerekleştirilebilirlii zerine bir neri [A rec- ommendation on the feasibility of the
flipped classroom model in mobile learning environments]. Adnan Menderes
niversitesi Eitim Fakltesi Eitim Bilimleri Dergisi, 6(2), 20–29.
Turan, Z., & Gktaş, Y. (2015). A new approach in higher education: Students’ views
on the flipped class- room method]. Yksekretim ve Bilim Dergisi, 5(2), 156–164.

290 Effectiveness of Flipped Classroom Pedagogy in …
International Journal of Instruction, April 2023 ● Vol.16, No.2
van Alten D.C.D, Phielix, C., Janssen, J., & Kester, L. (2020). Effects of self-regulated
learning prompts in a flipped history classroom, Computers in Human Behavior,
Volume 108, 106318.
Wang, G., Zhao, H, Guo, Y., & Li, M. (2019). Integration of Flipped Classroom and
Problem Based Learning Model and its Implementation in University Programming
Course. The 14th International Conference on Computer Science & Education (ICCSE
2019), August 19-21, 2019. Toronto, Canada
Williams, B. (2013). How I flipped my classroom. In NNNC Conference.
Yavuz, M. (2016). The effect of flipped classroom applications on academic
achievement and examination of student experiences at secondary education level
[Yksek Lisans Tezi]. Atatrk niversitesi.
Yilmas, R.M., & Baydas, O., (2017). An examination of undergraduates’ metacognitive
strategies in pre-class asynchronous activity in a flipped classroom. Education
Technology Research Development, 65:1547–1567
Zhuo, L., & Qi-Xian, G. (2015). The Application of Hybrid Flipped Classroom in the
Course of Java Programming. 7th International Conference on Information Technology
in Medicine and Education. 978-1-4673-8302-8/15