Current Software Defect Prediction: A Systematic Review

Full text

117
978-1-7281-8428-9/22/$31.00 ©2022 IEEE
Current Software Defect Prediction: A Systematic
Review
Yahaya Zakariyau Bala
Software Engineering
Universiti Putra Malaysia
Selangor, Malaysia
gs61002@student.upm.my
Pathiah Abdul Samat
Software Engineering
Universiti Putra Malaysia
Selangor, Malaysia
pathiah@upm.edu.my
Khaironi Yatim Sharif
Software Engineering
Universiti Putra Malaysia
Selangor, Malaysia
khaironi@upm.edu.my
Noridayu Manshor
Software Engineering
Universiti Putra Malaysia
Selangor, Malaysia
ayu@upm.edu.my
Abstract— detecting defect in a software product prior to
testing reduces the cost of testing and improve the quality of the
software product. Various methods for enhancing the accuracy
of defect prediction model have been published. The goal of this
review is to identify and analyze the dataset, models, framework
and the performance of software defect prediction model. The
IEEExplore, Science Direct, Scopus, and Google Scholar
databases were used to search and download the relevant
papers. Sixty-eight (68) papers published between 2017 to 2021
were selected based on exclusion and inclusion criteria. Analysis
of the selected studies revealed that 100% of the selected articles
used the publicly available dataset from NASA, PROMISE and
others. The most frequently used models in software defect
prediction studies were identified. The analysis also revealed
IEEE Transactions on Software Engineering Journal is the most
significant journal with respect to software defect prediction
studies using Scimago Journal Ranking as criteria. The review
also identified studies on enhancing the predictive performance
of defect prediction models. Software defect prediction is still
active and sound. Thus, needs more research especially on
enhancement methods.
Keywords—Software Defect Prediction, Naïve Bayes, Support
Vector Machine, Software
I. INTRODUCTION
Identification of the defect in software product in the early
phase of the software development life cycle is very
important. Using Software defect prediction (SDP) for
identification and removal of software defect contributes
greatly in producing cost-effective and qualitative software
products. This is due to the fact that, the software defect
prediction process is less expensive than software testing and
reviews. As reported in some studies, software defect models
outperform currently software reviews methods utilized by
industries [1]. Furthermore, SDP help software testing teams
to focus on the defect prone component. Thus, minimizing
the effort of testing and enhancing the quality of software [2],
[3]. For these reasons, the SDP has become an important and
attractive research topic in software engineering domain.
Most of the existing review on SDP studies have not capture
the comprehensive image of current approaches such as
ensemble techniques and frameworks proposed in studies on
software defect prediction. Therefore, in this review work,
the datasets, models used, ensemble and frameworks
employed in various studies on software defect prediction
models between 2017 and 2021 were identified and analyzed.
This systematic review is organized as follows; the
methodology is explained in section II, the result of the
review is presented in section III and IV. Finally, the
conclusion in section V.
A. Research Questions
The research questions addressed by this work are;
RQ1. Which of the journals is the most significant in the SDP
area?
RQ2. Who are the most influential and active researchers in
SDP?
RQ3. What are research topics in SDP area?
RQ4. What are the datasets mostly used in SDP?
RQ5. What are the models used for SDP?
RQ6. What are the most commonly used models for SDP?
RQ7. What are the best performing models for SDP?
RQ8. What are the improvement methods proposed for SDP?
RQ9. What are the frameworks proposed for SDP?
RQ10. What are the feature selection methods proposed in
SDP?
RQ11. What are the ensemble methods proposed in SDP?
II. METHOD
The work followed the guideline proposed by [4]. as
shown in Fig.1. The planning stage includes; identifying
reasons for the review, the formation of research questions,
designing and following the suitable method for searching
relevant materials, extracting and synthesizing of information
for performing the review and as well as evaluation. The next
step is performing the review where the searching method,
extraction and data synthesis are implemented.
2022 Applied Informatics International Conference (AiIC) | 978-1-7281-8428-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/AiIC54368.2022.9914586
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on November 01,2022 at 00:01:19 UTC from IEEE Xplore. Restrictions apply.

118
Fig. 1. Systematics Literature review process
A. Search process
The search process begins by defining search strings,
selecting databases and finally stating the inclusion and
exclusion criteria. Alternative spellings, synonyms and
Boolean ANDs and ORs were considered for constructing
search strings. Digital libraries such as IEEExplore, Science
Direct, Scopus and Google Scholar were searched.
B. Inclusion and Exclusion criteria
Peer-reviewed articles SDP published between 2017 and
2021 were included:
Inclusion Criteria;
Empirical studies on SDP, study comparing the performance
of models in SDP, Studies that used feature selection
technique in SDP, Studies that used ensemble methods for
SDP were all considered
Exclusion Criteria
Studies without experiment or empirical analysis and studies
in any language other than English
C. Assessment of the Quality
Each article was evaluated based on stated assessment of
quality questions as shown in Table II. The articles were
score as follows; for each question, the score of 1 was given
to Yes, 0.5 to Partial and 0 to No.
D. Collection of Data
The extracted data from each article were:
● Authors
● Titles
● Publishers
● Independent variable
● Datasets
● Evaluation metrics
● Models
E. Data analysis
MS excel was used to tabulate the data extracted from each
article to show;
● Number of articles published per year
● Number of articles published per author
● Number of articles per model
● Quality score for each article
TABLE I. Q
UALITY
A
SSESSMENT
S/N Question Yes Partially NO
1 Are the study’s objectives clearly
stated?
2 Is the study scope clearly stated?
3 Are there relevant literature in the
study?
4 Are the dependent variable(metrics)
clearly stated
5 Are the independent variable clearly
stated?
6 Are the dataset used clearly presented?
7 Has the study been cited?
8 Are the performance evaluation
techniques properly written?
III. R
ESULTS
A. Search result
A total of 2609 articles were identified initially, after
applying exclusion criteria, some based on the title, some
based on abstract and finally based on content and removal
of duplicate 68 articles were considered.
B. Quality Evaluation of Selected Articles
As shown in Table I, articles scored 5 or above were
selected otherwise, rejected. A nomenclature was assigned to
each study i.e., PS (Primary Study). After quality assessment,
68 studies were considered and named as primary studies.
IV. D
ISCUSSION
Response to research questions is discus in this section.
A. Significant Journal Publications
Sixty-eight (68) studies on software defect prediction are
considered in this review. Fig. 2 shows the distribution of
selected studies per year. As can be seen from the Fig. 2, most
of the studies were published in 2020. Fig. 2 also shows that,
the relevancy of research on SDP is sound and active till
today. To address the research question RQ1: the most
significant journal in selected primary studies, Scimago
ranking metrics such as SJR value and Quartiles categories
from Q1 to Q4 were used to find the most significant as
shown in Table II. IEEE Transactions on Software
Engineering emerge the most significant journal.
Fig. 2. Distribution of study per year
5 2 14
29
15 1
2017 2018 2019 2020 2021 2022
Total
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on November 01,2022 at 00:01:19 UTC from IEEE Xplore. Restrictions apply.

119
B. Most Active and Influential Researchers
The researchers in the area of SDP who contributed a lot
in the selected papers were identified based on the number of
studies they appeared as first Autor. Balogun A.O.
contributed as the first Autor in four studies followed by Yu
Q and Zhang N who contributed in three (3) studies and as
the first Autor in two (2) studies each. Shown in Fig. 3.
Fig. 4. Distribution of the number of studies per model
C. Research Topics in the Area of Software Defect
Prediction
Analysis of selected studies showed that recent research in
software defect prediction concentrate on four topics:
• Just-in-time SDP: The major aim of this topic is the
identification of defect inducing changes.
• Defect analysis: This topic involves classifying
defect into different categories and identifying the possible
causes of the defect so as to prevent them from reoccurring.
• Cross Version Defect Prediction (CVDP): in this
topic, defect labels of the current version of software is
predicted using the classification algorithm trained on the
historical dataset of the previous version of the same
software.
• Cross-project defect prediction (CPDP): major aim
of this topic is to predict the defect labels of one project is
predicted using classification model trained on the historical
data of another project.
D. Datasets Used for SDP Studies
A dataset is a group of data. Dataset is used for training
and evaluating the SDP model. The major component in SDP
study is dataset. Public datasets are obtained from NASA,
PROMISE, AEEEM, Relink, SOFTLAB, etc. Most of the
selected studies used the combination of multiple datasets for
analysis.
Fig. 3. Distribution of Influential Researchers and Number of Studies
TABLE II.
RANKING
O
F
S
ELECTED
J
OURNALS
B
ASED
O
N
S
CIMAGO
J
OURNAL
R
ANKING
(SJR)
Journal Publication SJR Quartile
IEEE Transactions on Soft. Eng. 857 Software (Q1)
Journal of King Saud University
– Comp. and Infor. Sci.
617 Comp. Sci. (miscellaneous)
(Q1)
Information and Software
Technology
606 Software (Q2)
International Journal of
Computational Intelligence
Systems
385 Computer Science
(miscellaneous) (Q2)
Symmetry 385 Computer Science
(miscellaneous) (Q2)
IET Software 305 Computer Graphics and
Computer-Aided Design (Q3)
Journal of Systems Engineering
and Electronics
301 Computer Science
Applications (Q3)
International Journal of
Electrical and Computer
Engineering
277 Computer Science
(miscellaneous) (Q2)
E. Methods(Models) Used in SDP Studies
As shown in Fig. 4, the most useful models are NB and
SVM.
F. Most Frequently Used Methods(Models) in SDP Studies
From the used methods in software defect prediction in
Section E, six most frequently used models in SDP were
identified based on the number of studies they were employed
in. Naïve Bayes (NB) and Support Vector Machine (SVM)
emerged as the most useful methods followed by
Convolutional Neural Network (CNN).
G. Best Performing Models for SDP
Two out of selected studies reported RF and SVM as better
performing models such studies are; [5] and [6].
H. Proposed Improvements Method for SDP models
In order to improve the SDP models’ predictive
performance, researchers focus on improving the quality of
0
2
4
6
8
ANFIS
ANN
C4.5
CNN
DT
EML
GA
J48
KNN
LDA
LR
LSTM
MLP
NB
NN
RF
RIPPER
RNN
Number of Studies
4 3 34 2 2
BALOGUN A.O., YU Q., ZHANG N.,
Active Researchers in
SDP
Number of Studies First Autor
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on November 01,2022 at 00:01:19 UTC from IEEE Xplore. Restrictions apply.

120
training and testing datasets (using feature selection
techniques), optimizing classifiers (ensemble techniques) or
the combination of both.
1) Feature Selection
Different researchers concentrated their studies toward
improving SDP models through feature selection techniques
such as studies performed by [7], [8], [9], [10], [11], [12] and
[13]
2) Ensemble Techniques
On the other hand, some studies concentrate on using the
ensemble method to improve software defect prediction.
Ensemble method is a procedure in which multiple classifiers
are combined together in order to achieve better performance.
In addition to improving classifiers’ accuracy, ensemble
method has been used for mitigating the issue of class
imbalance in software defect prediction. Examples of such
studies are [14], [15], [16], [17] and [18].
I. Proposed Frameworks for Software Defect Prediction
Various studies proposed different framework to improve
SDP model. Below are the frameworks proposed for software
defect prediction.
[19] proposed a framework which consists three layers;
the first layer is the pe-processing layer where missing values
and outliers in the dataset are handled, and dataset are splited
in to training and testing data. The classification through four
different back-propagation techniques such as BR, LM, SCG
and BFGS-QN was done in the second layer. The third layer
includes fuzzy technique where the bet training algorithms
selected based on the performance and stored in the cloud so
that the development team can access the model anywhere.
Thus, reducing the cost of testing. The framework was
evaluated on NAA dataset using various metrics like Recall,
Precision, Accuracy, F-measure, AUC, R2 and MSE. It was
observed that the BR training function outperformed the
other training functions.
[20] proposed an integrated machine learning framework
for Infrastructure as a Code (IaC) defect prediction which
allows crawling of the repository, extraction of metrics,
building model and evaluation. Five machine learning
algorithms and 104 projects were used to evaluate the
performance of the framework in identifying defective IaC
scripts. The result of the experiment shows that Random
Forest outperformed the other classification algorithm
especially for AUC and MCC evaluation metrics.
[21] proposed a framework to solve the issue of class
imbalance SDP models. In this approach, a balance between
the defect and non-defect instances in the imbalance dataset
are created using the distribution properties of the dataset.
Experiment result shows that, the proposed framework
outperformed SMOTE and K-Means SOMTE.
[22] to improve the performance of SDP, this study
proposed a framework in which the variant selection activity
was used to identify the best optimized version of
classification techniques was combined with feature selection
technique. Based on evaluation, the framework outperformed
some commonly used classification techniques.
[23] proposed a framework which improve the
performance of Extreme Machine Learning (EML). The
framework comprises three components; component 1:
KPCA leverages a nonlinear mapping function to extract
optimal representative features. Component 2: leverages
adaptive genetic algorithm to enhance the prediction
performance of EML. Component 3: the multiple EML
optimized by the adaptive genetic algorithm re integrated by
Adaboost algorithm. The framework was evaluated on eleven
open-source datasets and compared with some machine
learning techniques, including EML and three other variants.
The result shows that, the proposed framework outperformed
the base line models.
[24] proposed a framework which consist of three stages;
stage1: the influence of the minority class in the dataset is
minimized using Sampling with Minority Method (SWIM)
based on Mahalanobis distance technique. Stage2: the most
relevant representative feature between the training and the
testing dataset was selected. Stage3: performs the transfer
learning from the training to the testing dataset in the
Grossmann manifold. The experimental result shows that, the
proposed framework is superior in terms of AUC.
[25] proposed a framework which identify defect using
features extracted from the commit automatically when
messages and code changes. Experiment result shows that,
the approach achieves 10.36 to 11.02% improvement over the
state-of-the-art methods.
J. Limitations of this study
Only open access papers were downloaded out of which
68 studies were selected for this review and as such, there
might have been an exclusion of some relevant software
defect prediction studies from journals or conference
proceedings.
V. CONCLUSION
The aim of systematic review was to identify and analyze
current models used, datasets, best performing models,
topics, improvement methods proposed and frameworks in
SDP. Analysis performed on the selected primary studies
revealed that, 100% of the selected papers utilized public
datasets for analysis. RF and SVM performed better than
most of the classification algorithms in SDP. The feature
selection techniques play a vital role in improving the quality
of the dataset used for SDP likewise, ensemble techniques
have been used for enhancing the predictive performance of
SDP models especially weak classifiers.
ACKNOWLEDGMENT
This work has been supported by Universiti Putra
Malaysia and TedFund Nigeria
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on November 01,2022 at 00:01:19 UTC from IEEE Xplore. Restrictions apply.

121
TABLE III. LIST OF SELECTED PRIMARY STUDIES
REFERENCES
[1] T.Menzies, Z.Milton, B.Turhan, B.Cukic, Y.Jiang, And A.Bener,.
“Defect Prediction From Static Code Features: Current Results,
Limitations, New Approaches,” Automated Soft. Eng., 17(4), 375-407,
2010
[2] C.Catal, “Software fault prediction: A literature review and current
trends,” Expert systems with applications, 38(4), 4626-4636, 2011.
[3] T.Hall and D.Bowes, “The state of machine learning methodology in
software fault prediction,” In 2012 11th international conference on
machine learning and applications (Vol. 2, pp. 308-313). IEEE,
December, 2012.
[4] B.Kitchenham, O.P.Brereton, D.Budgen, M.Turner, J.Bailey and
S.Linkman, “Systematic literature reviews in software eng.–a
systematic literature review,” Information and soft. Technology, 51(1),
7-15, 2009.
[5] Iqbal, A., Aftab, S., Ali, U., Nawaz, Z., Sana, L., Ahmad,M., and
Husen, A. (2019). “Performance analysis of machine learning
techniques on software defect prediction using NASA datasets. Int. J.
Adv. Comp. Sci. Appl, 10(5), 300-308.
[6] B.Khan, R.Naseem, M. A.Shah, K.Wakil, A.Khan, M. I.Uddin, and
M.Mahmoud, “Software defect prediction for healthcare big data: An
empirical evaluation of machine learning techniques,” Journal of
Healthcare Eng., 2021.
[7] K.Bashir, T.Li and M.Yahaya, “A novel feature selection method based
on maximum likelihood logistic regression for imbalanced learning in
software defect prediction,” Int. Arab J. Inf. Technol., 17(5), 721-730,
2020.
[8] J.Lin, and L.Lu, “Semantic Feature Learning via Dual Sequences for
Defect Prediction,” IEEE Access, 9, 13112-13124, 2021.
[9] A. O.Balogun, S.Basri, S.Mahamad, S. J.Abdulkadir, L. F.Capretz, A.
A.Imam, and G.Kumar, “Empirical Analysis of Rank Aggregation-
Based Multi-Filter Feature Selection Methods in Software Defect
Prediction,” Electronics, 10(2), 179, 2021.
[10] M.Mustaqeem and M.Saqib, “Principal component-based support
vector machine (PC-SVM): a hybrid technique for software defect
detection,” Cluster Computing, 1-15, 2021.
[11] X. U.Xiaolong, C. H. E. N.Wen, and W. A. N. G.Xinheng,"RFC: a
feature selection algorithm for software defect prediction,” Journal of
Systems Eng. and Electronics, 32(2), 389-398, 2021.
[12] A. O.Balogun, S.Basri, L. F.Capretz, S.Mahamad, A. A.Imam, M.
A.Almomani, and G.Kumar, “An adaptive rank aggregation-based
ensemble multi-filter feature selection method in software defect
prediction,” Entropy, 23(10), 1274, 2021.
[13] B.Mumtaz, S.Kanwal, S.Alamri, and F.Khan, “Feature Selection Using
Artificial Immune Network: An Approach for Software Defect
Prediction,” INTELLIGENT AUTOMATION AND SOFT
COMPUTING, 29(3), 669-684, 2021.
[14] S. S. Maddipati and M.Srinivas, “A Hybrid Approach for Cost
Effective Prediction of Software Defects,” International Journal of
Adv. Comp. Sci. and App. 12(2), 145 – 152, 2021.
[15] H.Alsawalqah, N.Hijazi, M.Eshtay, H.Faris, A.Radaideh, I.Aljarah,
and Y.Alshamaileh, “Software defect prediction using heterogeneous
ensemble classification based on segmented patterns,” Applied Sci.,
10(5), 1745, 2020.
[16] T. T.Khuat, and M. H.Le, “Ensemble learning for software fault
prediction problem with imbalanced data,” International Journal of
Electrical and Comp. Eng., 9(4), 3241, 2019.
[17] H.He, X.Zhang, Q.Wang, J.Ren, J.Liu, X.Zhao, and Y. Cheng,
“Ensemble multiboost based on ripper classifier for prediction of
imbalanced soft. defect data,” IEEE Access, 7, 110333-110343, 2019.
[18] X.Yang, D.Lo, X.Xia and J.Sun, “TLEL: A two-layer ensemble
learning approach for just-in-time defect prediction,” Information and
Soft.Technology, 87, 206-220, 2017.
[19] M. S.Daoud, S.Aftab, M.Ahmad, , M. A.Khan, A.Iqbal, S.Abbas, and
B.Ihnaini, “Machine Learning Empowered Soft,” Defect Prediction
System, 2022.
[20] S.Dalla Palma, D.Di Nucci, F.Palomba, and D. A. Tamburri, “ Within-
project defect prediction of infrastructure-as-code using product and
process metrics,” IEEE Transactions on Soft. Eng., 1-1, 2021.
[21] K. K.Bejjanki, J.Gyani, and N.Gugulothu, “Class imbalance reduction
(CIR): a novel approach to software defect prediction in the presence
of class imbalance,” Symmetry, 12(3), 407, 2020.
[22] U.Ali, S.Aftab, A.Iqbal, Z.Nawaz, M. S.Bashir, and M. A. Saeed,
“Software Defect Prediction Using Variant based Ensemble Learning
and Feature Selection Techniques,” International Journal of Modern
Education & Comp. Sci., 12(5), 2020.
[23] N.Zhang, K.Zhu, S.Ying, and X.Wang, “KAEA: A novel three-stage
ensemble model for software defect prediction,” Computers, Materials
and Continua, 64(1), 471-499, 2020.
[24] K.Jiang, Y.Zhang, H.Wu, A.Wang, and Y.Iwahori, “Heterogeneous
defect prediction based on transfer learning to handle extreme
imbalance,” Applied Sci., 10(1), 396, 2020.
[25] T.Hoang, H. K.Dam, Y.Kamei, D.Lo, and N.Ubayashi, “DeepJIT: an
end-to-end deep learning framework for just-in-time defect
prediction,” In 2019 IEEE/ACM 16th International Conference on
Mining Soft. Repositories (MSR) (pp. 34-45). IEEE, May, 2019.
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on November 01,2022 at 00:01:19 UTC from IEEE Xplore. Restrictions apply.