Artificial Bee Colony Algorithm for t-Way Test Suite Generation

Full text

WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa
WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa,(((
Abstract— Exhaustive testing is a very strenuous undertaking
due to its numerous combinations. Consequently, searching for and
sampling an optimal test suite from viable groups of test cases (TC) has
turned out to be a central concern. To tackle this matter, the use of t-
way testing (Where t represents the strength of the interaction) has be-
come well know n. In order to facil itate future development and
summarize the realization so far, the major objective of this paper is to
portray an important comparison of the introduced optimization
algorithms (OA) as a base of the t-way test suite generation strategy,
and to suggest a new strategy focusing on the Artificial Bee Colony
(ABC) Algorithm, known as Artificial Bee Colony Strategy (ABCS).
The experimental results showed that ABCS can compete against the
existing (both AI-based and computational-based) strategies in terms of
generating the optimum test case.
Keywords—software testing, t-way testing, artificial bee colony, meta-
heuristics, optimization problem
I.
I
NTRODUCTION
Performing everyday chores such as operating smart phones,
driving, washing, laundry, watching TV and all other activities
requires using software. With the progress of technology
nowadays, software is not only installed on computers in the
present time, but also installed in devices like mobile phones and
other electronic gadgets at homes due to the improvement of
technology. To ascertain the acceptability of quality, the software
requires examination before delivery. All likely combinations of
the configurations require examination to note interaction issues for
an extremely configurable software. Moreover, the resource
restraints and time-to-market are considered when carrying out an
exhaustive examination of the configured combination, which is
inefficient and impractical owing to the combinatorial disruption
problem. For instance, let us consider a system that has 20 inputs
in which each input has the possibilities of 2 values. In order to
carry out exhaustive combinatorial testing 2
20
(1, 048, 576) test
cases need to be executed, which is clearly impracticable within a
restricted time limit.
For this reason, various t-way interaction-testing strategies for
the past 20 years have been propounded in works of literature. All
the designed strategies assist in building test suites with minimal
test cases to achieve all the interaction strength intended. These
strategies [1-8]are: “GTWay”[1], “Automatic Efficient Test
Generator (AETG)” [2], “
Bat-inspired pairwise testing strategy
(BPTS)” [3],
”Bat-Testing Strategy (BTS)” [4-6], “In Parameter
Order (IPO)” [7], “Harmony Search Strategy (HSS)”[8] and
“Jenny” [9] and they have generated test cases for a uniform
strength t-way (that is, all the parameters have interacted uniformly
together). In a few cases, researchers have described the necessity
for producing a combination of inputs with varying t-way strength.
Due to this, the following strategies have recently been introduced;
“Simulated Annealing (SA)” [10], “Ant Colony System (ACS)”
[11], “Particle Swam Optimization (PSO)” [12] and “Harmony
Search Strategy (HSS)” [8]. The rise of Artificial Intelligence (AI)
based strategies were depicted to have good outcomes regarding
the production of the optimal test case size. Depending on such
enticing point of views, the Artificial Bee Colony Algorithm has
been studied as the elementary skill for t-way test generation.
Introducing other substitute optimization algorithms except for
ACS, SA, HSS, and PSO might lead to new views in generating t-
way test. The implementation and outcomes of our experiment
have been discussed within this paper related to other Artificial
Intelligence based t-way strategies.
The remaining of the paper is organized in the following
manner. Section 2 discuss the theoretical background of t-way
strategies. Section 3 presents the related works, Section 4
introduces the proposed algorithm, while the 5th section discusses
the benchmarking result and finally, the conclusion and future
work.
II.
THEORETICAL BACKGROUND OF T-WAY STRATEGIES
All systems are composed of a series of components that
relates together (known as parameters with values associated with
them). In order to demonstrate this idea of t-way interaction testing,
the instance of the Find Dialog Box as portrayed in Fig. 1 should
be considered.
All the matching records were returned by the dialog box via
the acceptance of an input text string. In this instance, the dialog
box constituted four inputs (that were, case sensitivity, Text string,
Match whole word, and search direction). By neglecting the
improper values for the sake of simplicity, the following were the
values for each parameter: “Find what” text string:
uppercase/lowercase /mixed, “Match whole words”: match/do not
match, “Match case”: match /do not match and “Direction”: Up /
Down. Assuming the interaction strength t =2. The covering array
is represented as MCA (N; 2, 2
3
3
1
). The exhaustive test for the
dialog box is 3x2x2x2=24 test cases were re-quired to test the
system exhaustively.
Arti cial Bee Colony Algorithm for
t-Way Test Suite Generation
Ammar K Alazzawi1*, Helmi Md Rais2 andShuib Basri3
1,2,3 Department of Computer and Information Sciences, Universiti Teknologi PETRONAS,
32610 Seri Iskandar, Perak Darul Ridzuan, Malaysia
1ammar_16000020@utp.edu.my, 2helmim@utp.edu.my, 3shuib_basri@utp.edu.my

WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa
By implementing a 2-way testing method, where there were 30
pairs that require to be dealt with, the initial test case could cover
6 pairs by any greedy strategy (e.g, ABCS). As a candid issue, it
can be demonstrated that all 30 pairs can be dealt with using six
test cases.
Fig.1. Example of the Find Dialog Box.
A fascinating attribute of a t-way test is that the t-columns
comprise of every possible column's combination, which happens
somewhere in some manner. Fig. 2 shows the six test cases that
covered the 30 Pairs and each test case was characterized by
specific colors to depict each test case and the pairs that covered it.
Fig.2. 2-way test suite generation for the Find Dialog example.
III.
EXISTING T-WAY TESTING STRATEGIES AND TOOLS
This current section is aimed to give a summary of existing
studies for generating a test suite of the t-way. The existing t-way
approaches could be grouped into a major pair of classes:
computational approaches and algebraic approaches [13]. Test data
sets are structured in the Algebraic Approaches, without the
enumeration of all combinations. Thus, the records related to
algebraic approaches tend to be lightweight. On the other hand, the
approaches mentioned above are usually restricted to small
configurations. The strategies that adopt algebraic methods include
MCA, TConfig, CA and orthogonal Latin squares (OLS). Mandl
[14] used an orthogonal test design to examine a compiler.
Williams et al., [15] invented TConfig, which uses a repetitive
process of structuring centered on orthogonal arrays to build test
suites. The usability of the orthogonal array is limited to small
configurations most of the time, though it is advantageous.
To construct test cases in computational approaches, a greedy
algorithm is utilized. Every single step aims to cover as many
combinations as possible of the uncovered combinations.
Producing a test set is achieved by either implementing or using
OPAT (one parameter at-a-time strategy) or using the one-test-at-
a-time strategy (OTAT).
OTAT centered strategies construct an entire test case per
iteration and checks if the test case is an outstanding test case to
address the majority of the unaddressed interactions. Similar
processes recur until every one of the combinations is handled. In
the existing literature work, lots of tools and strategies have
emerged using OTAT (one-test-at-a-time) techniques. For
instance, AETG [2] builds every necessary interaction then
produces the last test case for all iterations in all cycles. AETG
techniques produce a series of test case candidates, and one is
avariciously chosen from these candidates as the last test case (that
is, covering most of the uncovered interactions). In many years, a
few series of variations of the AETG have appeared such as
mAETG SAT [16] and mAETG [17]. GTWay [18] also
implements the OTAT approach to produce the last test suite, just
like AETG. GTWay allows the implementation of real values of a
parameter as a significant string and follows the automated
accomplishment of the test cases, unlike AETG.
Jenny [9] implemented the OTAT approach by producing test
data to cover the whole one-way interaction. The initial phase test
data will then be extended by Jenny to greedily encompass thetwo-
way interactions. Optionally, this method can be a continual
process till the n
th
number of t-way interactions as stated by the
user. At almost the same period, Hartman invented the ITCH
(Intelligent Test Case Handler) [19]as an Eclipse Java plug-in tool.
ITCH depends on an exhaustive search to build the test suites for
t-way testing. ITCH’s accomplishment requires a long period, and
its outcome is not often optimal due to its exhaustive search
algorithm.
PSTG [20] is a meta-heuristic t-way strategy centered on
Particle Swarm Optimization that imitates the swarm behavior of
birds. Iteratively, the PSTG strategy is the ¿rst AI-based strategy
(PSO algorithm) which is able to support strengths up to t = 6. The
results of this strategy for t  3 are weaker than other AI-based
strategies, while it is faster. This strategy presents a new approach
to calculate the weight of test cases, which needs big data structures
to be stored for calculating the test cases. PSTG strategy supports
the VCA structure, which is called the VS-PSTG.
Complementary to the PSTG strategy, the HSS [8] is a meta-
heuristic strategy centered on the HSA (Harmony Search
Algorithm). The HSS imitates musicians attempting to produce a
nice music from available resources to make a unique tune from
random sampling or their archive. In order to achieve this, the HSS
uses the Harmony archive to keep unique solutions derived via a
series of improvisations within its global and local process of
search. In every improvisation, a test case would be chosen to be
the Last test suite until all the interactions required are covered.
HSS also tackles the support for constraint combinations (or
forbidden) unlike the PSTG.
Similar to the Particle Swarm Test Generator (PSTG) and the
Harmony Search (HSS) is the Bat-Inspired strategy [21, 22], which
has been executed by centering on the bat algorithm as the search

WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa
engine. The Bat-Inspired strategy implements global and local
searches to construct the candidate solution to be added to the last
suite until all the t-way interactions are covered.
Complementing the OTAT approach, is the One-parameter-at-
a-time (OPAT) approach and this starts by constructing an already
existing test suite for the initial two factors, or the components,
then extends it horizontally through adding a parameter for each
iteration, and the time is extended vertically till every parameter is
addressed sometime. The most recognized strategy in OPAT is the
in parameter-order (IPO) strategy [7]; it is a strategy that
implements the OPAT approach. The In parameter-order produce
a test set in a pair of the initial two parameters, and then elongates
the test set by producing the pair for the initial three parameters,
etc., till all the parameters of the system are addressed. This is
followed by a vertical extension to address the unaddressed
interactions if need be. The in parameter-order (IPO) strategy was
later generalized into a series of variants; IPOG [23], IPOG-D [23]
and CTE-XL [24]. Due to its simple nature, the in parameter-order
has been introduced by other research works, especially in the
construction development of [25]. Dissimilar to the in parameter-
order and its relatives, reliance in-between vertical and horizontal
extension is removed using the MIPOG in order to allow parallel
t-way test suite production.
IV.
THE PROPOSED ALGORITHM
The ABC algorithm was built to mimic the foraging behavior
of a colony of honeybees. A normal beehive is made up of three
important segments: unemployed /employed foragers (bees)
/source of food. The bees employed to exploit a specific source of
food are referred to as employed foragers. These honeybees carry
information relating to the profitability direction and distance of the
food source (test case) and connect the data with every single
honeybee in the hive. The unemployed honeybees are categorized
as either a scout honeybee or an onlooker bee. The later strive to
get a source for food with the use of information obtained by the
employed bee; while the onlookers haphazardly observe the
environment to locate a fresh source of food (better) [26, 27].
Supposedly, an employed forager bee that has a depleted food
source turns into a scout bee and begins to look out for another fresh
source of food. Moreover, it assumes the aggregate number of the
employed honeybees to be the same as the number of sources of
food. Imaginable, the position of a source for food depicts a
probable test case; though the size of a food source relates to the
quality (fitness) of all associated test cases.
Primarily, the ABC produces a haphazardly dispersed
population of test cases of the SN (location of food source) in the
exploration location, where SN represents the onlooker bee's
population or employed forager bees. Supposing the number of
optimization parameters is D, then for each of the test cases xi (i =
1, 2... SN) i basically would be a vector of D-dimensional vector.
Every result produced here could be attained by Eq.1 [24]:
ݔ
௜௝
= ݔ
௠௜௡ǡ௝
+ rand(0,1)(ݔ
௠௔௫ǡ௝
- ݔ
௠௜௡ǡ௝
) (1)
Here, ݔ
௠௔௫
and ݔ
௠௜௡
depict both upper and lower boundary
parameters for the test cases x
i
while in Rand [0, 1] and dimension
j (j=1, 2… D) there is a scaling factor representing a haphazard
integer in between [0, 1]. The positions of the source of food (D-
dimensional results) produced in the initial stage (C=0) are liable
to repeating C= (1, 2…, MCN) cycles, till a back out condition is
met. Both the local and the global probable selection/search are
executed in just a single cycle of ABC. Each cycle comprises a
series of responsibilities executed by various types of honeybee.
These processes are principally independent, and can be elucidated
in a distinct way as shown below for more clarification of the ABC
methodology:
Employed Bee Step
After the employed bee haves been allocated to their sources of
food, these honeybees assess the capability of their test cases s
(sources) and converse on the data with onlooker honeybees
Inclusively, every one of the employed bee produces a food
position candidate (test cases) by destabilizing the previous source
of food (test case) if (x
ij
) is in its archive, using the expression below
in Eq.2 [24]:
V
ij
= X
i, j
+ rand [-1, 1] (X
ij
–X
kj
) (2)
Here, k א{1,2,..., SN} (k  i) and jא{1,2,..., D} are the
haphazardly selected indexes, and Rand [+1,-1] is any number
between [1,-1], which works as a scaling factor. It is obvious that
as the optimal result in the search space is approached, this
distortion on a result becomes decreased. The capability of the
distorted (fresh) result will be assessed by the employed bee, and
in case the new food source has better quality values are set, the
new test cases take over the previous one in the archive of the
employed honeybee (a greedy-selection scheme).
Onlooker Bee Step
The main duty of an onlooker bee is to choose a test case (source
of food), based on the possible quantity related to the food source,
Pi, which is calculated by the following the mathematical
expression from Eq.3 [28]:
Pi=
௙௜௧
೔
σ௙௜௧
೙
ೞ೙
೙సభ
(3)
Where fit denotes the fitness value of a specific test case, and
the subscript depicts the number of test cases. This probable choice
is effected by relating Pi against a haphazardly selected number
between [0, 1]. A choice is allowed to be made if the produced
haphazard number is less than or equivalent to Pi if otherwise, it
would not be accepted. The duty of the onlooker honeybee to
specified test cases will be approved if the equivalent probable
selection is sanctioned. Normally, in the minimization issues, the
fitness value of test cases s is evaluated using the following
mathematical expression 3:
݂݅ݐ݊݁ݏݏ
௜
ൌ൝
ଵ
ଵା௙
೔
ǡ݂݅݂
௜ஹ଴

ͳ൅ȁ݂
௜
ȁǡ݂݂݅
௜ழ଴
 (4)
Where fi is the objective function value for the test cases. After
the selection of a source of food (test cases) with a Pi possibility,
the onlooker honeybee will select a fresh test case (food source) in
the area of the preceding one in its archive, by using Eq. 2. In case
the fresh test cases (food sources) have a better-quality value, then
an onlooker honeybee will update the fresh test cases (food

WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa
sources) inside its, and forgo the previous one, relative to the
employed bees.
Scout Bee Step
The duty of scout bees is to haphazardly search the whole
environment to obtain an improved (new) result to the overall
optimization issue. Unlike the situation with onlooker /employed
honeybees (where they are bound to create a trial results around a
previous result), the scout honeybees are not restricted in this
instance. The scout bees obtain samples from a wide set of vectors
with D-dimensions, in so far that they are within the boundaries in
the search environment (search space). Inside ABC, if a (non-
global) test case cannot be developed further after a pre-planned
series of cycles, then the test cases will be neglected, and the
employed bee allocated to that exact spot will be retransformed to
a scout bee with an essentially scout-type behavior. The value of
the pre-planned series of cycles, termed the limit, will however, be
a significant control parameter in the algorithm. The limit is
calculated when practiced through the mathematical expression
below.
limit = c.n
e
.D
Where n
e
represents the number of unemployed honeybees and,
c represents a coefficient constant with an acclaimed value between
1 or 0.5 [28, 29]. However, one scout bee must exist during the
implementation of ABC. The scout type processes give an
excellent ability to the ABC process in getting the paramount
global result, by creating stochastic inquiry in the whole D-
dimensional area. This is to say that scout bees will independently
search for a global optimal result, while all other types of bees
(onlooker /employed) are concurrently scrutinizing their confined
candidate test cases s for the overall best. For this reason, the
probability of being ensnared in the local optimum will never be
appropriate to the ABC. Figure .3. shows the flow chart of the
Artificial Bee Colony (ABC).
V. RESULTS
The main purpose of the experiments in this section is to
benchmark the proposed ABCS against existing AI-strategies and
tools, such as efficiency (i.e., produced a final test list). The
experimental setup used for the conducting of the experiments
consisted of a Windows 10 based laptop computer with 2.30 GHz
Intel Core (TM) i5 CPU and 8 GB RAM. The Java language was
used to code and implement the ABCS. In order to evaluate the
performance of the ABCS against the existing AI-strategies and
tools, we relied on problem instances, divided into three groups
reported throughout the literature for t-way testing. A wide range
of real-world applications can possibly be represented by this. The
first table consisted of eight problem instances, where the number
of values and parameters for each one was different, with a
different number of interaction strengths (i.e., 2” t”3) as well.
Conversely, the second table contained a total of nine problem
instances, where the numbers of the interaction strength are
different (i.e., 2” p”10) but the number of values and parameters
for each one was the same (i.e. V=2, P=10). These two tables of
experiments were useful, in order to determine the effectiveness of
ABCS, when the values, parameters and interaction strengths were
varied.
The benchmarking of the AI-strategies included the PSTG
[20], IPOG [30], IPOG-D [23] CTE-XL [24], TVG [31], PICT
[32],ITCH [19], TConfig [15] and Jenny [9] which was adopted
from the published results in the works by Mohd Hazli and Zamli
(2013)[33]. It should be mentioned that the HSS and PSTG are inly
used in the natural-search t-way strategies that support the
interaction strength (t  6) and other reach (t  3)). In our
comparison, tables I to II show the obtained results. The results
indicate the best test suite; whereas, some results with NA (not
available) lack of results from the literature. As some degree of
randomness is common in meta-heuristic based strategies, the
obtained results are non-deterministic.
Fig.3. Flow chart of ABCS

WK,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHUDQG,QIRUPDWLRQ6FLHQFHV,&&2,16
aa
Therefore, 10 independent runs were implemented to achieve
the appropriate statistical confidence. The best results obtained
from these runs have been reported here. The mean results for the
ABCS have also been reported to further mitigate the effect of
randomness. Moreover, the ABCS parameters were set at Nbees =
50, number of cycles = 90 and the limit = 100. The first results in
Table I show that, clearly the ABCS has the ability to outsmart the
previous tools and strategies from the initial outcome obtained in
Table I.
TABLE I.
C
OMPARISON OF
ABCS
TEST SUITE AGAINST EXISTING
AI-
BASED STRATEGIES
.
Configuration PSTG ACA GA SA HSS BA ABCS
Best A. size
CA(N,2,3
4
) 9 9 9 9 9 9 9 10.20
CA(N,2,3
13
)1717 17
16 18 19 16 17.5
CA(N,2,10
10
) NA 159 157* NA 155 183 183 185.8
CA(N,2,5
10
)45
NA NA NA 43 47 43 45.2
CA(N,3,3
6
)42
33 33 33 39 42 46 47.4
CA(N,3,4
6
)102
64 64 64 70 108 106 109.1
CA(N,3,5
6
) NA 125 125 152 199 198 202 205.8
CA(N,3,5
7
) 229 218 218
201* 236 227 231 232.9
TABLE II.
CA
(N;
T
,
2
10
),
T
IS
VARIABLE
FROM
2
TO
10.
t Tconfig Jenny IPOG PSTG HSS BA ABCS
Best A. size
2 9 10 10 8 78 7 8.5
320 18 19 17 16 18 16 16.9
445 39 49 37 3739
35* 38.7
5 95 87 128 82 81 85 80* 83.7
6 183 169 3 52 158 158 162 156* 159.5
7 NS 311 NS NS 298 298 NS NS
8 NS 521 NS NS 498* 503 NS NS
9 NS 788 NS NS 512* 545 NS NS
10 NS 1024 NS NS 1024 1024 NS NS
ABCS was capable of producing the optimal test suite in 8
problems in this present experiment, (as portrayed in the shaded
cells), the cell with NA signified “Not Available results." and the
cell with NS indicated “Not Supported." Which means that the
configuration was not supported by the given strategy. For one out
of eight problems, ABCS produced only three outright test suites
shared with others. Concerning the other problems, our strategy has
generated competitive test suites. All the test size averages, show
the consistency of the ABCS in terms of efficiency. The SA
produced 6 best test suites, where 5 were matched with some other
strategies and tools, and only one only was the most optimal.
Followed by the GA, which produced only one of the most optimal
and the others were shared with some other strategies and tools.
ACA and HSS shared 4 and 2 respectively best results with other
strategies and tools. While both the PSTG and BA generated a best
t-way test suite list, but only one time.
Regarding Table II, HSS outperformed of all the strategies by
producing the best results for 6 out of 9 problems followed by the
ABCS with 5 best results, where the ABCS outperformed all the
strategies and tools by producing optimal test suites for 3 out of 9
without matching with other strategies, and 2 matched with the
HSS. The average results indicate that the proposed strategy has
generated almost the closest best test suite sizes to the least number
of t-way test cases. While the BA and Jenny produced 2 and 1 best
results, respectively, which were shared with other strategies and
tools. The Tconfig, IPOG and PSTG commonly produce the worst
results overall.
VI.
CONCLUSION AND FUTURE WORK
Conclusively, a survey has been carried out on the most-recent
strategies implemented for t-way testing. This survey has analyzed
and highlighted the strengths, weakness and limitations for each of
these strategies. Owing to this, a new t-way strategy, known as the
ABCS, has been suggested. This new strategy utilizes an artificial
bee colony algorithm as a core algorithm for implementation. The
first outcome demonstrates that the ABCS is capable of
outperforming some existing strategies. As part of our plan to
improve the design of the ABCS in two parts in the future work.
We plan at first to cover high interaction parameters. Then, we
intend to amalgamate (hybridize) the artificial bee colony algorithm
with other Optimization algorithms to develop its search
capabilities in all aspects.
REFERENCES
[1] M. F. Klaib, K. Z. Zamli, N. A. M. Isa, M. I. Younis, and R.
Abdullah, "G2Way a backtracking strategy for pairwise test
data generation," presented at the 15th Asia-Pacific Software
Engineering Conference, 2008.
[2] D. M. Cohen, S. R. Dalal, A. Kajla, and G. C. Patton, "The
automatic efficient test generator (AETG) system,"
presented at the Software Reliability Engineering, 1994.
Proceedings., 5th International Symposium on, 1994.
[3] Y. A. Alsariera, M. A. Majid, and K. Z. Zamli, "A bat-
inspired Strategy for Pairwise Testing," ARPN Journal of
Engineering and Applied Sciences, vol. 10, pp. 8500-8506,
2015.