No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
The Dynamic Domain Reduction Procedure for Test Data Generation
Abstract
This paper presents results from a project that is trying to find ways to incorporate test data generation into practical test processes. The results include a new procedure for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and dynamically "pushes" the values through the control-flow graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed. This procedure uses new analysis techniques, offers improvements over previous research results in constraint-based testing, and combines several steps into one coherent process. The dynamic nature of this procedure yields several benefits. Moving through the control flow graph dynamically allows path constraints to be resolved immediately, which is more efficient both in space and time, and more often successful than constraint-based testing. This new procedure also incorporates an intelligent search technique based on bisection. The dynamic nature of this procedure also allows certain improvements to be made in the handling of arrays, loops, and expressions; language features that are traditionally difficult to handle in test data generation systems. The paper presents the test data generation procedure, examples to explain the working of the procedure, and results from a proof-of-concept implementation.
... The approach proposed in this paper is an extension of our previous works IP-PRT [7] and RDDR [8] to generate the domain of the numerical path constraints. In his doctoral dissertation, Offutt pioneered dynamic domain reduction, DDR, a technique that statically detects subdomains of a program input domain, covering an execution path [9]. In DDR, a path constraint is a conjunction of relational expressions representing branching conditions along the path. ...
... The original ExprDomain algorithm [9] does not correctly work when the lower bound, c, or upper bound, d, in rightDom of a division expression is 0. To avoid the division by zero error, we modify the corresponding rule by assigning 1 to c and -1 to d. For example, leftDom for the target constraint is computed as follows: ...
... f(y) is (0, 2). To compute the domain of the variables A[x] and y from the domain of (A[x] + y), we must run the Update algorithm [9]. The algorithm receives an expression and its domain as inputs. ...
Context: Coverage criteria are satisfied by at least one examination of the test target, while many faults are not revealed by one execution. However, despite executing the faulty statement, the test result is correct in certain circumstances. Such coincidentally passing test cases execute the faulty statement but do not cause failures. Objective: This paper introduces the new concept of domain solver. Domain solvers attempt to detect the domain of inputs rather than a single input satisfying a path constraint. Domain coverage is a new metric to evaluate the relative accuracy of the detected domains. The promising point is that the proposed approach similarly treats nonlinear and linear constraints. Method: Domain solver splits a path constraint into conjunctions of simple conditions comparing two expressions. Such a splitting simplifies the constraint-solving task to detect regions of the input space satisfying a comparison between two expressions. After finding a region, an improved version of an algorithm, update, is used to determine the domain of variables involved in the comparing expressions. Results: Our proposed approach, COSMOS, is implemented using the Roslyn compiler. We compared COSMOS with well-known constraint solvers using various linear/nonlinear constraints. The results show that COSMOS improves the number of supported data types involved in a constraint and solves100% of the instances in which the other solvers fail. Besides, COSMOS achieves the best relative accuracy of 84% compared to the existing domain-oriented test suite generation approaches. Moreover, our experiment results illustrate that COSMOS improves the fault-finding capability of other existing test coverage criteria by detecting coincidentally correct test cases. Conclusion: Combining domain coverage and compiler as a service makes a powerful constraint satisfaction method outperforming the existing constraint solvers regarding the number of solved linear/nonlinear constraints. Augmenting other structural test coverage criteria with domain coverage reveals the coincidentally correct test cases.
... In search of appropriate input variables subdomains covering a given path, the domains of all the involved input variables could be considered as a search space for finding the shortest subdomains whose boundary values satisfy the path constraints (PC). Dynamic domain reduction (DDR) (Offutt et al., 1999) and path-oriented test data generation (PRT) (Gotlieb and Petit, 2006) are two known techniques for path oriented test data generation. These techniques shrink the input domain and refute useless subdomains, until the number of the refuted test data reach an acceptable minimum. ...
... 1 static analysis techniques including symbolic execution (Offutt et al., 1999;Gotlieb and Petit, 2008;Dinh et al., 2017;Lin et al., 2017) 2 dynamic execution, including heuristic and exact search methods (Korel, 1992;Pargas et al., 1999;Malhotra and Khari, 2013;Suresh and Rath, 2015;Kumar et al., 2017;Gupta et al., 2015). ...
... There are many test data generation methods which use symbolic execution to find inputs that satisfy a test requirement Gotlieb and Petit, 2008). Two of these techniques have applied the symbolic execution to shrink the initial domain of input variables with the aim of exercising a given path (Offutt et al., 1999;Gotlieb and Petit, 2006). The shrinking process is performed by removing the redundant parts of the input variables domain which contain values that cannot satisfy the desired PCs. ...
... Dynamic Domain Reduction, or DDR in short (Offutt et al. 1999), is a static technique to detect subdomains of a program input domain, covering a feasible execution path. DDR considers a path constraint as conjunction of relational expressions each representing a branching condition along the path. ...
... --Compute the current search point. Fig. 1 An algorithm to compute the next search point (Offutt et al. 1999) Any values in the remaining subdomains will satisfy the condition x > = y, but, no guarantee to satisfy the branching conditions that are derived later. In such cases, DDR backtracks and calls GetSplit to select a different split point, split. ...
... 1. Our GetSplit algorithm is similar to the algorithm, in Fig. 1, except that in our algorithm, srchPt does not vary and it is always equal to constant 1/2. 2. To compute the domains LeftDom and RightDom in terms of the domains of the input variables, x and y, and vice versa, the ExprDomain algorithm, provided by DDR (Offutt et al. 1999), is invoked. This algorithm accepts an arithmetic expression as its input, extracts the abstract syntax tree for the expression, and propagates the domain of the input variables at leaves of the tree up by applying the operations at the interior nodes of the tree and finally ends up with the domain of the given expression. ...
Test data adequacy is a major challenge in software testing literature. The difficulty is to provide sufficient test data to assure the correctness of the program under test. Especially, in the case of latent faults, the fault does not reveal itself unless specific combinations of input values are used to run the program. In this respect, detection of subdomains of the input domain that cover a specific execution path seems promising. A subdomain covers an execution path provided that each test data taken from the subdomain satisfies the path constraint. Dynamic Domain Reduction, or DDR in short, is a very well-known test data generation procedure, targeted at detection of the subdomains of the input domain that satisfy a given path constraint. In this paper, an improved version of DDR called Rapid Dynamic Domain Reduction, or RDDR in short, is introduced. RDDR is intended to explore subdomains of a program input domain, satisfying a given path constraint. For each feasible path, there is a distinct subdomain of the input domain that causes the program to execute the path. Hereby, we introduce a new metric named domain coverage, to qualify input data sets, in terms of the percentage of the subdomain of a feasible path covered by the data set. The main inspiration behind the domain coverage metric is to support test data adequacy. Our empirical results based on some well-known case studies confirms that RDDR significantly outperforms DDR in terms of speed and accuracy.
... Consequently, these techniques are likely to become stuck at a local optimum [52]. DeMillo and Offutt [53] and Offutt et al. [54] proposed approaches for test data generation for mutation testing. The technique proposed in [53] is based on mutation analysis to create test data approximating the test adequacy. ...
... The technique is to automatically generate constraints and solve them to create test cases for unit testing. Another technique proposed in [54] is an extension of previous research [53] in constraint-based testing. This later method takes an initial set of values for each input and then refine the values through the control-flow graph of the program and resolve the path constraint dynamically. ...
... This later method takes an initial set of values for each input and then refine the values through the control-flow graph of the program and resolve the path constraint dynamically. In this work, to generate the test data for test cases we follow the approach proposed in [54]. ...
In general, modern programs are large and complex and it is essential that they should be highly reliable in applications. In order to develop highly reliable software, Java programming language developer provides a rich set of exceptions and exception handling mechanisms. Exception handling mechanisms are intended to help developers build robust programs. Given a program with exception handling constructs, for an effective testing, we are to detect whether all possible exceptions are raised and caught or not. However, complex exception handling constructs make it tedious to trace which exceptions are handled and where and which exceptions are passed on. In this paper, we address this problem and propose a mutation analysis approach to develop reliable object-oriented programs. We have applied a number of mutation operators to create a large set of mutant programs with different type of faults. We then generate test cases and test data to uncover exception related faults. The test suite so obtained is applied to the mutant programs measuring the mutation score and hence verifying whether mutant programs are effective or not. We have tested our approach with a number of case studies to substantiate the efficacy of the proposed mutation analysis technique.
... In this paper we are discussing about the old techniques used in the basis path testing to reduce the test cases. Dynamic Domain Reduction [5] [8], Basis path testing with exception handling [6], Test Case Reduction [9][10]. This paper gives a brief introduction and algorithms proposed by the authors. ...
... In this paper, we are discussing about the Survey on Test case Reduction and compare the techniques in paper [5][6] [9]. All papers are about, reduction of the test cases by using different methods, but, Control Flow Graph in all the methods. ...
... It is an automatic test generation method uses constraints derived from test program controls the execution path in CFG to reduce domains until test data satisfies constraints found for test program [5].In this, split algorithm is the main, it divides the variables. The algorithm is as follows [5]: ...
Test case reduction is traditional in nature. It is very old field on which engineers work from the beginning of the third generation. Reduction is not an easy task, several techniques in this field is developed. There are several challenges faced by the engineers, for instance, cost and time. The reduction is the nightmare of the developer, as, the level of testing is increases, techniques of reduction also becomes more complex. An old technique teaches us the basic of the reduction of test cases without any complexity and through simple algorithms. So, this paper gives the overview of the existing techniques and gives the literature review of test case reduction techniques in software testing field.
... A second approach to test generation is dynamic test generation (e.g., [29,68,80,97,114]): it consists of executing the program P , typically starting with some random inputs, while performing symbolic execution dynamically, collecting symbolic constraints on inputs gathered from predicates in branch statements along the execution, and then using a constraint solver to infer variants of the previous inputs in order to steer the next execution of the program towards an alternative program branch. The conventional stance on the role of symbolic execution is thus turned upside-down: symbolic execution is now an adjunct to concrete execution. ...
... Dynamic test generation was discussed in the 1990s (e.g., [80,97,114]) in a property-guided setting, where the goal is to execute a given specific target program branch or statement. More recently, new variants of dynamic test generation [29,68] blend it with model-checking techniques with the goal of systematically executing all feasible program paths of a program while detecting various types of errors using run-time checking tools (like Purify, Valgrind, or AppVerifier, for instance). ...
Model checking and testing have a lot in common. Over the last two decades, significant progress has been made on how to broaden the scope of model checking from finite-state abstractions to actual software implementations. One way to do this consists of adapting model checking into a form of systematic testing that is applicable to industrial-size software. This chapter presents an overview of this strand of software model checking. © Springer International Publishing AG, part of Springer Nature 2018. All rights reserved.
... Since there is no deterministic solution to satisfy every constraint on the bounded domain (Hentenryck et al., 1998), all previous works are devoted to produce more desirable test data (instead of solving the problem completely). In Offutt et al. (1999), the constraints of the program are obtained after traversing the program path by utilizing the Dynamic Domain Reduction (DDR) process. In the next step, some parts of these constraint allowable areas are eliminated by a search method and by using the split-point method. ...
... Eliminating a part of the allowable area: In this approach, a part of the allowable area eliminates, by a legal method, the partial solving of constraints and appropriate test data generation. DDR method (Offutt et al., 1999) is one of the most famous works that has been done in this field. In this method, after getting the constraints of a path, a hypercuboid would be obtained by eliminating parts of the allowable area of these constraints. ...
Designing test cases and generating data are very important phases in software engineering these days. In order to generate test data, some generators such as random test data generators, data specification generators and path-oriented (Path-Wise) test data generators are employed. One of the most important problems in the path-oriented test data generator is the lack of attention given to discovering faults by the test data. In this paper an approach is proposed to generate some test data automatically so that we can realize the goal of discovering more faults in less time. The number of faults near the boundaries of the input domain is more than the center, according to the Pareto 80–20 principle the test data of this approach will be generated at 20% of the allowable area boundary. To do this, we extracted the boundary hypercuboids and then the test data will be generated by exploiting these hypercuboids. The experimental results show that the fault detection probability and the fault detection speed are improved significantly, compared with the previous approaches. By generating data in this way, more faults are discovered in a short period of time which makes it more possible to deliver products on time.
... Domain reduction methods also suffer from the same problems (e.g., loops, arrays, pointers, and procedure) as symbolic execution methods because the constraints used in domain reduction methods are created by symbolic execution methods. To relieve these problems, dynamic domain reduction method is proposed [13], which dynamically reduces the domain of input variables. However, it does not solve the index problem of arrays and loops. ...
... end if (13) until satisfy or reach maximum iteration (14) = ∪ Σ (15) end for Algorithm 1: Pseudocode of random algorithm. active state and the values of the variables of a specific point of time can be obtained by generating a snapshot of the point. ...
In Model-Based Testing (MBT) area, Search-Based Software Testing (SBST) has been employed to generate test cases from the model of a system under test. However, many types of models have been used in MBT. If the type of a model has changed from one to another, all functions of a search technique must be reimplemented because the types of models are different even if the same search technique has been applied. It requires too much time and effort to implement the same algorithm over and over again. We propose a model-independent software framework for SBST, which can reduce redundant works. The framework provides a reusable common software platform to reduce time and effort. The software framework not only presents design patterns to find test cases for a target model but also reduces development time by using common functions provided in the framework. We show the effectiveness and efficiency of the proposed framework with two case studies. The framework improves the productivity by about 50% when changing the type of a model.
... The former includes iterative relaxation [4], interval computation [5], symbol execution [6,7] and so on. Its main idea is to convert the coverage of construct elements to a constraint solving problem so as to generate test inputs satisfying a specific coverage criterion [8][9][10]. ...
... Then, the fitness function of whole program can be defined by comprehensively considering fitness of each branch. Suppose a program has s branches, the fitness function of whole program can be calculated via formula (5). ...
Test data generation is always a key task in the field of software testing. In recent years, meta-heuristic search techniques have been considered as an effective way to assist test data generation in software structural testing. In this way, some representative test cases with high-coverage capability can be picked out from program input space. Harmony search (HS) is a recently developed algorithm and has been vigorously applied to various optimization problems. In the paper, we attempt to apply harmony search algorithm to generate test data satisfying branch coverage. At the preprocessing stage, the probes used for gathering coverage information are inserted into all branches via program static analysis. At the same time, the encoding and decoding styles between a test case and a harmony are also determined in advance. At the stage of test data searching, the subset of test data that has much stronger covering ability is stored in harmony memory. During the evolution process, one part of test suite is selected and adjusted from the harmony memory, and the other part is randomly generated from input space. Once a test suite is yielded after one-round search, its coverage can be measured by fitness function in our search algorithm. In our work, a new fitness function for branch coverage is constructed by comprehensively considering branch distance and branch weight. Here, the branch weight is determined by branch information in program, that is, the nesting level of a specific branch and the predicate types in it. Subsequently, the computed coverage metric is used for updating the test suite in the next round of searching. In order to validate the effectiveness of our proposed method, eight well-known programs are used for experimental evaluation. Experimental results show that the coverage of HS-based method is usually higher than those of other search algorithms, such as simulated annealing (SA) and genetic algorithm (GA). Meanwhile, HS demonstrates greater stability than SA and GA when varying the population size or performing repeated trials. That is to say, music-inspired HS algorithm is more suitable to generate test data for branch coverage in software structural testing.
... Whereas the literature has delved into what factors make a physical configuration more or less successful [9,10,11], no work to date has considered how to augment the robot optimization with knowledge from external sources, despite it being common in other fields [12], like test [13] and content generation [14]. Just like most animal lifeforms, humans evolved powerful mental ''world models'' for predicting the future states of the world [15]. ...
Robots with a modular body permit a wide range of physical configurations, which can be obtained by arranging the composing modules differently. While this freedom makes modular robots capable of performing different tasks, finding the optimal physical configuration for a given task is not trivial. In fact, practitioners attempt to jointly optimize the body and the controller of the robot for a given task, but the result is not always satisfactory. More broadly, it is not clear what factors make a physical configuration more or less successful. In this paper, we aim to fill this gap and verify if humans can be predictive with respect to the performance of an optimized controller for a given robot body. We consider the case of Voxel-based Soft Robots (VSRs), whose rich dynamic induced by the softness of the modules makes the body particularly relevant for the robot ability to perform a task. We instantiate a number of (simulated) VSR bodies, differing in shape and actuation mechanism, and let a panel of humans control them, by means of online interaction with the simulator, while performing the task of locomotion. We use the same bodies with controllers obtained with evolutionary optimization, for the same task. We compare the ranking of human- and optimized AI-based VSRs, finding them very similar. We believe that our results strengthen the hypothesis that intrinsic factors in the body of modular robots determine their success.
... It it has proved immense diminution in test suits but it is less proficient and huge time consuming. and is moderately more expensive technique [13]. exhaustion, prioritization, lastly test suits will be executed so that it can shrink debugging exertion as in [6], afterward we order test suits based on test case rank, here rectification of fault, apfd calculations, proportion of error discovery, risk discovery examination, are performed, here numerous formulas are integrated and test case priority is allotted for dissimilar test suits then we run based on test case priority, lastly contrast among different prioritized test suits is evaluated based on error revealing rate which exhibit that contrast to existing techniques our procedure has optimal efficiency over existing, finally we execute test suits based on precedence, this method exposes maximum numeral of bugs and runs test cases with brutal test cases primary and helps in protecting product from breakdown. ...
Software Testing is investigation of software, it is used with goal of discovering defects in a system. It evaluates the system against faults,errors, bugs, failures and it is used to verify and validate application and improve efficiency of software. Hence testing system plays foremost role in developing product, but when test cases increases execution time , expenditure, effort, bugs, complication surpass, therefore proposed method is implemented by performing test case diminution, apfd,test case ordering to attain maximum efficiency
... In his research, the researcher [3], wrote a literature review which shows techniques of test-suit reduction. The review is basically of these papers [4]- [6]. In these papers different algorithms are discussed separately to reduce the test suits. ...
... Herman [24] in his paper has elucidated the automatic generation of test cases or test data with the help of search-based optimization techniques. Offult et al. [25] in their research have portrayed the dynamic test case generation mechanism by taking input values and established the control flow graph where the test paths are traversed. They have also argued that the search-based method when incorporated takes less time. ...
Software testing is leading toward automation that reduces the effort to find errors or bugs. The identification of test cases and its critical domain requirements is done with generation of test cases. The brooding characteristic of the cuckoo bird is explained through the adaptive cuckoo search meta-heuristic algorithm (ACSA) that further narrates that host nest is used by the cuckoo bird for laying their eggs and the next generation also sees the best quality eggs from the host bird’s nest. This paper focuses on the adoption of ACSA for analysis, generation, and optimization of random test cases. In addition to that, the present work also explains the model driven approach to automatically generate and optimize the test cases with the help of unified modeling language diagram like sequence diagram. Then, the respective sequence diagram is converted into a sequence diagram graph that shows the flow of sequences being produced. Thereafter, it is optimized using ACSA by taking a case study of withdrawal operation of ATM transaction. The said approach is also evaluated in terms of efficiency and usefulness for generating the test cases through simulated experiments. In addition to that, the projected approach also identifies the operational faults as well as message faults.
... Determining the validity of a subdomain with respect to a PC is dependent on the constraint propagation algorithm. Thus, similar to other related works [9,14,26], a valid (or invalid) subdomain may be considered as invalid (or valid) and refuted (or kept) by the proposed method. Elimination of a whole input subdomain because it consists of negative and positive test results may cause the test runner to miss some false positive errors, especially corner cases, in the proposed method. ...
... is a logical value, when is T, the true value of the constraint t is True, and the logical value F represents False. The particularity of the logical expression constraint solving problem studied in this paper is not finding the value of variables, but making domain reduction [6], so that when the variable is searched in the domain , it can reduce the search space in the constraint solving process, for the reason that can improve the efficiency of generating testing cases. ...
This paper extends the theory of constraint solving and presents a method to solve logical expression constraint problems existing in the process of generating test cases automatically. It defines the concept of the logical expression constraint on the path of C programs. In order to improve solving efficiency, it also presents an algorithm of pre-processing and heuristic backtracking. Our experiment results show that the method can narrow the domain of variables from meaningless infinite intervals to a finite range and achieve better coverage, and the results also show that the algorithm can improve the testing efficiency and halve the time of generating test cases.
... However, tests reliability depends on the count of test oracles and the efficiency of the input data to parse the maximum states of the System Under Test SUT. In this context, constraint-based testing introduced by Offutt in 1991 [3], combines a symbolic execution and dynamic constraint solving [14] in order to generate test inputs. ...
... Similar to us, Godzilla [40] relies on constraint solving. This work was later optimized with dynamic domain reduction techniques to cover complex input data [41]. Dynamic symbolic execution overcomes the limitations of symbolic execution, and can be used in directed automated random testing [42]. ...
... A test case contains test input values, expected output and the constraints of preconditions and post conditions for the input values. Generation and selection of the effective test cases from UML models is one of the most challenging tasks [3]. ...
This paper proposes an approach for selecting best testing scenarios using Genetic Algorithm. Test cases generation approach using UML sequence diagrams, class diagrams and Object Constraint Language (OCL) as software specifications sources. There are three main concepts: Edges Relation Table (ERT), test scenarios generation and test cases generation used in this work. The ERT is used to detect edges in sequence diagrams, identifies their relationships based on the information available in sequence diagrams and OCL information. ERT is also used to generate the Testing Scenarios Graph (TSG). The test scenarios generation technique concerns the generation of scenarios from the testable model of the sequence diagram. Path coverage technique is proposed to solve the problem of test scenario generation that controls explosion of paths which arise due to loops and concurrencies. Furthermore, GA used to generates test cases that covers most of message paths and most of combined fragments (loop, par, alt, opt and break), in addition to some structural specifications.
... Therefore, per each clause, we calculate values for involved input variables such that changing one of the values makes the different evaluation for the clause, i.e., from True to False or vice versa. It must be noted that this is not Dynamic Domain Reduction (DDR) proposed by Offutt and Pan [6]:"It takes an initial set of values for each input, and dynamically pushes the values through the control-flow graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed." ...
The search space of potential inputs of a program is very large, even for the very small one, while this size is a key determining factor affecting the performance of any search-based test data generation approach. However, despite the large volume of work on search-based test data generation, the literature contains little work that concerns this problem. In this paper, by analysis of the clauses of the program, in addition to proposing a new search space reduction strategy, a new seeding approach is introduced.
... The main idea behind our domain detection algorithm is inspired by the dynamic domain reduction, DDR, algorithm, proposed by Offutt [7]. DDR uses divide-and-conquer method to reduce an input variable domain to valid subdomains for exercising a given path. ...
In this article, a new criterion, domain coverage, for white box testing is introduced. In search of the biggest subdomains for input variables exercising a given path, the variable domains are broken down into overlapping subregions in such a way that the resultant subregions either exercise or deviate from the path. To this aim, an incremental method is applied to detect the subregions, covering the longest subpaths from the start of the given path. The detected subdomain is further subdivided, and this process is repeated as far as the path is fully covered. If no subdomain is detected, the method is backtracked. The backtrack begins with the subdomain covering the longest subpath at the beginning of the given path, but this time the subdomain is divided into relatively smaller subregions, compared with the last time. A directed acyclic graph representation of each input variable domain is used to ignore re-examination of those subdomains which deviate from the path. Conducted experiments reveal that our proposed domain-oriented algorithm outperforms the existing path-oriented test data generation algorithms.
... It incrementally generate test cases for single class, and then multiple classes. Khamis et al. [76] propose a dynamic domain reduction (DDR)-based method [77] for DFT. The DDR technique basically integrates the idea of symbolic execution and constraint-based testing, and starts from the initial domains of input variables and program flow graph to dynamically find a path that can reach the target def-use pair. ...
Data-flow testing (DFT) checks the correctness of variable definitions by observing their corresponding uses. It has been empirically proved to be more effective than control-flow testing in fault detection, however, its complexities still overwhelm the testers in practice. To tackle this problem, we introduce a hybrid testing framework: (1) The core of our framework is symbolic execution, enhanced by a novel guided path search to improve testing performance; and (2) we systematically cast DFT as reachability checking in software model checking to complement SE, yielding practical DFT that combines the two techniques' strengths. We implemented our framework on the state-of-the-art symbolic execution tool KLEE and model checking tools BLAST, CPAchecker and CBMC, and extensively evaluate it on 30 real-world subjects with collectively 22,793 def-use pairs. The enhanced SE approach not only covers more pairs, but also reduces testing time by 10$\sim$43%. The model checking approach can effectively weed out infeasible pairs that KLEE cannot infer by 70.1$\sim$95.8%. Finally, for all subjects, our hybrid approach can improve data-flow coverage by 28.7$\sim$46.3%, and reduce testing time by up to 64.6% than the symbolic execution approach alone. This hybrid approach also enables the cross-checking of each component for reliable and robust testing results.
... In addition, CEGAR is essentially a static approach; its testing performance may not be as high as other dynamic testing approaches (e.g., dynamic symbolic execution-based approach [Su et al. 2015]) when generating test cases for feasible pairs. Khamis et al. [2011] enhance the Dynamic Domain Reduction procedure [Offutt et al. 1999] (DRR) to perform data-flow testing for Pascal programs. The DDR technique basically integrates the ideas of symbolic execution and constraint-based testing. ...
Data-flow testing (DFT) is a family of testing strategies designed to verify the interactions between each program variable's definition and its uses. Such a test objective of interest is referred to as a def-use pair. DFT selects test data with respect to various test adequacy criteria (i.e., data-flow coverage criteria) to exercise each pair. The original conception of DFT was introduced by Herman in 1976. Since then, a number of studies have been conducted, both theoretically and empirically, to analyze DFT's complexity and effectiveness. In the past four decades, DFT has been continuously concerned, and various approaches from different aspects are proposed to pursue automatic and efficient data-flow testing. This survey presents a detailed overview of data-flow testing, including challenges and approaches in enforcing and automating it: (1) it introduces the data-flow analysis techniques that are used to identify def-use pairs; (2) it classifies and discusses techniques for data-flow-based test data generation, such as search-based testing, random testing, collateral-coverage-based testing, symbolic-execution-based testing, and model-checking-based testing; (3) it discusses techniques for tracking data-flow coverage; (4) it presents several DFT applications, including software fault localization, web security testing, and specification consistency checking; and (5) it summarizes recent advances and discusses future research directions toward more practical data-flow testing.
... In software testing field a survey of Chaurasia et al. [11] is given which is a literature review of test case reduction techniques. They discuss and compare three algorithms presented in [12][13] [14]. All papers are about test case reduction but Control Flow Graph is used in all the methods. ...
Regression testing is considered to be the most expensive phase in software testing. Therefore, regression testing reduction eliminates the redundant test cases in the regression testing suite and saves cost of this phase. In order to validate the correctness of the new version software project that resulted from maintenance phase, Regression testing reruns the regression testing suite to ensure that the new version. Several techniques are used to deal with the problem of regression testing reduction. This research is going to classify these techniques regression testing reduction problem.
... The result shows that this methodology significantly reduces time and effort spent in executing thousands of test cases. On the other hand, some other studies use different classifiers [5,16,17,18] in different domains using tools such as WEKA, Rapid Miner and ORANGE. Lee and Chan [19] proposed an approach that can be used to enhance the process of automated test tool model using machine learning technique from association rule mining. ...
The main purpose of test case reduction is to decrease the number of test cases in order to minimize the time and cost of executing them. We used the data mining approach, mainly because of its ability to extract patterns of test cases that are invisible. In this paper, we use two data mining classifiers, Naï ve based and J48, in order to classify a set of test cases and identify which test case is redundant or irredundant. The results show the applicability of data mining classification in removing the number of redundant test cases.
... The result shows that this methodology significantly reduces time and effort spent in executing thousands of test cases. On the other hand, some other studies use different classifiers [5], [16], [17], [18] in different domains using tools such as WEKA, Rapid Miner and ORANGE. Lee and Chan [19] proposed an approach that can be used to enhance the process of automated test tool model using machine learning technique from association rule mining. ...
... In this method, a path is selected and upcoming paths if it is cover any new literals which did not cover by above paths then it is selected otherwise rejected so that number of paths is reduced and tool will generate test cases for These Selected path.[0]Void function(int a, String b, int[1]c, int d) {[2]if (a >=1 d >=5 b! =null)[3]a :=a+1;[4]else[5]a :=a-1;[6]if (c == 10)[7];[8]else[9];[10]If (a >=1)[11]b.concat(˄˄XXX˄˄);[12]else[13]b.concat(˄˄YYY˄˄); } International Journal of Applied Engineering Research, ISSN 0973-4562 Vol. ...
For generating test cases by symbolic execution technique has recently become an active research area. When program has large number of paths, symbolic execution generates large number of test case, It will take a lot of time to run all test cases so in practice it is impractical. In this paper, we propose a new approach for test case selection in symbolic execution based test case generation. This method has following characteristics. 1) Number of path is collected during symbolic execution, we are going to apply sorting on this path condition based on number of constraint in descending order so that the path which cover maximum number of atomic conditions during symbolic execution is selected first and remaining path if it is not subset of already cover atomic condition then accept it otherwise reject it, For keeping fault detection capability. 2) This method does not depend on any symbolic execution tool because it is based on analysis on path condition. We implement this method in klee tool and evaluate this method in real time. Our evaluation show that method produces a significant reduction in the size of test cases while effectively preserves fault detection capability.
... AutoGen [31] DiffGen [15] Gozilla [21], [48] [23], [35], [36] Juta [25] SimC [30] Slueth [44], [50] Symclat [23] Symstra [39], [40] TAO [32] TestGen [45] [10], [17] Tgen [33] [14], [27] In the table 9, it can be seen that almost all the tools have the primary citation field filled, which is somewhat necessary for prototype tools. Only two papers do not have the primary citation, but it could be possible that our keywords defined in section 4.1 did not match with it and failed to recover any paper. ...
Master thesis on automaton of unit testing
... Some improved methods have been proposed to heighten the diversity of test data [6, 7]. Static method only needs static analysis and transformation , without involving actual execution of the program under test, such as symbolic execution [2, 8], and domain reduction [9]. But this method usually requires a significant amount of algebra and (or) interval arithmetic [10]. ...
The application of genetic algorithms in automatically generating test data has aroused broad concerns and obtained delightful achievements in recent years. However, the efficiency of genetic algorithm-based test data generation for path testing needs to be further improved. In this paper, we establish a mathematical model of generating test data for multiple paths coverage. Then, a multipopulation genetic algorithm with individual sharing is presented to solve the established model. We not only analyzed the performance of the proposed method theoretically, but also applied it to various programs under test. The experimental results show that the proposed method can improve the efficiency of generating test data for many paths' coverage significantly.
... It is hard to use CBT approach to deal with the loop conditions for input variables, the subscription of array elements and module calling; and its constraints solving technique may cause no solution about the constraints system. Offutt et al [3] improved CBT approach and presented a dynamic domain reduction approach that is called DDR for short. DDR approach only generates the testing cases that kill the single mutant and it does not consider the relation among several mutants. ...
The mutants for the given program are generated in use of mutating operators, the accessibility and necessity conditions for each mutant are obtained, and then the accessibility and necessity condition sets are reduced. By combining the reduced accessibility and necessity condition sets with the program statements, the program paths are created, and the mutating test cases generation is transformed into the path-based automatic testing cases generation. A linear constraints system with input variables is constructed and solved by the improved iterative relaxation method, and the testing cases that can kill simultaneously several mutants are generated. The counterexample is generated by model checker Java PathFinder (JPF), and then the killed mutants are achieved. Compared with the single mutating test case generation method with no testing requirement reduction, the experimental results show that, on the precondition of guaranteeing the same rate of mutating sufficiency, the presented approach can obtain stronger revealing faults ability and higher mutating test efficiency with fewer testing cases.
... To decrease costs, reduction techniques [13]have attempted to reduce the size of the test data by identifying a representative set from the original test suite, that satisfies all the testing objectives. These techniques have used various reduction algorithms and have achieved great success in reducing the volume of the test data [10]. ...
The test data produced by automatic test data generators are often 'unnatural' partic-ularly for the programs that make use of human-recognisable variables such as 'country', 'name', 'date', 'time', 'age' and so on. The test data generated for these variables are usually arbitrary-looking values that are complex for human testers to comprehend and evaluate. This is due to the fact that automatic test data generators have no domain knowledge about the program under test and thus the test data they produce are hardly recognised by human testers. As a result, the tester is likely to spend additional time in order to understand such data. This paper demonstrates how the incorporation of some domain knowledge into an automatic test data generator can significantly improve the quality of the generated test data. Empirical studies are proposed to investigate how this incorporation of knowledge can reduce the overall testing costs.
... 7 DDR X X [10] 8 ATGen X X [11] 9 ...
Recently, a new testing technique called concolic testing receives lots of attention. Concolic testing generates test data by combining concrete program execution and symbolic execution to achieve high test coverage. CREST is a representative open-source test tool implementing concolic testing. Currently, however, CREST only deals with integer type as input. This paper presents a new rule for automated test data generation in presence of inputs of pointer type. The rules effectively handles multi-level stack-directed pointers that are mainly used in C programs. In addition, we describe a tool named vCREST implementing the proposed rules together with the results of applying the tool to some C programs.
Test data generation techniques optimally seek to provide test data sets that, when applied, ensure the accuracy of the program under test, or expose any existing error.
Compared with traditional structural test criteria, test data generated based on mutation testing are proved more effective at detecting faults. However, not all test data have the same potence in detecting software faults. If test data are prioritized while generating for mutation testing, the defect detectability of the test suite can be further strengthened. In view of this, we propose a method of test data generation for weak mutation testing via sorting mutant branches based on their dominance degrees. First, the problem of weak mutation testing is transformed into that of covering mutant branches for a transformed program. Then, the dominance relation of mutant branches in the transformed program is analyzed to obtain the non-dominated mutant branches and their dominance degrees. Following that, we prioritize all non-dominated mutant branches in descending order by virtue of their dominance degrees. Finally, the test data are generated in an orderly manner by selecting the mutant branches sequentially. The experimental results on 15 programs show that compared with other methods, the proposed test data generation method can not only improve the error detectability of the test suite, but also has higher efficiency.
Software-based optimization techniques have been increasingly used to automate code coverage analysis since the nineties. Although several studies suggest that interdependencies can exist between condition constructs in branching conditions of real life programs e.g. (\(i<=100\)) or (\(i==j\)), etc., to date, only the Ariadne system, a Grammatical Evolution (GE)-based Search Based Software Testing (SBST) technique, exploits interdependencies between variables to efficiently automate code coverage analysis.
Mutation testing is a fault-oriented software testing technique, and a test suite generated based on the criterion of mutation testing generally has a high capability in detecting faults. A mutant that is hard killed is called a stubborn one. The traditional methods of test data generation often fail to generate test data that kill stubborn mutants. To improve the efficiency of killing stubborn mutants, in this article, we propose a method of generating test data by dynamically reducing the search domain under the criterion of strong mutation testing. To fulfill this task, we first present a method of measuring the stubbornness of a mutant based on the reachability condition of a mutated statement. Then, we formulate the problem of generating test data to kill the mutant as an optimization one with a unique constraint. Finally, we generate test data using a coevolutionary genetic algorithm. Given the fact that the domain of test data that kills a stubborn mutant is generally small, we adopt a method of dynamically reducing the search domain to improve the efficiency of the algorithm. We apply the proposed method to test eight benchmark and industrial programs. The experimental results demonstrate that the proposed method has capabilities in seeking stubborn mutants and efficiently generating test data to kill stubborn mutants.
The automation of software testing promises to delegate to machines what is otherwise the most labor-intensive and expensive part of software development. The past decade has seen a resurgence in research interest for this problem, bringing about significant progress. In this article, we provide an overview of automated test generation for software, and then discuss recent developments that have had significant impact on real-life software.KeywordsSoftware testingProgram analysisSymbolic execution
In software testing, testing of all program statements is a very crucial issue as it consumes a lot of time, effort and cost. The time, effort and cost can be reduced by using an efficient technique to reduce the test case and a good optimization algorithm to generate efficient, reliable and unique test cases. In this paper, the concept of dominance tree is used which covers all edges/statement by using minimum test case. Nature inspired algorithm - PSO (Particle Swarm Optimization) by applying different inertia weights is used to generate unique, reliable and efficient test cases to cover the leaf nodes of dominance tree. Inertia weights like fixed inertia weight (FIW), global-local best (GLbestIW), Time-Dependent weight (TDW), and proposed GLbestRandIW weights are used with PSO to investigate the effect of inertia weights on the execution of PSO with respect to number of generation required, percentage coverage , total test cases generated to test the software under consideration.
One promising way to improve the accuracy of fault localization based on statistical debugging is to increase diversity among test cases in the underlying test suite. In many practical situations, adding test cases is not a cost-free option because test oracles are developed manually or running test cases is expensive. Hence, we require to have test suites that are both diverse and small to improve debugging. In this paper, we focus on improving fault localization of Simulink models by generating test cases. We identify three test objectives that aim to increase test suite diversity. We use these objectives in a search-based algorithm to generate diversified but small test suites. To further minimize test suite sizes, we develop a prediction model to stop test generation when adding test cases is unlikely to improve fault localization. We evaluate our approach using three industrial subjects. Our results show (1) the three selected test objectives are able to significantly improve the accuracy of fault localization for small test suite sizes, and (2) our prediction model is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.
This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based on the insight, we present the FIFL framework, which uses both the spectrum information of edits (obtained using the existing FaultTracer approach) as well as the potential impacts of edits (simulated by mutation changes) to achieve more accurate fault localization. We evaluate FIFL on real-world repositories of nine Java projects ranging from 5.7KLoC to 88.8KLoC. The experimental results show that FIFL is able to outperform the state-of-the-art FaultTracer technique for localizing failure-inducing program edits significantly. For example, all 19 FIFL strategies that use both the spectrum information and simulated impact information for each edit outperform the existing FaultTracer approach statistically at the significance level of 0.01. In addition, FIFL with its default settings outperforms FaultTracer by 2.33% to 86.26% on 16 of the 26 studied version pairs, and is only inferior than FaultTracer on one version pair.
Software testing sufficiency means that the software's performance on limited testing data can represent its performance on all input data. Ideally, in software testing, the testing should be carried on till all errors in the program are detected and removed. As a testing strategy to measure the completeness of the test cases set, mutation testing is a defects-oriented unit testing technology, and a feasible software testing method to generate a complete set of test cases. The article systematically simulates the different defects in software by using mutation operators to create mutants, and then construct testing data set to be able to kill these mutants. It analyzes the procedures of mutation testing and the generation of mutation operators with specific examples. Experiment indicates that the program mutation technology has enhanced the test cases, which greatly improved the software testing sufficiency.
We give an overview over the principles of constraint-based test data generation, discuss its limitations and potentials and touch some of the domains which may be interesting to combine with constraint-based testing techniques. Automated generation of test data is an example where this technique can be applied and significantly increase the degree of automation, but it is not limited to. This paper is intended to give interested readers a quick entry into the methods and applications to allow a deeper understanding and an informed verdict about the actual capabilities and potential future directions.
Search-based test generators, such as those using genetic algorithms and alternative variable methods, can automatically generate test inputs. They typically rely on fitness functions to calculate fitness scores for guiding the search process. This paper presents a novel rule-based testing (RBT) approach to automated generation of test inputs from Java byte code without using fitness functions. It extracts tagged paths from the control flow graph of given byte code, analyzes and monitors the predicates in the tagged paths at runtime, and generates test inputs using predefined rules. Our case studies show that RBT has outperformed the test input generators using genetic algorithms and alternative variable methods.
This paper proposes an approach for selecting best testing scenarios using Genetic Algorithm. Test cases generation approach uses UML sequence diagrams, class diagrams and Object Constraint Language (OCL) as software specifications sources. There are three main concepts: Edges Relation Table (ERT), test scenarios generation and test cases generation used in this work. The ERT is used to detect edges in sequence diagrams, identifies their relationships based on the information available in sequence diagrams and OCL information. ERT is also used to generate the Testing Scenarios Graph (TSG). The test scenarios generation technique concerns the generation of scenarios from the testable model of the sequence diagram. Path coverage technique is proposed to solve the problem of test scenario generation that controls explosion of paths which arise due to loops and concurrencies. Furthermore, GA used to generates test cases that covers most of message paths and most of combined fragments (loop, par, alt, opt and break), in addition to some structural specifications.
As a test criterion, mutation analysis is known for yielding very effective tests. It is also known for creating many test requirements, each of which is represented by a "mutant" that must be "killed." In recent years, researchers have found that these test requirements have a lot of duplication, in that many test requirements yield the same tests. Put another way, hundreds of mutants can usually be killed by only a few dozen tests. If we could reduce this duplication without reducing mutation's effectiveness, mutation testing could become more cost-effective. One avenue of this research has been to use only one type of mutant, the statement deletion mutation operator. Researchers have found that statement deletion mutation has relatively few mutants, but yields tests that are almost as effective as using all mutants, with the significant benefit that fewer equivalent mutants are generated. This paper extends this idea by asking a simple question: if deleting statements is a cost-effective way to design tests, will deleting other program elements also be effective? This paper presents results from mutation operators that delete variables, operators, and constants, finding that indeed, this is an efficient and effective approach.
This paper presents a technique that based on a combination of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called GPSCA (Genetic-Particle Swarm Combined Algorithm) which is used to generate automatic test data for data flow coverage with using dominance concept between two nodes. The performance of the proposed approach is analyzed on a number of programs having different size and complexity. Finally, the performance of GPSCA is compared to both GA and PSO for generation of automatic test cases to demonstrate its superiority.
Despite decades of work by researchers and practitioners on numerous software quality assurance techniques, testing remains one of the most widely practiced and studied approaches for assessing and improving software quality. Our goal, in this paper, is to provide an accounting of some of the most successful research performed in software testing since the year 2000, and to present what appear to be some of the most significant challenges and opportunities in this area. To be more inclusive in this effort, and to go beyond our own personal opinions and biases, we began by contacting over 50 of our colleagues who are active in the testing research area, and asked them what they believed were (1) the most significant contributions to software testing since 2000 and (2) the greatest open challenges and opportunities for future research in this area. While our colleagues’ input (consisting of about 30 responses) helped guide our choice of topics to cover and ultimately the writing of this paper, we by no means claim that our paper represents all the relevant and noteworthy research performed in the area of software testing in the time period considered—a task that would require far more space and time than we have available. Nevertheless, we hope that the approach we followed helps this paper better reflect not only our views, but also those of the software testing community in general.
This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based on the insight, we present the FIFL framework, which uses both the spectrum information of edits (obtained using the existing FaultTracer approach) as well as the potential impacts of edits (simulated by mutation changes) to achieve more accurate fault localization. We evaluate FIFL on real-world repositories of nine Java projects ranging from 5.7KLoC to 88.8KLoC. The experimental results show that FIFL is able to outperform the state-of-the-art FaultTracer technique for localizing failure-inducing program edits significantly. For example, all 19 FIFL strategies that use both the spectrum information and simulated impact information for each edit outperform the existing FaultTracer approach statistically at the significance level of 0.01. In addition, FIFL with its default settings outperforms FaultTracer by 2.33% to 86.26% on 16 of the 26 studied version pairs, and is only inferior than FaultTracer on one version pair.
BZ-TESTING-TOOLS (BZ-TT) is a tool set for automated test case generation from B and Z specifications. BZ-TT uses boundary and cause–effect testing on the basis of the formal model. It has been used and validated on several industrial applications in the domain of critical software, particularly smart card and transport systems. This paper presents the test coverage criteria supported by BZ-TT. On the one hand, these correspond to various classical structural coverage criteria, but specialized to the case of B abstract machines. The paper gives algorithms for these in Prolog. On the other hand, BZ-TT introduces new coverage criteria for complex data structures, based on boundary analysis: this paper defines weak and strong state-boundary coverage, input-boundary coverage and output-boundary coverage. Finally, the paper describes how BZ-TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable). Copyright © 2004 John Wiley & Sons, Ltd.
A new technique and tool are presented for test data generation for path testing. They are based on the dynamic technique and on a Genetic Algorithm, which evolves a population of input data towards reaching and solving the predicates along the program paths. We improve the performance of test data generation by using past input data to compose the initial population for the search. An experiment was done to assess the performance of the techniques compared to that of random data generation.
Database applications are built using two different programming language constructs: one that controls the behavior of the application, also referred to as the host language; and the other that allows the application to access/retrieve information from the backend database, also referred to as the query language. The interplay between these two languages makes testing of database applications a challenging process. Independent approaches have been developed to evaluate test case quality for host languages and query languages. Typically, the quality of test cases for the host language (e.g., Java) is evaluated on the basis of the number of lines, statements and blocks covered by the test cases. High quality test cases for host languages can be automatically generated using recently developed concolic testing techniques, which rely on manipulating and guiding the search of test cases based on carefully comparing the concrete and symbolic execution of the program written in the host language. Query language test case quality (e.g., SQL), on the other hand, is evaluated using mutation analysis, which is considered to be a stronger criterion for assessing quality. In this case, several mutants or variants of the original SQL query are generated and the quality is measured using a metric called mutation score. Higher mutation score indicates higher quality for the test cases. In this paper we present a framework, called ConSMutate, which guides concolic testing using mutation analysis for test case generation for database applications. The novelty of the framework is that it ensures that the test cases are of high quality not only in terms of coverage of code written in the host language, but also in terms of mutant detection of the queries written in the query language. We present a prototype implementation of our technique and show its advantages using two non-trivial case studies.
A novel technique for automatically generating test data is presented. The technique is based on mutation analysis and creates test data that approximate relative adequacy. It is a fault-based technique that uses algebraic constraints to describe test cases designed to find particular types of faults. A set of tools (collectively called Godzilla) that automatically generates constraints and solves them to create test cases for unit and module testing has been implemented. Godzilla has been integrated with the Mothratesting system and has been used as an effective way to generate test data that kill program mutants. The authors present an initial list of constraints and discuss some of the problems that have been solved to develop the complete implementation of the technique.
__________________ * Work performed by the author while affiliated with Kestrel Institute. Author's current affiliation is the Soft-ware Technology Center, Research and Development Div-ision, Lockheed Missiles and Space Company, Palo Alto, CA. For certain structural testing criteria a significant proportion of tests instances are infeasible in the sense the semantics of the program implies that test data cannot be constructed that meet the test requirement. This paper describes the design and prototype implementation of a structural testing system that uses a theorem prover to determine feasibility of testing requirements and to optimize the number of test cases required to achieve test coverage. Using this approach, we were able to accurately and efficiently determine path feas-ibility for moderately-sized program units of production code written in a subset of Ada. On these problems, the computer solutions were obtained much faster and with greater accuracy then manual analysis. The paper describes how we formalize test criteria as control flow graph path expressions; how the criteria are mapped to logic formulas; and how we control the com-plexity of the inference task. It describes the limitations of the system and proposals for its improvement as well as other applications of the analysis.
Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.
Constraint-based testing is a novel way of generating test data to detect specific types of common programming faults. The conditions under which faults will be detected are encoded as mathematical systems of constraints in terms of program symbols. A set of tools, collectively called Godzilla, has been implemented that automatically generates constraint systems and solves them to create test cases for use by the Mothra testing system. Experimental results from using Godzilla show that the technique can produce test data that is very close in terms of mutation adequacy to test data that is produced manually, and at substantially reduced cost. Additionally, these experiments have suggested a new procedure for unit testing, where test cases are viewed as throw-away items rather than scarce resources.
Enhanced context free grammars are an effective means of generating test data, even for simple programs. Grammars can be used to create intelligent random samples of tests as well as to create exhaustive tests. Action routines and variables may be used to compute expected results and perform other actions that are difficult or impossible to perform with ordinary context free grammars. Grammars may be used to create subroutine test data that contains pointers and other binary data. Test grammars not only provide a method for improving software quality, but should prove to be the foundation of much future research.
The authors give a brief introduction to mutation analysis. They
they discuss Mothra, emphasizing how it interacts with the tester. The
authors present some major problems with using mutation analysis and
discuss possible solutions. They conclude with a solution to one of
these problems-a method of automatically generating mutation-adequate
data
Sensitivity analysis, which estimates the probability that a program location can hide a failure-causing fault, is addressed. The concept of sensitivity is discussed, and a fault/failure model that accounts for fault location is presented. Sensitivity analysis requires that every location be analyzed for three properties: the probability of execution occurring, the probability of infection occurring, and the probability of propagation occurring. One type of analysis is required to handle each part of the fault/failure model. Each of these analyses is examined, and the interpretation of the resulting three sets of probability estimates for each location is discussed. The relationship of the approach to testability is considered.< >
The authors extend the definitions of the previously introduced
family of data flow testing criteria to apply to programs written in a
large subset of Pascal. They then define a family of adequacy criteria
called feasible data flow testing criteria, which are derived from the
data-flow testing criteria. The feasible data flow testing criteria
circumvent the problem of nonapplicability of the data flow testing
criteria by requiring the test data to exercise only those
definition-use associations which are executable. It is shown that there
are significant differences between the relationships among the data
flow testing criteria and the relationships among the feasible data flow
testing criteria. The authors discuss a generalized notion of the
executability of a path through a program unit. A script of a testing
session using their data flow testing tool, ASSET, is included
A new, empirically observed effect is introduced. Called ″the coupling effect,″ it may become a very important principle in practical testing activities. The idea is that programs appear to have the property - the ″coupling effect″ - that tests designed to detect simple kinds of errors are also effective in detecting much more complicated errors. This relationship may seem counter-intuitive, but the authors give a way of analyzing it through the use of program mutations (i. e. , incorrect variations from a correct program). One of the most interesting possibilities is that the mutation idea could form the basis for statistically inferring the likelihood of remaining errors in a program.
This paper describes an experimental program testing facility called the interactive semantic modelling system (ISMS). The ISMS is designed to allow experimentation with a wide variety of tools for collecting, analyzing, and displaying testing information. The design methodology is applicable to procedural programming languages, and Algol 60 is being used as the vehicle for elaboration of design principles and implementation techniques. This paper discusses the ISMS design, and describes the various types of analysis and display tools being developed to facilitate program testing. The ISM Preprocessor is described, an an example is presented to illustrate the data structures utilized in the ISMS.
Software quality enhancement can be achieved in the near term through use of a systematic program testing methodology. The methodology attempts to relate functional software testcases with formal software specifications as a means to achieve correspondence between the software and its specifications. To do this requires generation of appropriate testcase data.
Automatic testcase generation is based on a priori knowledge of two forms of internal structure information: a representation of the tree of subschema automatically identified from within each program text, and a representation of the iteration structure of each subschema. This partition of a large program allows for efficient and effective automatic testcase generation using straightforward backtracking techniques.
During backtracking a number of simplifying, consolidating, and consistency analyses are applied. The result is either (1) early recognition of the impossibility of a particular program flow, or (2) efficient generation of input variable specifications which cause the testcase to traverse each portion of the required program flow.
A number of machine output examples of the backtracking facility are given, and the general effectiveness of the entire process is discussed.
Test data generation is one of the most technically challenging steps of testing software, but most commercial systems currently incorporate very little automation for this step. This paper presents results from a project that is trying to find ways to incorporate test data generation into practical test processes, The results include a new procedure for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and dynamically 'pushes' the values through the control-how graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed. This procedure uses new analysis techniques, offers improvements over previous research results in constraint-based testing, and combines several steps into one coherent process. The dynamic nature of this procedure yields several benefits, Moving through the control flow graph dynamically allows path constraints to be resolved immediately, which is more efficient both in space and time, and more often successful than constraint-based testing, This new procedure also incorporates an intelligent search technique based on bisection. The dynamic nature of this procedure also allows certain improvements to be made in the handling of arrays, loops, and expressions; language features that are traditionally difficult to handle in test data generation systems, The paper presents the test data generation procedure, examples to explain the working of the procedure, and results from a proof-of-concept implementation. Copyright (C) 1999 John Wiley & Sons, Ltd.
Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.
A description is given of ASSET, a tool which uses information about a program's data flow to aid in selecting test data for the program and to evaluate test data adequacy. ASSET is based on the family of data flow test selection and test data adequacy criteria developed by S. Rapps and E. J. Weyuker (1981). ASSET accepts as input a program written in a subset of Pascal, a set of test data, and one of the data flow adequacy criteria and indicates to what extent the criterion has been satisfied by the test data.
Although there are many techniques and tools available to support the software testing process, one of the most crucial parts of testing, generating the test data, is usually done by hand. Test data generation is one of the most technically challenging steps in testing software, but unfortunately, most practical systems do not incorporate any automation in this step. This paper presents a new method for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and "pushes" the values through the control-flow graph of the program in a dynamic way, modifying the sets of values as branches in the program are taken. This method is an outgrowth of previous research in constraint-based testing, and combines several of the steps that were previously separate into one coherent process. The major difference is that this method is dynamic, and resolves path constraints immediately; it also includes an intelligent search technique, and improved handling of arrays, loops, and pointers.
SELECT is an experimental system for assisting in the formal systematic debugging of programs. It is intended to be a compromise between an automated program proving system and the current ad hoc debugging practice, and is similar to a system being developed by King et al. of IBM. SELECT systematically handles the paths of programs written in a LISP subset that includes arrays. For each execution path SELECT returns simplified conditions on input variables that cause the path to be executed, and simplified symbolic values for program variables at the path output. For conditions which form a system of linear equalities and inequalities SELECT will return input variable values that can serve as sample test data. The user can insert constraint conditions, at any point in the program including the output, in the form of symbolically executable assertions. These conditions can induce the system to select test data in user-specified regions. SELECT can also determine if the path is correct with respect to an output assertion. We present four examples demonstrating the various modes of system operation and their effectiveness in finding bugs. In some examples, SELECT was successful in automatically finding useful test data. In others, user interaction was required in the form of output assertions. SELECT appears to be a useful tool for rapidly revealing program errors, but for the future there is a need to expand its expressive and deductive power.
An abstract is not available.
Symbolic evaluation is a program analysis method that represents a program's computations and domain by symbolic expressions. In this paper a general functional model of a program is first presented. Then, three related methods of symbolic evaluation, which create this functional description from a program, are described: path-dependent symbolic evaluation provides a representation of a specified path; dynamic symbolic evaluation, which is more restrictive but less costly than path-dependent symbolic evaluation, is a data-dependent method; and global symbolic evaluation, which is the most general yet most costly method, captures the functional behavior of an entire program when successful. All three methods have been implemented in experimental systems. Some of the major implementation concerns, which include effectively representing loops, determining path feasibility, dealing with compound data structures, and handling routine invocations, are explained. The remainder of the paper surveys the range of applications to which symbolic evaluation techniques are being applied. The current and potential role of symbolic evaluation in verification, testing, debugging, optimization, and software development is explored.
Program slicing is a method for automatically decomposing programs by analyzing their data flow and control flow. Starting from a subset of a program's behavior, slicing reduces that program to a minimal form which still produces that behavior. The reduced program, called a ``slice,'' is an independent program guaranteed to represent faithfully the original program within the domain of the specified subset of behavior. Some properties of slices are presented. In particular, finding statement-minimal slices is in general unsolvable, but using data flow analysis is sufficient to find approximate slices. Potential applications include automatic slicing tools for debuggng and parallel processing of slices.
The Godzilla automatic test data generator is an integrated collection of tools that implements a relatively new test data generation method—constraint-based testing—that is based on mutation analysis. Constraint-based testing integrates mutation analysis with several other testing techniques, including statement coverage, branch coverage, domain perturbation, and symbolic evaluation. Because Godzilla uses a rule-based approach to generate test data, it is easily extendible to allow new testing techniques to be integrated into the current system. This article describes the system that has been built to implement constraint-based testing. Godzilla's design emphasizes orthogonality and modularity, allowing relatively easy extensions. Godzilla's internal structure and algorithms are described with emphasis on internal structures of the system and the engineering problems that were solved during the implementation.
Test data generation in program testing is the process of identifying a set of test data which satisfies a given testing criterion. Existing pathwise test data generators proceed by selecting program paths that satisfy the selected criterion and then generating program inputs for these paths. One of the problems with this approach is that unfeasible paths are often selected; as a result, significant computational effort can be wasted in analysing those paths. In this paper, an approach to test data generation, referred to as a dynamic approach for test data generation, is presented. In this approach, the path selection stage is eliminated. Test data are derived based on the actual execution of the program under test and function minimization methods. The approach starts by executing a program for an arbitrary program input. During program execution for each executed branch, a search procedure decides whether the execution should continue through the current branch or an alternative branch should be taken. If an undesirable execution flow is observed at the current branch, then a real-valued function is associated with this branch, and function minimization search algorithms are used to locate values of input variables automatically, which will change the flow of execution at this branch.
Software testing is very labor intensive and expensive and accounts for a significant portion of software system development cost. If the testing process could be automated, the cost of developing software could be significantly reduced. Test data generation in program testing is the process of identifying a set of test data that satisfies a selected testing criterion, such as statement coverage and branch coverage. In this article we present a chaining approach for automated software test data generation which builds on the current theory of execution-oriented test data generation. In the chaining approach, test data are derived based on the actual execution of the program under test. For many programs, the execution of the selected statement may require prior execution of some other statements. The existing methods of test data generation may not efficiently generate test data for these types of programs because they only use control flow information of a program during the search process. The chaining approach uses data dependence analysis to guide the search process, i.e., data dependence analysis automatically identifies statements that affect the execution of the selected statement. The chaining approach uses these statements to form a sequence of statements that is to be executed prior to the execution of the selected statement. The experiments have shown that the chaining approach may significantly improve the chances of finding test data as compared to the existing methods of automated test data generation.
The extension of dataflow testing to interprocedural testing is described. This was done by developing both an analysis technique that computes the required interprocedural definition-use information, for both direct and indirect dependencies and a testing technique that uses this information in selecting and executing the subpaths across procedure boundaries. A testing tool that implements this technique is presented. For the interprocedural dataflow analysis, the technique summarizes the individual procedures' definition and use information at call sites and then propagates this information throughout the interacting procedures. By efficiently computing the interprocedural data dependencies before testing, the approach lets the testing tool use existing path-selection techniques based on dataflow for interprocedural testing. To track the execution path, the technique recognizes the calls to and returns from procedures and handles the association of various names with a definition as the execution path is being inspected. The technique handles recursive procedures and supports separate compilation of procedures.
Program slicing is a method for automatically decomposing programs by analyzing their data flow and control flow. Starting from a subset of a program's behavior, slicing reduces that program to a minimal form which still produces that behavior. The reduced program, called a 'slice', is an independent program guaranteed to represent faithfully the original program within the domain of the specified subset of behavior. Some properties of slices are presented. In particular, finding statement-minimal slices is in general unsolvable, but using data flow analysis is sufficient to find approximate slices. Potential applications include automatic slicing tools for debugging and parallel processing of slices.
Symbolic testing and a symbolic evaluation system called DISSECT are described. The principle features of DISSECT are outlined. The results of two classes of experiments in the use of symbolic evaluadon are summarized. Several classes of program errors are defined and the reliability of symbolic testing in finding bugs is related to the classes of errors. The relationship of symbolic evaluation systems like DISSECT to classes of program errors and to other kinds of program testing and program analysis tools is also discussed. Desirable improvements in DISSECT, whose importance was revealed by the experiments, are mentioned.
This paper describes a system that attempts to generate test data for programs written in ANSI Fortran. Given a path, the system symbolically executes the path and creates a set of constraints on the program's input variables. If the set of constraints is linear, linear programming techniques are employed to obtain a solution. A solution to the set of constraints is test data that will drive execution down the given path. If it can be determined that the set of constraints is inconsistent, then the given path is shown to be nonexecutable. To increase the chance of detecting some of the more common programming errors, artificial constraints are temporarily created that simulate error conditions and then an attempt is made to solve each augmented set of constraints. A symbolic representation of the program's output variables in terms of the program's input variables is also created. The symbolic representation is in a human readable form that facilitates error detection as well as being a possible aid in assertion generation and automatic program documentation.
A testing system based on a compiler is described, in which compiled code is utilized under interactive control, but ″semantic″ errors are reported in the style of conventional syntax errors. The implementation is entirely in the high-level language on which the system is based, using some novel ideas for improving documentation without sacrificing efficiency.
A program slice consists of the parts of a program that (potentially) affect the values computed at some point of interest, referred to as a slicing criterion. The task of computing program slices is called program slicing. The original definition of a program slice was presented by Weiser in 1979. Since then, various slightly different notions of program slices have been proposed, as well as a number of methods to compute them. An important distinction is that between a static and a dynamic slice. The former notion is computed without making assumptions regarding a program's input, whereas the latter relies on some specific test case. Procedures, arbitrary control flow, composite datatypes and pointers, and interprocess communication each require a specific solution. We classify static and dynamic slicing methods for each of these features, and compare their accuracy and efficiency. Moreover, the possibilities for combining solutions for different features are investigated.
Mutation testing is a technique for testing software units that
has great potential for improving the quality of testing, and thereby
increasing our ability to assure the high reliability of critical
software. The paper presents a technique that uses mathematical
constraints to automatically detect equivalent mutant programs. The
paper also describes how the approach is used for the feasible path
problem. The paper describes how test criteria are formalized as
mathematical constraint systems, how equivalent mutants are represented
as infeasible constraints, and how infeasible constraints are detected.
A proof of concept implementation has been developed to demonstrate this
technique, and experimental results from using this tool are presented.
Limitations of the system and the method are described, and proposals
for improvements are made
Describes ATAC (Automatic Test Analysis for C), a tool for data
flow coverage testing of C programs. ATAC is being used as a research
instrument at Purdue and Bellcore and as a software development tool at
Bellcore. The authors discuss the design of ATAC, a preliminary view of
its uses in development, and its research uses
Software quality can be engineered under statistical quality control and delivered with better quality. The Cleanroom process gives management an engineering approach to release reliable products.
The extension of dataflow testing to interprocedural testing is described. This was done by developing both an analysis technique that computes the required interprocedural definition-use information, for both direct and indirect dependencies and a testing technique that uses this information in selecting and executing the subpaths across procedure boundaries. A testing tool that implements this technique is presented. For the interprocedural dataflow analysis, the technique summarizes the individual procedures' definition and use information at call sites and then propagates this information throughout the interacting procedures. By efficiently computing the interprocedural data dependencies before testing, the approach lets the testing tool use existing path-selection techniques based on dataflow for interprocedural testing. To track the execution path, the technique recognizes the calls to and returns from procedures and handles the association of various names with a definition as the execution path is being inspected. The technique handles recursive procedures and supports separate compilation of procedures
Symbolic evaluation is a form of static program analysis in which symbolic expressions are used to denote the values of program variables and computations. It does not require the user to specify which path at a conditional branch to follow nor how many cycles of a loop to consider. Instead, a symbolic evaluator uses conditional expressions to represent the uncertainty that arises from branching and develops and attempts to solve recurrence relations that describe the behavior of loop variables.
Symbolic testing and a symbolic evaluation system called DISSECT are described. The principle features of DISSECT are outlined. The results of two classes of experiments in the use of symbolic evaluadon are summarized. Several classes of program errors are defined and the reliability of symbolic testing in finding bugs is related to the classes of errors. The relationship of symbolic evaluation systems like DISSECT to classes of program errors and to other kinds of program testing and program analysis tools is also discussed. Desirable improvements in DISSECT, whose importance was revealed by the experiments, are mentioned.
if finite input-output specifications are added to the syntax of programs, these specifications can be verified at compile time. Programs which carry adequate tests with them in this way should be resistant to maintenance errors. If the specifications are independent of program details they are easy to give, and unlikely to contain errors in common with the program. Furthermore, certain finite specifications are maximal in that they exercise the control and expression structure of a program as well as any tests can.
Software validation through testing will continue to be a very important tool for ensuring correctness of large scale software systems. Automation of testing tools can greatly enhance their power and reduce testing cost. In this paper, techniques for automated test data generation are discussed. Given a program graph, a set of paths are identified to satisfy some given testing criteria. When a path or a program segment is specified, symbolic execution is used for generating input constraints which define a set of inputs for executing this path or segment. Problems encountered in symbolic execution are discussed. A new approach for resolving array reference ambiguities and a procedure for generating test inputs satisfying input constraints are proposed. References to arrays are recorded in a table. during symbolic execution and ambiguities are resolved when test data are generated to evaluate the subscript expressions. The implementation of a test data generator for Fortran programs incorporating these techniques is also described.
This paper describes a system that attempts to generate test data for programs written in ANSI Fortran. Given a path, the system symbolically executes the path and creates a set of constraints on the program's input variables. If the set of constraints is linear, linear programming techniques are employed to obtain a solution. A solution to the set of constraints is test data that will drive execution down the given path. If it can be determined that the set of constraints is inconsistent, then the given path is shown to be nonexecutable. To increase the chance of detecting some of the more common programming errors, artificial constraints are temporarily created that simulate error conditions and then an attempt is made to solve each augmented set of constraints. A symbolic representation of the program's output variables in terms of the program's input variables is also created. The symbolic representation is in a human readable form that facilitates error detection as well as being a possible aid in assertion generation and automatic program documentation.
Def-use analysis links possible value-setting statements for a variable (i.e. definitions) to potential value-fetches (i.e. uses) of that value. This paper describes the first algorithm that calculates accurate interprocedural def-use associations in C software systems. Our algorithm accounts for program-point-specific pointer-induced aliases, though it is currently limited to programs using a single level of indirection. We prove the NP-hardness of the interprocedural reaching definitions problem and describe the approximations made by our polynomial-time algorithm. Initial empirical results are also presented
An alternative approach to test-data generation based on actual execution of the program under test, function-minimization methods and dynamic data-flow analysis is presented. Test data are developed for the program using actual values of input variables. When the program is executed, the program execution flow is monitored. If during program execution an undesirable execution flow is observed then function-minimization search algorithms are used to automatically locate the values of input variables for which the selected path is traversed. In addition, dynamic data-flow analysis is used to determine those input variables responsible for the undesirable program behavior, significantly increasing the speed of the search process. The approach to generating test data is then extended to programs with dynamic data structures and a search method based on dynamic data-flow analysis and backtracking is presented. In the approach described, values of array indexes and pointers are known at each step of program execution; this information is used to overcome difficulties of array and pointer handling
A methodology for generating program test data is described. The methodology is a model of the test data generation process and can be used to characterize the basic problems of test data generation. It is well defined and can be used to build an automatic test data generation system. The methodology decomposes a program into a finite set of classes of paths in such a way that an intuitively complete set of test cases would cause the execution of one path in each class. The test data generation problem is theoretically unsolvable: there is no algorithm which, given any class of paths, will either generate a test case that causes some path in that class to be followed or determine that no such data exist. The methodology attempts to generate test data for as many of the classes of paths as possible. It operates by constructing descriptions of the input data subsets which cause the classes of paths to be followed. It transforms these descriptions into systems of predicates which it attempts to solve. Copyright © 1975 by The Institute of Electrical and Electronics Engineers, Inc.
A symbolic interpreter can provide a broad and useful spectrum of services to a programmer attempting to determine a program's correctness. With no information beyond the program, tests can be performed with particular values. In addition, symbolic tests can be performed to determine the program's behavior over a class of executions each following the same control-flow path. If partial specifications can be supplied, test results can be checked against them by evaluating them over the current program variables' (symbolic) values and by attempting to establish their truth for the particular input subspace described by the pc. A symbolic interpreter can give an indication of the coverage of a set of non-symbolic tests and help in the generation of additional distinct tests. Also, it can be used to analyze the behavior of loops and help in the formulation of inductive arguments necessary for a proof of correctness. Finally, if the inductive arguments can be formalized and loop-invariant assertions added to the program, the interpreter can assist in a complete formal proof of the program's correctness.
Although there are many techniques and tools available to support the software testing process, one of the most crucial parts of testing, generating the test data, is usually done by hand. Test data generation is one of the most technically challenging steps in testing software, but unfortunately, most practical systems do not incorporate any automation in this step. This paper presents a new method for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and "pushes" the values through the control-flow graph of the program in a dynamic way, modifying the sets of values as branches in the program are taken. This method is an outgrowth of previous research in constraint-based testing, and combines several of the steps that were previously separate into one coherent process. The major difference is that this method is dynamic, and resolve...
Software Engineering
- I Sommerville
I. Sommerville. Software Engineering. Addison-Wesley Publishing Company Inc., 4th edition,
1992.
Select{a formal system for testing and debugging programs by symbolic execution
- R S Boyer
- B Elpas
- K N Levitt
R. S. Boyer, B. Elpas, and K. N. Levitt. Select{a formal system for testing and debugging
programs by symbolic execution. In Proceedings of the International Conference on Reliable
Software, June 1975. SIGPLAN Notices, vol. 10, no. 6.
An extended overview of the Mothra software testing environment
- R A Demillo
- D S Guindi
- K N King
- W M Mccracken
- A J Utt
R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. O utt. An extended
overview of the Mothra software testing environment. In Proceedings of the Second Workshop
on Software Testing, Veri cation, and Analysis, pages 142{151, Ban Alberta, July 1988.
IEEE Computer Society Press.