No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Maduración de Conocimiento Mediante una Familia de Experimentos
Abstract
Resumen. Este artículo parte de un estudio preliminar llevado a cabo sobre los experimentos que se han realizado con técnicas de pruebas. El objetivo es exa-minar con precisión una de las familias que se identificaron para averiguar si los mecanismos de maduración de conocimiento que se han utilizado en la fa-milia han sido suficientes para lograr un fragmento consolidado de conocimien-to. La familia elegida ha sido la más aventajada de las que se encontraron, re-sultando ser la que compara la efectividad relativa de técnicas pertenecientes a los grupos funcional, estructural y revisión de código. Tras detallar cada uno de los experimentos que componen la familia, se hace un análisis comparativo de los mismos, así como de los mecanismos de maduración de conocimiento usa-dos en la misma. Este análisis ha arrojado a la luz que los mecanismos utiliza-dos en la familia: paquetes de laboratorio y asociación a experimentos anterio-res, no son suficientes, siendo necesaria una mayor interacción o cooperación entre experimentadores.
... For reasons of space, this description is founded on the results of these replications and the ITIs used. The full details of these replications are given in [19]. The goal of these experiments is to examine the relative effectiveness of different code evaluation techniques. ...
... In this case, the programs are in C,. We assume that the ITI used for this replication was publications We have tried to combine the results from the replications described here, the outcome of which can be found in [19]. However, it was fruitless. ...
... The first is a general part that reflects how the item of knowledge examined by the experiment is matured and fashioned by the experiences have led us to believe that the replicating researchers are obliged in this case to lend their attention to too many details, without knowing exactly which will and which will not be relevant for their replication. The ICM proposed here is halfway between Occasional Collaboration [3], [17] and Heavy Collaboration [19]. It improves upon Occasional Collaboration in that it establishes permanent channels of communication between researchers. ...
As in other spheres of science and technology, the replication of experiments in SE is an indispensable task. However, replication is extremely difficult in SE. This is primarily due to the complexity of the context in which experiments are run. The success in accurately describing the context of an experiment will be later reflected in the possibility of combining the results of the replications. If the context is not properly characterised, it will be impossible to isolate the variables causing any discrepancies between the results of the different replications. In this paper, we have used and then compared the instruments for transmitting information about experiments proposed in the literature to conduct replications of SE experiments. Based on this experience, we propose an improved instrument.
The replication of experiments is a key undertaking in SE. Successful replications enable a discipline's body of knowledge to grow, as the results are added to those of earlier replications. However, replication is extremely difficult in SE, primarily because it is difficult to get a setting that is exactly the same as in the original experiment. Consequently, changes have to be made to the experiment to adapt it to the new site. To be able to replicate an experiment, information also has to be transmitted (usually orally and in writing) between the researchers who ran the experiment earlier and the ones who are going to replicate the experiment. This article examines the influence of the type of communication there is between experimenters on how successful a replication is. We have studied three replications of the same experiment in which different types of communication were used.
Testing technique-related empirical studies have been performed for 25 years. We have managed to accumulate a fair number of experiments in this time, which might lead us to think that we now could have a sizeable empirically backed body of knowledge (BoK) on testing techniques. However, the experiments in this field have some flaws, and, consequently, the empirical BoK we have on testing techniques is far from solid. In this paper, we use the results of a survey that we did on empirical testing techniques studies to identify and discuss solutions that could lead to the formation of a solid empirical BoK. The solutions are related to two fundamental experimental issues: (1) the rigorousness of the experimental design and analysis, and (2) the need for a series of community-wide agreements to coordinate empirical research and assure that studies ratify and complement each other.
Mature knowledge allows engineering disciplines the achievement of predictable results. Unfortunately, the type of knowledge used in software engineering can be considered to be of a relatively low maturity, and developers are guided by intuition, fashion or market-speak rather than by facts or undisputed statements proper to an engineering discipline. Testing techniques determine different criteria for selecting the test cases that will be used as input to the system under examination, which means that an effective and efficient selection of test cases conditions the success of the tests. The knowledge for selecting testing techniques should come from studies that empirically justify the benefits and application conditions of the different techniques. This paper analyzes the maturity level of the knowledge about testing techniques by examining existing empirical studies about these techniques. We have analyzed their results, and obtained a testing technique knowledge classification based on their factuality and objectivity, according to four parameters.
In software engineering, we distinguish valid ideas from false beliefs by applying the test of time. We judge an idea's certainty by whether or not people use it. Confronted by a confusing array of options for producing software, software engineers need proof of a particular approach or technique's superiority. They must know the clear-cut benefits of one approach versus another. This need to work with facts rather than assumptions is a property of any engineering discipline. Software engineering has the tools to transition from working with conjecture to working with fact. Widely using these tools will bring software engineering an important step closer toward maturity.
Experimentation in software engineering is necessary but
difficult. One reason is that there are a large number of context
variables and, so, creating a cohesive understanding of experimental
results requires a mechanism for motivating studies and integrating
results. It requires a community of researchers that can replicate
studies, vary context variables, and build models that represent the
common observations about the discipline. The paper discusses the
experience of the authors, based upon a collection of experiments, in
terms of a framework for organizing sets of related studies. With such a
framework, experiments can be viewed as part of common families of
studies, rather than being isolated events. Common families of studies
can contribute to important and relevant hypotheses that may not be
suggested by individual experiments. A framework also facilitates
building knowledge in an incremental manner through the replication of
experiments within families of studies. To support the framework, the
paper discusses the experiences of the authors in carrying out empirical
studies, with specific emphasis on persistent problems encountered in
experimental design, threats to validity, criteria for evaluation, and
execution of experiments in the domain of software engineering
We replicated a controlled experiment first run in the early 1980's to evaluate the effectiveness and efficiency of 50 student subjects who used three defectdetection techniques to observe failures and isolate faults in small C programs. The three techniques were code reading by stepwise abstraction, functional (black-box) testing, and structural (white-box) testing. Two internal replications showed that our relatively inexperienced subjects were similarly effective at observing failures and isolating faults with all three techniques. However, our subjects were most efficient at both tasks when they used functional testing. Some significant differences among the techniques in their effectiveness at isolating faults of different types were seen. These results suggest that inexperienced subjects can apply a formal verification technique (code reading) as effectively as an execution-based validation technique, but they are most efficient when using functional testing. 1 Introduction Sof...
This report describes an empirical study comparing three defect detection techniques: a) code reading by stepwise abstraction, b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using branch coverage. It is a replication of a study that has been carried out at least four times previously over the last 20 years. This study used 47 student subjects to apply the techniques to small C programs in a fractional factorial experimental design. The major findings of the study are: a) that the individual techniques are of broadly similar effectiveness in terms of observing failures and finding faults, b) that the relative effectiveness of the techniques depends on the nature of the program and its faults, c) these techniques are consistently much more effective when used in combination with each other. These results contribute to a growing body of empirical evidence that supports generally held beliefs about the effectiveness of defect detection techniques in software engineering.
The origin of the study described here is the experiment performed by Basili and Selby, further replicated by Kamsties and Lott, and once again by Wood et al. These experiments investigated the effectiveness and efficiency of different code evaluation techniques (functional and structural testing and code reading). The working hypotheses are the same in all three experiments, although some experimental conditions were changed. The experiments described here use the experiment package elaborated by Kamsties and Lott and examine some of the questions posed as a result of these experiments. Wood et al. concluded in their replication of the original study that the relative effectiveness of the techniques depends on the program and fault type. In fact, they suggest formulating a fault taxonomy based on technique sensitivity. Our study intends to compare the relative effectiveness of the testing techniques and to relate the testing techniques to fault types.
Basics of Software Engineering Experimentation is a practical guide to experimentation in a field which has long been underpinned by suppositions, assumptions, speculations and beliefs. It demonstrates to software engineers how Experimental Design and Analysis can be used to validate their beliefs and ideas. The book does not assume its readers have an in-depth knowledge of mathematics, specifying the conceptual essence of the techniques to use in the design and analysis of experiments and keeping the mathematical calculations clear and simple. Basics of Software Engineering Experimentation is practically oriented and is specially written for software engineers, all the examples being based on real and fictitious software engineering experiments.
This paper describes an experiment in program testing, employing 59 highly experienced data processing professionals using seven methods to test a small PL/I program. The results show that the popular code walkthrough/inspection method was as effective as other computer-based methods in finding errors and that the most effective methods (in terms of errors found and cost) employed pairs of subjects who tested the program independently and then pooled their findings. The study also shows that there is a tremendous amount of variability among subjects and that the ability to detect certain types of errors varies from method to method.
This study applies an experimentation methodology to compare three state-of-the-practice software testing techniques: a) code reading by stepwise abstraction, b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using 100 percent statement coverage criteria. The study compares the strategies in three aspects of software testing: fault detection effectiveness, fault detection cost, and classes of faults detected. Thirty-two professional programmers and 42 advanced students applied the three techniques to four unit-sized programs in a fractional factorial experimental design. The major results of this study are the following. 1) With the professional programmers, code reading detected more software faults and had a higher fault detection rate than did functional or structural testing, while functional testing detected more faults than did structural testing, but functional and structural testing were not different in fault detection rate. 2) In one advanced student subject group, code reading and functional testing were not different in faults found, but were both superior to structural testing, while in the other advanced student subject group there was no difference among the techniques. 3) With the advanced student subjects, the three techniques were not different in fault detection rate. 4) Number of faults observed, fault detection rate, and total effort in detection depended on the type of software tested. 5) Code reading detected more interface faults than did the other methods. 6) Functional testing detected more control faults than did the other methods.
Aggregating the results of 25 years of testing technique experiments. International Software Engineering Research Network
- S Vegas
- N Juristo
- A M Moreno
S. Vegas, N. Juristo and A.M. Moreno. Aggregating the results of 25 years of testing
technique experiments. International Software Engineering Research Network. ISERN'04.
16-18 August. Redondo Beach, California, USA.
Functional testing, structural testing and code reading: What fault type do they each detect? Empirical Methods and Studies in Software Engineering-Experiences from ESERNET
- N Juristo
- S Vegas
N. Juristo and S. Vegas. Functional testing, structural testing and code reading: What fault
type do they each detect? Empirical Methods and Studies in Software Engineering-Experiences from ESERNET. Springer-Verlag. Volume 2785. Chapter 12. Pages 235-261.2003.