Figure 4 - uploaded by Teemu Kanstrén
Content may be subject to copyright.
Source publication
We present a model-based testing approach to support automated test
generation with domain-specific concepts. This includes a language expert who
is an expert at building test models and domain experts who are experts in the
domain of the system under test. First, we provide a framework to support the
language expert in building test models using a...
Citations
... Our solution relies on the page object pattern to represent app screens as classes [31], and the OSMO tool to design, generate, and execute the test cases [27]. However, we extended the OSMO tool to be flexible regarding the agility of mobile development, i.e., to cope with new screens in apps that are not yet defined in the test models. ...
... Various automated MBT tools are available, e.g. JTorX [3,33], Phact [14], OSMO [23], APSL [34], and RT-Tester [26]. ...
In model-based testing, we may have to deal with a non-deterministic model, e.g. because abstraction was applied, or because the software under test itself is non-deterministic. The same test case may then trigger multiple possible execution paths, depending on some internal decisions made by the software. Consequently, performing precise test analyses, e.g. to calculate the test coverage, are not possible.. This can be mitigated if developers can annotate the model with estimated probabilities for taking each transition. A probabilistic model checking algorithm can subsequently be used to do simple probabilistic coverage analysis. However, in practice developers often want to know what the achieved aggregate coverage is, which unfortunately cannot be re-expressed as a standard model checking problem. This paper presents an extension to allow efficient calculation of probabilistic aggregate coverage, and also of probabilistic aggregate coverage in combination with k-wise coverage.
... We report on a study to assess effectiveness in failure detection on mutations of classes that implement object protocols compared to other tools. Although there are many tools that use code abstractions to guide test case generation (i.e., Model Based Testing [58,63,82,93]) only some of them target the Java language (e.g., References [2,22,53,92,97]). However, in all cases, they require a full model of the unit under test to be provided manually. ...
A significant proportion of classes in modern software introduce or use object protocols, prescriptions on the temporal orderings of method calls on objects. This article studies search-based test generation techniques that aim to exploit a particular abstraction of object protocols (enabledness preserving abstractions (EPAs)) to find failures. We define coverage criteria over an extension of EPAs that includes abnormal method termination and define a search-based test case generation technique aimed at achieving high coverage. Results suggest that the proposed case generation technique with a fitness function that aims at combined structural and extended EPA coverage can provide better failure-detection capabilities not only for protocol failures but also for general failures when compared to random testing and search-based test generation for standard structural coverage.
... Thereby, the state machine of the test model is implemented in a common programming language using a modelbased testing framework. We used the framework OSMO Tester [14]. It allows writing the test model in Java (Fig. 4) and generating test sequences by running the resulting program. ...
Model-based testing (MBT) has been proposed as an effective and versatile approach for testing graphical user interfaces (GUIs) by automatically generating executable test cases from a model of the GUI. Model-based GUI testing has received increasing attention in research, but it is still rarely applied in practice. In this paper, we present our experiences and share the lessons we learned from successfully introducing MBT for GUI testing in three industry projects. We describe the underlying modeling approach, the development of tests models in joint workshops, the implementation of the test model in form of model programs, and the integration of MBT in the test automation architecture. The findings distilled from the three cases are summarized as lessons learned to support the adoption of a model-based approach for GUI testing in practice.
... Kanstrén and Puolitaival present a framework that involves the generation of a DSML from a test model to guide domain experts in creating test cases (Kanstrén and Puolitaival 2012). The generated DSML, or GUI alternative, can be used to express one test case, or multiple test cases. ...
Domain-specific modelling languages (DSML) enable domain users
to model systems in their problem domain, using concepts and
notations they are familiar with. The process of domain-specific
modelling (DSM) consists of two stages: a language engineering
stage where a DSML is created, and a system modelling stage where
the DSML is used. Because techniques such as metamodelling and
model transformation allow for a efficient creation of DSMLs, and
using DSMLs significantly increases productivity, DSM is very
suitable for early prototyping. Many systems that are modelled
using DSMLs are reactive, meaning that during their execution, they
respond to external input. Because of the complexity of input and
response behaviour of reactive systems, it is desirable to test models
as early as possible. However, while dedicated testing support for
specific DSMLs has been provided, no systematic support exists for
testing DSML models according to DSM principles.
In this paper, we introduce a technique to automatically generate
a domain-specific testing framework from an annotated DSML
definition. In our approach, the DSML definition consists of a metamodel,
a concrete syntax definition and operational semantics described
as a schedule of graph rewrite rules, thus covering a large
class of DSMLs. Currently, DSMLs with deterministic behaviour
are supported, but we provide an outlook to other (nondeterministic,
real-time or continuous-time) DSMLs. We illustrate the approach
with a DSML for describing an elevator controller. We evaluate
the approach and conclude that compared to the state-of-the-art,
our testing support is significantly less costly, and similar or better
(according to DSM principles) testing support is achieved. Additionally,
the generative nature of the approach makes testing support for
DSMLs less error-prone while catering the need for early testing.
... Our work covers test actions instead, spanning sequences of multiple function calls. Existing tools to generate test actions [9,15,16,21,28,29] define a programming interface against which the model is coded; the programmer defines classes and functions that implement the semantics of the model. It was shown that this introduces a layer of abstraction in the model that makes it difficult to clearly express the behavior of event-driven systems, which include databases, file systems, and cloud computing middleware [2]. ...
... the structure of the model is defined via annotated methods [15,16]. NModel and its successor SpecExplorer [29] also distinguish themselves by splitting the model into a scenario model (generating the tests) and a contract model (verifying the outcome), while most other tools combine both aspects in one model. ...
Model-based testing derives test executions from an abstract model that describes the system behavior. However, existing approaches are not tailored to event-driven or input/output-driven systems. In particular, there is a need to support non-blocking I/O operations, or operations throwing exceptions when communication is disrupted.
Our new tool “Modbat” is specialized for testing systems where these issues are common. Modbat uses extended finite-state machines to model system behavior. Unlike most existing tools, Modbat offers a domain-specific language that supports state machines and exceptions as first-class constructs. Our model notation also handles non-determinism in the system under test, and supports alternative continuations of test cases depending on the outcome of non-deterministic operations.
These features allow us to model a number of interesting libraries succinctly. Our experiments show the flexibility of Modbat and how language support for model features benefits their correct use.
In model-based testing (MBT) we may have to deal with a non-deterministic model, e.g. because abstraction was applied, or because the software under test itself is non-deterministic. The same test case may then trigger multiple possible execution paths, depending on some internal decisions made by the software. Consequently, performing precise test analyses, e.g. to calculate the test coverage, are not possible. This can be mitigated if developers can annotate the model with estimated probabilities for taking each transition. A probabilistic model checking algorithm can subsequently be used to do simple probabilistic coverage analysis. However, in practice developers often want to know what the achieved aggregate coverage is, which unfortunately cannot be re-expressed as a standard model checking problem. This paper presents an extension to allow efficient calculation of probabilistic aggregate coverage, and moreover also in combination with k-wise coverage.
Model-based testing (MBT) is a technique for generating test cases from test models. One of the benefits of MBT is the ability to have a computer generate and execute extensive test sets from the test models, achieving high coverage. However, when such large test sets are automatically generated and executed, the resulting failure traces can be very large and difficult to debug for root cause analysis. In this paper, we present a technique for minimizing the length of a failure trace, creating variants of it, and for pattern mining the trace variants to assist in root cause analysis. We demonstrate the technique on a model of a GSM SIM card.
