Content uploaded by Jan Peleska
Author content
All content in this area was uploaded by Jan Peleska on May 17, 2016
Content may be subject to copyright.
A preview of the PDF is not available
Test automation for safety-critical systems: Industrial application and future developments
Abstract
Design, execution and evaluation of tests for safety-critical systems require considerable effort and skill and consume a large part of today's development costs. Due to the growing complexity of control systems, it has to be expected that their trustworthy test will become unmanageable in the future, if only conventional techniques, requiring a high degree of human interaction during the test process, are applied. In this article, we will focus on test automation for reactive real-time systems, with emphasis on Hardware-in-the-Loop tests analyzing the behaviour of combined software and hardware components. To illustrate possible approaches for this test problem, we describe a concept based on specifications written in Real-Time CSP. For the implementation of test generation and evaluation algorithms transition system representations are used, as can be obtained by Formal Systems' FDR tool. An industrial application of the method is presented and used for the evaluation of the benefits of formal methods-based testing in comparison with conventional techniques. Furthermore, we will indicate research topics in this field which are likely to become important for further improvements of the test process. Specifically, the benefits arising from an approach combining formal verification and testing will be discussed. Our presentation aims less at promoting a specific solution, but tries to illustrate the basic problems to be tackled with any formal method, when trying to develop test automation concepts to be applied in the context of reactive systems.
... In this case, the path π would not be a suitable witness for testing this requirement. 7 This concept has already been used in 1996 in the VVT-RT Tool [27] whose commercial version is called RT-Tester today www.verified.de. the test execution might take too much time, whereas online testing delivers information about passed and failed test steps right from the start. ...
In this position paper, a novel approach to testing complex autonomous transportation systems (ATS) in the automotive, avionic, and railway domains is described. It is intended to mitigate some of the most critical problems regarding verification and validation (V&V) effort for ATS. V&V is known to become infeasible for complex ATS, when using conventional methods only. The approach advocated here uses complete testing methods on the module level, because these establish formal proofs for the logical correctness of the software. Having established logical correctness, system-level tests are performed in simulated cloud environments and on the target system. To give evidence that 'sufficiently many' system tests have been performed with the target system, a formally justified coverage criterion is introduced. To optimise the execution of very large system test suites, we advocate an online testing approach where multiple tests are executed in parallel, and test steps are identified on-the-fly. The coordination and optimisation of these executions is achieved by an agent-based approach. Each aspect of the testing approach advocated here is shown to either be consistent with existing standards for development and V&V of safety-critical transportation systems, or it is justified why it should become acceptable in future revisions of the applicable standards.
... In this case, the path π would not be a suitable witness for testing this requirement. 7 This concept has already been used in 1996 in the VVT-RT Tool [27] whose commercial version is called RT-Tester today www.verified.de. the test execution might take too much time, whereas online testing delivers information about passed and failed test steps right from the start. ...
In this position paper, a novel approach to testing complex autonomous transportation systems (ATS) in the automotive, avionic, and railway domains is described. It is intended to mitigate some of the most critical problems regarding verification and validation (V&V) effort for ATS. V&V is known to become infeasible for complex ATS, when using conventional methods only. The approach advocated here uses complete testing methods on the module level, because these establish formal proofs for the logical correctness of the software. Having established logical correctness, system-level tests are performed in simulated cloud environments and on the target system. To give evidence that 'sufficiently many' system tests have been performed with the target system, a formally justified coverage criterion is introduced. To optimise the execution of very large system test suites, we advocate an online testing approach where multiple tests are executed in parallel, and test steps are identified on-the-fly. The coordination and optimisation of these executions is achieved by an agent-based approach. Each aspect of the testing approach advocated here is shown to either be consistent with existing standards for development and V&V of safety-critical transportation systems, or it is justified why it should become acceptable in future revisions of the applicable standards.
... Here, the authors are concentrated on testing for traces and failures refinement. In [Pel96], J. Peleska presents a pioneering work on CSP-based testing. ...
... Monitoring an application to ensure the consistency with highlevel requirement specifications is an efficient approach for correctness checking and it can also be used to detect runtime errors or as a verification technique. Recently, there has been increasing attention from the research community to the design of monitors which can be used to assure the correctness of a system at runtime [18, 17, 15, 10, 14]. Those monitoring approaches usually add instrumentation code to the program to collect interesting data at runtime. ...
With current trends towards more complex software system and use of higher level languages, a monitoring technique is of increasing importance for the areas such as performance enhancement, dependability, correctness checking and so on. In this paper, we present a formal specification-based online monitoring technique. The key idea of our technique is to build a linking system, which connects a specification animator and a program debugger. The required information about dynamic behaviors of the formal specification and concrete implementation of a target system is obtained from the animator and the debugger. Based on that information, the judgment on the consistency of the concrete implementation with the formal specification will be provided. Not embedding any instrumentation code into the target system, our monitoring technique will not alter the dynamic behavior of the target system. Animating the formal specification, rather than annotating the target system with extra formal specifications, our monitoring technique separates the implementation-dependent description of the monitored objects and the formal requirement specification of them
Testing reactive systems is important to guarantee a precise and robust software development process. Pushdown reactive systems are complex applications where the interaction with the environment is regulated by a pushdown memory and, in general, can be specified by the formalism of Input/Output Visibly Pushdown Labeled Transition System (IOVPTS). A conformance checking can then be applied to verify whether an implementation is in compliance to a specification using an appropriate conformance relation. In this work, we establish a conformance relation based on Visibly Pushdown Languages (VPLs) to model sets of desirable and undesirable behaviors of systems. Further, we show that test suites with a complete fault coverage can be generated using this new conformance relation for pushdown reactive systems.
In recent work, Cavalcanti and her group, including Miyazawa and Timmis, have developed a CSP-based framework for model-based engineering of robotic systems, called RoboStar. In this paper, we describe our current effort to ally RoboStar and RT-Tester, an award-winning tool that embodies many of Jan Peleska’s beautiful results on formal testing. With our work, RoboStar users can benefit from the testing infrastructure of RT-Tester to run simulations and tests generated using the RoboStar automated techniques. The testing primitives of RT-Tester simplify the implementation of test cases, and the RT-Tester execution engine provides state-of-the-art high-performance real-time facilities to carry out and report the traceable results of test experiments.
KeywordsTestingFormal modelsCSPAutomation
The process algebra CSP has been studied as a modeling notation for test derivation. Work has been developed using its trace and failure semantics, and their refinement notions as conformance relations. In this paper, we propose a procedure for online test generation for selection of finite test sets for traces refinement from CSP models, based on the notion of fault domains, that is, focusing on the set of faulty implementations of interest. We investigate scenarios where the verdict of a test campaign can be reached after a finite number of test executions. We illustrate the usage of the procedure with a small case study.
Since 1985, CSP has been applied by the author, his research team at Bremen University and verification engineers at Verified Systems International to a variety of "real-world" projects. These include the veri- fication of high-availability database servers, of fault-tolerant computers now operable in the International Space Station, hardware-in-the-loop tests for the novel Airbus A380 aircraft controller family and confor- mance tests for the European Train Control System. Illustrated by ex- amples from these projects, we highlight important aspects of the CSP language design, its semantics and tool support, and describe the impact of these features on the quality and eciency of verification and test- ing. New requirements with regard to the test of hybrid control systems, the demand for executable formal specifications, as well as the ongoing discussion about the practical applicability of formal methods have led to the development of new specification formalisms. We sketch some key decisions in the formalism design and indicate how some of the funda- mental properties of CSP have been adopted, while others have been deliberately discarded in these new developments.
This book provides general and systematic introduction to the semantics of concurrent systems. The author presents his own theory of the behavioral semantics of processes ( testing equivalence ) and original results in example languages for distributed processes. The problems addressed are motivated from the standpoint of computer science, and all the required algebraic concepts are covered.