Figure 5 - uploaded by Andreas Wortmann
Content may be subject to copyright.
Source publication
![Figure 1: Complexity drivers in automotive [6].](profile/Andreas-Wortmann/publication/325728663/figure/fig1/AS:636882530607111@1528856036797/Complexity-drivers-in-automotive-6_Q320.jpg)
Testing is crucial to successfully engineering reliable automotive software. The manual derivation of test cases from ambiguous textual requirements is costly and error-prone. Model-based development can reduce the test case derivation effort by capturing requirements in structured models from which test cases can be generated with reduced effort....
Contexts in source publication
Context 1
... the perceived benefits of MBTCC for engineers of dif- ferent backgrounds (R1), we conclude a positive attitude towards MBTCC ( cf. Tbl. 1). This positive attitude is largely independent of experience level, professional background, and focus of testing. Furthermore, the evaluation reveals that more than one third of the participants expect to benefit from MBTCC, despite a widely- adopted model-based approach is not in place (cf. Fig. ...
Context 2
... the basic environment for testing, the test engineers indi- cated that their test cases focus is generally the complete system with about 42.0%. The component tests are the second most indi- cated focus with 24.6% followed by the focus on subsystem (17.4%) and software (16%). For automatically performed test cases, the system environment association Vehicle in the Loop with 11.7% is less important. In contrast, the Hardware in the Loop (HiL) test cases are 81.6%. Software in the Loop (SiL) with about 5% and Model in the Loop (MiL) with 1.7% are less targeted with automatically performed tests. We also considered the sources for test case cre- ation (Fig. 5). For 78% the most important source for test cases are natural language requirements, followed by -in this order -volatile personal experience, existing test cases, error descriptions and UML or SysML models. To investigate the quality focus R2, the questionnaire requests the focused quality assurance (Fig. 6). The data reveals that three out of four test engineers identify functionality as most important. The indicated "importance" measured by the mean of robustness is followed by reliability and functional safety, efficiency, reusability, integrability, and ...
Context 3
... considerable portion of test engineers use old test cases, per- sonal experience, and error descriptions as basis for test case cre- ation (cf. Fig. 5). Creating new test cases from old ones suggests that new or changed requirements usually entail only small adoptions for new test cases especially for test cases in upcoming ...
Context 4
... cases, the system environment association Vehicle in the Loop with 11.7% is less important. In contrast, the Hardware in the Loop (HiL) test cases are 81.6%. Software in the Loop (SiL) with about 5% and Model in the Loop (MiL) with 1.7% are less targeted with automatically performed tests. We also considered the sources for test case cre- ation (Fig. 5). For 78% the most important source for test cases are natural language requirements, followed by -in this order -volatile personal experience, existing test cases, error descriptions and UML or SysML models. To investigate the quality focus R2, the questionnaire requests the focused quality assurance (Fig. 6). The data reveals that ...
Context 5
... conclude a positive attitude towards MBTCC ( cf. Tbl. 1). This positive attitude is largely independent of experience level, professional background, and focus of testing. Furthermore, the evaluation reveals that more than one third of the participants expect to benefit from MBTCC, despite a widelyadopted model-based approach is not in place (cf. Fig. ...
Context 6
... considerable portion of test engineers use old test cases, personal experience, and error descriptions as basis for test case creation (cf. Fig. 5). Creating new test cases from old ones suggests that new or changed requirements usually entail only small adoptions for new test cases especially for test cases in upcoming ...
Citations
... Rapid advances have led automakers to develop and produce vehicles with electronic components and embedded software rather than only hardware parts (HAGHIGHATKHAH et al., 2018b) engineering and integration with various stakeholders are required to meet high reliability requirements (ROSA, 2013) and the current needs of customers and the market, which have resulted in very complex systems and features. Embedded automotive software is usually a low-cost solution that can be quickly implemented compared with hardware development (KRIEBEL et al., 2018). An automotive vehicle is a complex system with many components such as sensors, actuators, and the Electronic Control Unit (ECU), and the hardware components are usually managed by embedded software (GRAF et al., 2013). ...
... Robert Bosch GmbH was third with three articles (BURTON et al., 2012;NEUROHR, 2021;BRAUN et al., 2014). Fourth and fifth were BMW with two articles (KRIEBEL, 2018;DEICKE et al., 2012) and General Motors also with two articles (PETRENKO et al., 2015;NEUROHR, 2021). Other companies such as FEV Europe GmbH, SMR Automotive Next, we categorized the articles according to topics. ...
... Stuttgart GmbH, Toyota, ZF AG, and Hyundai Motor were also involved with articles(KRIEBEL, 2018;KLENDAUER et al., 2012;YAMAGUCHI et al., 2016;NEUROHR, 2021;SEO, 2012;CHOI, 2012;YANG, 2012). These results indicate that a small circle of companies is involved in researching the topics covered by this SLR, and most of them are in Europe. ...
In automotive context, the embedded software Verification and Validation (V&V) is always a critical step for each project that involves testing solutions for new function, system optimization and compliance with legal requirements. However, automotive software V&V is laborious and time-consuming. Activities such as planning workshops at test tracks and public roads and functionality and durability tests require significant effort and robust coordination. The rigorous management and storage of test results are also a challenge. This review consolidates the state of the art on automotive software V&V to realize the most common standards in the industry and understand current testing concepts. The consolidated knowledge will help in the future development of a flexible V&V framework for embedded automotive software. A Systematic Literature Review (SLR) was performed by searching four digital libraries from 2011 to March 2022. Sixty-two papers were selected, which indicated that the automotive software V&V process is usually based on the ISO 26262 standard and that the software development life cycle V-model is the most common test platform in the automotive domain. Automotive software for a specific domain has been developed to cover a wide variety of vehicles. Variables from specific regions or countries can influence the entire V&V process for automotive software, such as differences in homologation requirements, infrastructure, driver behavior, customer desires, and electromagnetic force interference. The SLR identified specific characteristics of automotive software and regional factors that can affect the V&V process, as well as significant considerations to ensure correct decision-making, resource allocation, and support of team members.
... From the backward references, 17 studies were selected, whereas 16 studies were selected from forward citations. From the 33 selected studies, P0064 [28] was subsumed 9 by P0253 [36]; furthermore, P0498 [48] and P0499 [49] were subsumed by P00487 [47]. Therefore, we ended up with a set of 30 studies that we analyse in the next sections of this article. ...
Model‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT: (i) some studies did not fully describe how tools transform abstract tests into concrete tests, (ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments.
... The use of model-based testing for testing a single software product is widely observed in the literature and in practice [1,23,29,43]. However, companies rarely develop one single product. ...
... Number, importance, and complexity of software is continuously increasing [FR07,KMS+18]. The increasing complexity of the software of such systems demands better concepts, methods, and tools that enable overcompensating this growth in complexity and harnessing their potential. ...
Digital twins becoming more prevalent: They are being used to support the design, operations, and analysis of complex systems in many domains, such as automotive, agriculture, avionics, construction, or medicine, and comprise much information about the systems and processes of the twinned system. Currently, digital twins are designed and engineered ad-hoc, in a piecemeal fashion. This hampers the research and application of digital twins. Based on our interdisciplinary research regarding the “Internet of Production”, we combine model-driven methods for the sustainable engineering of information systems, software architectures, and software language engineering to systematically engineer digital twins. Within this chapter, we discuss challenges on the road to a systematic engineering of digital twins, present our model-driven approach for the engineering of them as well as possible implementations. Our insights may guide researchers and practitioners to sustainable, planned, and efficient engineering and operations.
... Software engineering has successfully overcome the conceptual gap using model-based software engineering approaches [5,8]. Several studies have investigated the benefits of model-based software development [9,10] and show that considering systems as networks of interacting encapsulated subsystems leads to enhanced system quality especially when utilized at the early stages of development. ...
... The SMArDT method [24] finds its applications in automotive software engineering, and in particular testing [9,25]. The method relies on formalized models in the systems modeling language (SysML) that represent the system under development on four levels of abstraction. ...
... The former express expectations about the desired behavior of the system. Suitable modeling languages for behavioral are, e.g., SysML state machines or activity diagrams [9,39]. ...
Engineering Cyber-Physical Systems (CPS) is complex and time-consuming due to the heterogeneity of the involved engineering domains and the high number of physical and logical interactions of its subsystems. Model-based Systems Engineering (MSBE) approaches tackle the complexity of developing CPS by formally and explicitly modeling subsystems and their interactions. Newer approaches also integrate domain-specific models and modeling languages to cover different aspects of CPS. However, MBSE approaches are currently not fully applicable for CPS development since they do not integrate formal models for physical and mechanical behavior to an extent that allows to seamlessly link mechanical models to the digital models and reuse them. In this paper, we discuss the challenges arising from the missing integration of physics into MBSE and introduce a model-based methodology capable of integrating physical functions and effects into an MBSE approach on a level where detailed physical effects are considered. Our approach offers a fully virtual, model-based development methodology covering the whole development process for the development of CPS. Evaluating this methodology on a real automotive use case demonstrates benefits regarding virtual development and functional testing of CPS. It shows potentials regarding automated development and continuous integration of the whole CPS including all domains. As an outlook of this paper, we discuss potential further research topics extending our development workflow.
... Formal methods have been thoroughly employed in Model-Based Testing (MBT) to automate test activities. Several surveys (e.g., [15]- [17]) show that model-based testing is broadly used in the automotive domain and Khan et al. [16] highlight its ability to handle the complexity of testing modern vehicles. This raises the question whether transferring model-based methods to security testing can help to address current challenges, such as late testing and late vulnerability identification. ...
... Depending on the employed MBT approach and the availability of suitable tools, executable test scripts can be generated from the derived test cases to test the target system (5). VOLUME 11, 2023 is the Systems Modeling Language (SysML) [26], which is applied in systems engineering (e.g., in the automotive sector [17]). SysML adopts a subset of UML diagram types directly or in an adapted form and introduces new diagram types, such as requirement diagrams. ...
... Existing surveys [15], [17] suggest that MBT is commonly applied in the automotive domain. However, the available surveys treat MBT and security testing separately. ...
Modern connected or autonomous vehicles (AVs) are highly complex cyber-physical systems. As a result of the high number of different technologies and connectivity features involved, testing these systems to identify security vulnerabilities is a big challenge. Security testing techniques, such as penetration testing, are often manual methods that are applied comparatively late in the vehicle development process. Thus, vulnerabilities are only detected late or after development, leading to higher costs and more patching effort. To reduce the amount of testing resources in general and enable early and automated testing, model-based testing methods have been established in several domains, such as information technology and the automotive domain. The transfer of model-based testing approaches to automotive security testing could help to detect vulnerabilities earlier than other, manual methods by automatically generating, executing, or simulating security tests. In this study, we review the literature on model-based test approaches in the automotive domain. First, we consider security-independent approaches to obtain an overview of applied models, formalisms, test selection criteria, and test generation techniques. In addition, we investigate, whether and how model-based approaches are applied for automotive security testing. Overall, we identified 63 publications related to model-based testing and 29 publications with regard to model-based security testing. The aim of this study is to provide an overview and direct comparison between these approaches. In this manner, the state of model-based security testing in the automotive domain, current challenges, and potential research areas are determined.
... Test Case Generation for Behavioral Models. Some researchers use MDE or other automated means to generate test cases from modeling artifacts, most notably from requirement models, use cases, or activity diagrams [11,90,122,143]. Most of these efforts follow one of two main approaches for test case generation: path/coverage analysis [90,122], or category partition [122,143]. ...
... Some researchers use MDE or other automated means to generate test cases from modeling artifacts, most notably from requirement models, use cases, or activity diagrams [11,90,122,143]. Most of these efforts follow one of two main approaches for test case generation: path/coverage analysis [90,122], or category partition [122,143]. The former approach is based on analyzing all possible paths of behavior in the source model, and the latter partitions the requirements under test and generates test cases for combinations of such partitions. ...
The continuous growth of software complexity raises the need for effective complexity management. Model-Driven Engineering (MDE) is a development paradigm that meets this requirement by separating concerns through models. A model is a specific abstraction of a system that can be defined by a Domain-Specific Language (DSL). A DSL with execution facilities, referred to as Executable DSL (xDSL), enriches the modeling quality by enabling the employment of dynamic Verification & Validation (V&V) techniques. Testing is the most prevalent dynamic V&V technique in the field of software engineering. While many testing frameworks exist for general-purpose programming languages, providing testing facilities for any given xDSL remains a costly and challenging task. In this thesis, we propose a generic testing framework for executable DSLs. Given an xDSL, the framework provides a testing language that supports the use of xDSL-specific concepts in the definition of test cases. This enables the xDSL’s users, namely the domain experts, to write test cases for their models. The written test cases can be executed on the models and the test results will be produced. To further support the domain expert in efficiently testing models, the framework offers three supplementary services: (i) test quality measurement to ensure that the written test cases are good enough; (ii) test debugging to localize the fault of the model under test in case of test failure; and (iii) automatic test improvement to strengthen the ability of written test cases in detecting regression faults.
... With the increasing complexity of hardware and software, the topic of automotive software engineering has attracted much attention from scientists and engineers [1], [2]. Those developments must consider the incorporation of other technologies, such as artificial intelligence, computer vision, multi-domain sensors, and multi-agents [3], [4]. ...
... Testing could represent as much as half of the overall costs of software development, and with the increasing complexity of software, the proportion of testing costs could rise unless more efficient testing methodologies are developed [12], [13]. The main focus of industrial research, to reduce cycle time and development costs, and simultaneously increasing software quality, is improving the software testing process [2], [14]. The whole electronic vehicle test cycle has several levels, and the main challenge is to identify a test strategy in which most of the failures could be detected in the beginning of the project. ...
Road vehicles have incorporated several functionalities over the last decade, with an increasing incorporation of electronic embedded systems. Most of those functionalities are controlled, managed, and supervised by distributed software, within many interconnected Electronic Control Units (ECUs). Within such context, new methodologies for tests of the distributed software functions must be developed to ensure proper performance at the integration level, complying with the requirement specifications. Many strategies have emerged to organize the multiple levels of software testing in automotive embedded systems, to reduce costs and improve its effectiveness and robustness, but there is a need for a methodology to structure, optimize, and plan the tests with real automotive criteria. This work aims to extend the multiple levels of testing concept and propose a method to analyze, design, and evaluate the application system upfront to derive a software testing plan. This method has been developed incorporating the automotive application characteristics, including functional safety requirements specified in the ISO 26262 standard to structure and plan the embedded software test in vehicle development. The proposed method combines an enhancement on plan testing strategy, matching requirement specifications, with adherence to the current methods used in practice by the automotive industry. Such a process will help the automotive industry to follow some concrete steps during validation, to optimize the test volume and facilitate documentation of developed activities, improving the safety and security of automotive systems in the early stages of automotive embedded software, when the details of each function are not yet implemented at the component level.
... This is hampered by the fact that till date majority of test case creation process are still manual. The authors had carried out a comprehensive survey in order to investigate the possible benefits of model-based testing in the automotive sector [18]. The test cases can target different levels, ranging from units (individual methods) to the complete system. ...
The process of testing conventional programs is quite easy as compared to the programs using Deep Learning approach. The term Deep learning (DL) is used for a novel programming approach that is highly data centric and where the governing rules and logic are primarily dependent on the data used for training. Conventionally, Deep Learning models are evaluated by using a test dataset to evaluate their performance against set parameters. The difference in data and logic handling between programs using conventional methods and programs using the DL approach makes it difficult to apply the traditional approaches of testing directly to DL based programs. The accuracy of test data is currently the best measure of the adequacy of testing in the DL based systems. This poses a problem because of the difficulty in availability of test data that is of sufficient quality. This in turn restricts the level of confidence that can be established on the adequacy of testing of DL based systems. Unlike conventional applications, using the conventional programming approaches the lack of quality test data and the lack of interpretability makes the system analysis and detection of defects a difficult task in DL based systems. So testing of DL based models can be done automatically with a different approach compared to normal software.
... Model-based test case generation method is a prominent technique in software testing. Commonly used models include FSM (Finite State Machine) model [1], UML (Unified Modeling Language) model [2,3,4] and Markov model [5,6]. These models used to describe the function and operation flow of the system under test (SUT). ...
Web applications often face continuous updating due to functional change or UI renew, while it remains a challenge to guarantee their correctness. The goal of software testing is to find defects in a limited time range whereas exhaustive testing is an ideal yet time-consuming process. In this research, we propose an approach to generating test cases automatically based on the Markov reward process which innovatively contains a reward function for test results to guide the generation of test cases. By using the N-step algorithm, this approach can generate the test flow with the highest risk priority which can capture software defects as quickly as possible. The experiment on an e-commerce system shows that there is significant improvement on the defect detection capability of test cases generated through Markov reward process.
