Figure 1 - uploaded by Dietmar Winkler
Content may be subject to copyright.
Source publication
The share of software in industrial automation systems is steadily increasing. Thus, software quality issues become a critical concern for many automation projects, which require effective software quality assurance measures. In this paper we describe an architecture for automated testing of software applications part of industrial automation syste...
Contexts in source publication
Context 1
... this architecture has been defined for an automation system controlling manufacturing machines. Figure 1 provides a high-level overview of the architecture. On the left-hand side it shows the automation system in the production context, and on the right-hand side it illustrates how testing can be integrated in the system's architecture. ...
Context 2
... Figure 1, the left-hand side provides an overview of the auto- mation system configured for production. It consists of three lay- ers linked via industrial network technologies: ...
Context 3
... right-hand side of Figure 1 shows the setup of the automation system in context of testing. Testing focuses on the PLC software located at the middle layer of the system's architecture. ...
Context 4
... order to achieve representative testing results, the tested object has to be the same as in the production context. Thus, as indicated in Figure 1, the PLC as well as the middleware and (optionally) the machine are present in both contexts. These components and their software should not be modified when executed in the test- ing context, i.e., no test specific settings or compiler switches are required. ...
Context 5
... build system keeps track of the results and generates reports and statistics. Figure 10 shows the test results trend from the build system of the investigated industrial project. The regression test suite contains more than 200 automated test cases. ...
Similar publications
![Figure 1 -Example of step stress test [3]](profile/Lukas-Spendla/publication/269835659/figure/fig1/AS:392179383062553@1470514259701/Example-of-step-stress-test-3_Q320.jpg)
The main goal of article is the automation possibilities of accelerated stress testing. The proposal is focused into control systems as a part of Information systems. The contribution demonstrates the usage of Step Stress Test model in the fully automated test system, using minimum of human intervention. Our proposal captures the particular states...
Background:
Oppositional defiant disorder (ODD) that is characterized by markedly defiant, disobedient, and disruptive behavior in younger children has been regarded as disruptive behavior disorder (DBD), together with conduct disorder (CD). However, in contrast to CD, ODD does not include severe aggressive or antisocial behavior.
Aim:
This stud...
Test automation allows performing difficult and time consuming manual software testing tasks efficiently, quickly and repeatedly. However, development and maintenance of automated tests is expensive, so it needs a proper prioritization what to automate first. This paper describes a simple yet efficient approach for such prioritization of test cases...
Background. Testing is an essential activity in the software development life cycle. Nowadays, testing activities are widely spread along the software development process, since software products are continuously tested to meet the user's expectations and to compete in global markets. In this context, internationalization testing is defined as the...
Synopsis Despite advances in building automation technology, many building control systems do not work as intended. Thorough commissioning of control systems can significantly improve operation but is manually intensive and thus expensive and often neglected. Research in automated commissioning tools and processes has the potential to significantly...
Citations
... Studies from Lin et al. (2020), Hilton et al. (2017), Lee and Hwang (2012), Williams et al. (2009), andKasurinen et al. (2010), which hold this view, have observed the increased effort to develop and execute more rigorous automated tests in attaining higher product quality. An experience study (Ramler et al., 2014) reported that the direct effect of test automation maturity on test automation effort is negative. However, a different school of viewpoint reported by studies from Kumar andMishra (2016), Puri-Jobi (2015), and Tosun et al. (2009) views that, test automation effort will be reduced after improving the level of test automation maturity and product quality after a certain period of time -even though the considerable effort is required to develop and execute rigorous automated tests in attaining high product quality, the later effort savings may offset the invested effort. ...
The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software repositories of the same projects. The main results of regression analysis reveal that, higher levels of test automation maturity are positively associated with higher product quality (p-value=0.000624) and shorter release cycle (p-value=0.01891); There is no statistically significant evidence of increased test automation effort due to higher levels of test automation maturity and product quality. Thus, we conclude that, a potential benefit of improving test automation maturity (using standard best practices) is product quality improvement and release cycle acceleration in the CI context of open source projects. We encourage future research to extend our findings by adding more datasets with different programming languages and CI tools, closed source projects, and large-scale industrial projects. Our recommendation to practitioners (in the similar CI context) is to utilize standard best practices to improve test automation maturity.
... There exist few works, which cover the testing of industrial automation software. Ramler et al. present a test automation approach in the field of PLC software, share learned lessons and provide some recommendations [10]. Symbolic execution is applied by Suresh et al. [11] to create test cases for small PLC programs. ...
Product line management activities have to ensure that offered product options are valid and compatible. With the arise of the Internet of Things (IoT) movement not only the own product compatibility has to be managed by the vendors anymore, but also the compliance and openness to standardized interfaces has to be supported as well. The Machine to Machine (M2M) communication protocol standard Open Platform Communications Unified Architecture (OPC UA) has received great attention in the field of mechanical engineering recently. In this industrial experience report we describe our approach how to support the testing of automatically generated models for OPC UA, by applying test case generation at the integration level. We show the feasibility of our approach and report about found issues, discuss some general findings and provide an outlook for future work.
... Thus, Gherkin test scenarios can easily be used for Test-Driven Development [7], where developers implement tests prior to the implementation of the software functionality [4], [23]. As such tests are based on identified requirements and agreed interfaces, they are more robust and better to maintain [24]. Tools like Jenkins 1 are then used to automate the execution and reporting of test cases [23]. ...
... Among others, there is a growing importance of testing in ASE and PSE projects [18]. Ramler et al. [15] report on practical receipts and lessons learned in context of automated testing of industrial automation systems with focus on test driven and data driven testing. Furthermore, concepts of Capture & Replay can help to better create and execute test cases in the industrial automation domain [14] in a test automation context. ...
... report on an orchestrated survey of methodologies for automated software test generation [1]. However, the resulting test cases need to be embedded within a testing framework, such as [15]. Therefore, a forth challenge is to embed software test code and/or test code generation approaches in the test automation tool chain (C4: Test Code Management). ...
... The main goal of test automation in the automation systems domain is making automated tests more robust and maintainable [15]. Data-Driven Tests focus on the separation of test data and parametrized test scripts. ...
... The application of CV techniques in testing supports the first aspect but, naturally, it cannot solve the challenge of providing "physical input" to the system. In such cases the proposed solution can be used as an extension to a test setup based on hardware-in-the-loop testing with emulated sensors [7]. It depends on the specific system under test to what extent real-world testing can benefit from CV and where further extensions are necessary. ...
To support the vision of Industry 4.0 and smart factories, manufacturing companies need to evolve their machines towards intelligent, connected cyber-physical systems (CPS). Software is at the heart of these digital transformation processes. OPC UA (Open Platform Communications Unified Architecture) is an industrial standard for horizontal and vertical communication of CPS. The adoption of OPC UA has a significant impact on the design of industrial production systems and on the development process. In this paper, we report on our experiences of migrating a heterogeneous software stack of a series machine production company to OPC UA. We show that automated approaches (i.e., static code analysis, code generation, and documentation generation) are the foundation for a holistic approach for working with OPC UA across different system layers and development teams. Further, we outline challenges we have identified during the migration process.
Context
Continuous integration (CI) is a practice that aims to continuously verify quality aspects of a software intensive system both for functional and non-functional requirements (NFRs). Functional requirements are the inputs of development and can be tested in isolation, utilising either manual or automated tests. In contrast, some NFRs are difficult to test without functionality, for NFRs are often aspects of functionality and express quality aspects. Lacking this testability attribute makes NFR testing complicated and, therefore, underrepresented in industrial practice. However, the emergence of CI has radically affected software development and created new avenues for software quality evaluation and quality information acquisition. Research has, consequently, been devoted to the utilisation of this additional information for more efficient and effective NFR verification.
Objective
We aim to identify the state-of-the-art of utilising the CI environment for NFR testing, hereinafter referred to as CI-NFR testing.
Method
Through rigorous selection, from an initial set of 747 papers, we identified 47 papers that describe how NFRs are tested in a CI environment. Evidence-based analysis, through coding, is performed on the identified papers in this SLR.
Results
Firstly, ten CI approaches are described by the papers selected, each describing different tools and nine different NFRs where reported to be tested. Secondly, although possible, CI-NFR testing is associated with eight challenges that adversely affect its adoption. Thirdly, the identified CI-NFR testing processes are tool-driven, but there is a lack of NFR testing tools that can be used in the CI environment. Finally, we proposed a CI framework for NFRs testing.
Conclusion
A synthesised CI framework is proposed for testing various NFRs, and associated CI tools are also mapped. This contribution is valuable as results of the study also show that CI-NFR testing can help improve the quality of NFR testing in practices.
Control software for automated Production Systems (aPSs) becomes increasingly complex. Respective systems undergo constant evolution. Yet, proper documentation may not always be present, entailing maintenance issues in the long run. While manual examination of software for aPSs is an error-prone task, static analysis can improve system quality. However, it has not been applied to describe software evolution by means of changed systems artifacts. The authors address this issue and perform change analyses on IEC61131-3 projects, identifying introduced and removed systems artifacts as well as existing ones affected. By that, the authors aim to support sustainable evolution. Two feasibility studies, implemented independently, but for the same evolution scenarios for an automation plant, are used for evaluation. The technique is shown to be efficient and highly precise.
