Content uploaded by Nauman Bin Ali
Author content
All content in this area was uploaded by Nauman Bin Ali on Jan 06, 2021
Content may be subject to copyright.
JOURNAL OF SOFTWARE: EVOLUTION AND PROCESS
J. Softw. Evol. and Proc. 2018; 00:1–29
Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/smr
Towards a Benefits Dependency Network for DevOps based on a
Systematic Literature Review
Ramtin Jabbari∗1, Nauman bin Ali1, Kai Petersen1,2, Binish Tanveer3
Blekinge Institute of Technology, Sweden1, Flensburg University of Applied Sciences, Germany2, Fraunhofer Institute
for Experimental Software Engineering, Germany3
SUMMARY
Context: DevOps as a new way of thinking for software development and operations has received much
attention in the industry, while it has not been thoroughly investigated in academia yet.
Objective: Characterize DevOps by exploring its central components in terms of principles, practices and
their relations to the principles, challenges of DevOps adoption and benefits reported in the peer-reviewed
literature. As a key objective, we also aim to realize the relations between DevOps practices and benefits in
a systematic manner.
Method: A Systematic Literature Review (SLR) was conducted. Also, we used the concept of benefits
dependency network to synthesize the findings, in particular to specify dependencies between DevOps
practices and link the practices to benefits.
Results: We found that in many cases DevOps characteristics i.e., principles, practices, benefits, and
challenges were not sufficiently defined in detail in the peer-reviewed literature. In addition, only a few
empirical studies are available which can be attributed to the nascency of DevOps research. Also, an initial
version of the DevOps benefits dependency network has been derived.
Conclusion: The definition of DevOps principles and practices should be emphasized given the novelty of
the concept. Further empirical studies are needed to improve the benefits dependency network presented in
this study.
Copyright c
2018 John Wiley & Sons, Ltd.
Received . . .
KEY WORDS: DevOps, Development and Operations, Principles and Practices, Challenges, Benefits
and Values, Systematic literature review.
1. INTRODUCTION
Software has become more complex to be developed and also it has been a vital competitive
advantage in the market. Therefore any decision making to balance out innovation, feature
differentiation, high quality, low development investment and fast delivery to market is further
critical and directly impacts the whole software development process [6,34]. In fact, this situation
is a matter of trading-off between benefits and limitations mostly addressed by e.g., value-based
software engineering [11,12].
DevOps (a clipped compound of development and operations) as a new concept for developing
software has received considerable attention from the industry. According to ProfitBricks†there have
been 49 practitioner-oriented conferences and events for DevOps in 2016 [54]. However, it has not
been as widely investigated in the scientific community yet [19].
∗Correspondence to: Ramtin Jabbari, Blekinge Institute of Technology, Karlskrona, Sweden (Contact him at:
ramtin.jabbari@bth.se).
Copyright c
2018 John Wiley & Sons, Ltd.
Prepared using smrauth.cls [Version: 2012/07/12 v2.10]
2JABBARI ET AL.
Given that DevOps is a relatively new term, a common understanding of what it entails has not yet
been achieved. Consequently, the understanding of DevOps is often inconsistent and only represents
a specific perspective which constitutes only a part of the concept. In fact, a lack of systematic
investigation makes DevOps an unknown phenomenon in particular with respect to its principles,
practices, benefits, and challenges [49].
To this end, software engineering researchers have a strong interest in the topic to identify and
specify DevOps characteristics to bridge the gap between academia and industry e.g. to derive what
DevOps means. In this study, we conducted a systematic literature review to explore and investigate
how DevOps has been evaluated in available peer-reviewed literature. Broadly, the study had the
following objectives:
•Identify and specify DevOps principles.
•Identify and specify practices and activities associated with DevOps.
•Identify and specify challenges of DevOps adoption and relate them to specific practices.
•Identify and specify claimed and demonstrated benefits of DevOps.
•Synthesize the findings by (a) determining the dependencies between practices and (b)
determining the links between practices and corresponding benefits.
We conducted a systematic literature review [33] to achieve the above-stated objectives. The focus
of our study was on peer-reviewed literature. The findings have been synthesized with the use of
benefits dependency network [9,46] to illustrate the relations between practices and benefits. This
may enable practitioners to select the benefits they intend to achieve by adopting DevOps. Then,
using the links presented in this study between benefits and practices, they can determine which
practices are needed to achieve the desired benefits. By considering the benefits and the practices
together practitioners can make an informed decision reflecting on the cost-effectiveness of the
course of action [4]. For researchers, this work can serve as a framework for future research e.g. to
investigate in empirical studies whether certain practices lead to claimed benefits.
The remainder of the paper is structured as follows: Section 2describes the related work. Section 3
presents the research questions and research method. Thereafter, in Section 4, we present the results
including the DevOps principles, practices, challenges, and benefits. The findings are used as inputs
to construct the benefits dependency network for DevOps and presented in Section 5. Section 6
discusses the findings. The threats to validity have been discussed in Section 7and Section 8
concludes the paper.
2. RELATED WORK
This section presents secondary studies on DevOps and also elaborates on how these studies are
related to our study. Moreover, the software value map is described which has been used to illustrate
the DevOps benefits.
2.1. Existing SLRs on DevOps
Few systematic studies have been conducted on the topic of DevOps. Erich et al. [21] studied
the cooperation between information system (IS) development and operations. They identified 139
papers and summarized results from 26 of them in the form of a mapping study. They considered
DevOps as a conceptual framework that benefits IS development, therefore they attempted to find
the relevant empirical evidence in the literature. As the main focus of their work is particularly
on IS, thus many potential papers related to DevOps have been overlooked. The main conclusion
was that DevOps has not been studied in the literature adequately and few existing ones often
have low quality. Consequently, the potential for further research on DevOps and its relation to
IS development has been highlighted. Also, Erich et al. did not consider DevOps benefits and
challenges which are targeted in our study.
†A cloud computing IaaS company; profitbricks.com
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 3
Smeds et al. [52] aimed to specify the concept of DevOps based on the literature and identify
impediments perceived in practice to adopt DevOps through conducting interviews. For the first
purpose, they selected 27 papers out of 126 results of the search. They finally defined DevOps as
a set of engineering process capabilities interrelated to cultural and technological aspects. Also,
they emphasized that the existing definitions of DevOps are usually unclear or only concentrated on
a specific context. For the second purpose, due to the lack of relevant literature, semi-structured
interviews have been conducted in an international IT company that its DevOps adoption was
at a preliminary stage. In the end, they concluded that DevOps definitions, adoptions, and its
impediments are still vague which shows the need for further research on the topic.
Nicolau de Franc¸a et al. [19] have also conducted a literature review on DevOps to characterize
the industrial perspectives. They used various types of data source including websites, technical
reports, and few peer-reviewed studies. Hence, they did not perform a traditional systematic
literature review as they believe that DevOps as a recent idea has been more described in the
industry and considering only the peer-reviewed studies is not adequate enough. However, the focus
of Nicolau de Franc¸a’s study is similar to ours, in particular there is a shared focus on identifying
DevOps principles, practices and benefits. In Section 6.3, we have compared the two studies further
w.r.t. the type of studies included and the results.
2.2. Characterizing the Benefits and Limitations of Software Development
As mentioned in the introduction, software has become a vital competitive factor in the market and
also more complex due to rapid changes. Therefore any decision making with respect to software
development processes is critical and has direct effects on the responsiveness of the process to the
ongoing changes [6,34].
In fact, this situation is a matter of trading-off between benefits and limitations to prioritize
features, quality attributes, business goals, potential customers, etc. which has been explicitly
specified by Value-based Software Engineering (VBSE). In VBSE, each feature, decision, and
attribute of a software product setting has its own value which might not be equal to another [11,12].
Therefore, there is an essential need for considering different perspectives and corresponding
values as a whole in order to not unintentionally miss any software value component. To this end,
Khurum et al. [34] present an exhaustive view over values for the software development namely
Software Value Map (SVM) which has been used by us as a means to categorize the claimed
and demonstrated benefits of DevOps identified from the literature in this study. One of the main
motivations to use SVM is that the value dimensions directly represent benefits that companies want
to achieve such as functionality, faster time to market, lower cost, etc. Table Ipresents the customer
value perspective and corresponding value aspects, sub-aspects and components as an example from
SVM.
The SVM proposes a unified collection of software values by a detailed categorization of four
main value perspectives. This collection provides a common understanding of values that can
be used by practitioners as decision support to assure no value perspective is unintentionally
overlooked. The four value perspectives of SVM are as follows: [34]
•Financial: The financial perspective contains aspects that address the company’s
implementation and execution of its strategy, which is contributing to the bottom-line
improvement of the company. It represents the long-term strategic objectives of the
organization and thus it incorporates the tangible outcomes of the strategy in traditional
financial terms. Some of the most common financial measures that are incorporated in the
financial perspective are: earned value analysis, revenue growth, costs, profit margins, cash
flow, net operating income, and customer value analysis.
•Customer: The customer perspective defines the value proposition that a company applies to
satisfy customers and thus generate more sales to the most desired customer groups (i.e., the
most profitable). Measures selected for the customer perspective should consider the value
that is delivered to the customer which may involve time, quality, performance and cost as a
result of this value proposition (see Table I).
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
4JABBARI ET AL.
Table I. An example from SVM presenting value from customer perspective [34]
Value Perspective Value Aspect Sub-value Aspect Value Component
Customer
Perceived value
Intrinsic value
Functionality
Reliability
Usability
Maintainability
Portability
Delivery process value
Process w.r.t. time
Process w.r.t. quality
Process w.r.t. cost
Network externalities
Complementary value
User experience value
Functionality
Reliability
Usability
Maintainability
Portability
Hedonic value
Lifetime value
Retention rate
Revenue
Autonomous revenue
Upselling revenue
Cross-selling revenue
Reference value
Cost
Acquisition cost
Marketing cost
Sales cost
Termination cost
•Internal business process: The internal process perspective is concerned with the processes
that create and deliver the customer value proposition. It focuses on all the activities and key
processes required for the company to excel at providing the value expected by the customers
both productively and efficiently. These can include short-term and long-term objectives and
incorporating innovative process development to stimulate improvement. Quality, cycle time,
productivity, and cost are some aspects where performance value can be measured.
•Innovation and learning: The innovation and learning perspective is the basis of any strategy
and focuses on the intangible assets of an organization, mainly on the internal skills and
capabilities that are required to support the value-creating internal processes. This perspective
is concerned with the intellectual capital categorized as human capital, structural capital, and
the organization capital of a company.
To use SVM in this study, we (1) consider the descriptions provided by SVM for each value
perspective and value aspect and (2) map the identified DevOps benefits by our judgement based on
the descriptions. SVM and its details are available and explained by Khurum et al. [34].
3. RESEARCH METHOD
In this section, we present the research questions (RQs) and our approach to address them. We
performed a systematic literature review using the guidelines proposed by Kitchenham et al. [33].
3.1. Research questions
To get an understanding of DevOps characteristics, an overall question motivates this study: ‘Which
DevOps practice and ultimately a sequence of DevOps practices gives which benefit(s)?’. To this
end, three research questions have been derived and formulated as follows:
RQ1. What are the principles and practices associated with DevOps in the peer-reviewed
literature?
RQ1 aims to identify and extract the DevOps principles and practices which have been presented
by the peer-reviewed studies. To address the RQ1, we just extracted those principles (e.g., general
rules) and practices (e.g., activities) which were explicitly linked to DevOps.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 5
RQ2. What are the challenges explicitly associated with DevOps in the peer-reviewed
literature?
To complete the study on DevOps, the purpose of RQ2 is to extract the DevOps challenges from
the peer-reviewed literature. To address the RQ2, we only identified those challenges which were
explicitly presented as a challenge for DevOps (e.g., DevOps adoption).
RQ3. What are the benefits (claimed and demonstrated) explicitly associated with DevOps
in the peer-reviewed literature?
To follow up on characterizing DevOps, RQ3 aims to identify and specify the proposed benefits of
DevOps which exist in the available peer-reviewed studies. To address the RQ3, we only considered
those benefits which were explicitly presented as a DevOps benefit. We have also separated the
benefits (i.e., claimed and demonstrated) based on the evidence presented by the selected studies.
3.2. Database search
The search strings, databases, and number of papers found per database from the search conducted
on September 4, 2015 are shown in Table II.
The main aim of the search was to find all papers, in which DevOps is explicitly mentioned. The
reason why the search string (i.e., keyword) has been only limited to “DevOps” is to keep the search
results more general and avoid missing any potential paper that might have been related to DevOps.
In fact, we aimed at identifying the term “DevOps” itself which has been reported in the literature.
Moreover, we attempted to not exclude any relevant studies during the search as any characteristic
of DevOps (e.g., benefits) may be reported, but not called as the same naming in the paper. Hence,
all papers returned for “DevOps” and fulfilling our criteria are reviewed. Also, we only selected the
full-text literature to avoid missing any potential paper which has not mentioned DevOps in e.g.,
title, abstract, keywords, etc.
To increase the confidence in the search process in terms of the number of papers found per
database, the search has been conducted twice by the first two authors which is referred to the test-
retest procedure [38]. Moreover, to ensure not to overlook any paper related to DevOps we expanded
the database search to 6electronic databases and indexes in the area.
3.3. Selection criteria and process
A few simple criteria were used to assess the relevance of the articles in order to improve the
reliability of the study [5], as follows:
•Include an article published in English.
•Include only a peer-reviewed article.
•Exclude an article not available in full-text.
•Exclude proceedings of conferences. (e.g., messages from chair of editorial boards, etc.)
Table II. Search strings, databases, and results
Database Search string (query syntax generated by database) # papers
IEEE “DevOps” 46
ACM (“DevOps”) and (PublishedAs:journal OR PublishedAs:proceeding) 85
Inspec 1969-2016: ((“DevOps”) WN All fields) 41
Scopus TITLE-ABS-KEY ( “DevOps” ) AND (LIMIT-TO ( SRCTYPE, “p” ) OR LIMIT-TO (
SRCTYPE, “j” ))
51
Wiley Online Library “DevOps” in All Fields 7
Web of Science TOPIC:(“DevOps”) Timespan: All years. 8
Total 238
Duplicates (Auto detection) −79
Duplicates (Manual detection) −1
Remove proceedings −10
Non peer-reviewed −23
Irrelevant −2
Not available in full-text −5
Remaining 118
Rejected after full-text reading −78
Total number of studies 40
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
6JABBARI ET AL.
For those articles not available in full-text (7 papers), we contacted the university librarian and
the author(s) and at last two studies were obtained. To mitigate the risk of misinterpretation of the
selection criteria and avoid individual biases, the selection process has been jointly performed by
the first two authors in a pair selection session for allowing an immediate discussion about the
corresponding uncertainties for all papers.
After applying the selection criteria, we further excluded the irrelevant studies during full-text
reading and data extraction conducted by the four authors. In this step, we included those articles
which have at least a definition or principle or practice or challenge or benefit associated with
DevOps. We also excluded those which have no explicit discussion about DevOps that finally
resulted in 40 studies.
3.4. Data extraction
This section presents the data extraction form used to extract data based upon the objectives of this
study. We performed pilot extraction using the form on five articles that were selected randomly.
Then, data extraction was performed on the five studies by the first two authors separately to
evaluate the form. Based on this evaluation the form was refined. Subsequently, all four authors
extracted data individually from two new randomly selected articles. We developed consensus
regarding the interpretation of the data extraction form through a discussion about differences in
the extracted information. Afterwards, the remaining papers were distributed among the authors for
data extraction. Table III shows the data extraction form.
Table III. Data extraction form
Data extraction item Details
Study Eligibility Contributes to answer RQ 1
Definitions Author states; e.g., DevOps is OR defined as OR ...
Principles General rules to be observed or guiding the use of DevOps, e.g. value people over..., etc.
Practices Activities that are proposed to be executed in the context of DevOps.
Relevant measure(s) Evaluation criteria and measures, including indicators, attributes, etc. For instance, lead time
improvements, quality improvements, learning, etc.
Goal or Intend of the research Research contribution or gap to be filled by the paper, an example to evaluate DevOps with
respect to X, to provide an understanding or definition of ...
Limitations/
Challenges of DevOps
Contributes to answer RQ 2
Limitations/
Challenges of DevOps?
List of identified challenges
Benefits of using DevOps Contributes to answer RQ 3
Infrastructure/ Requirements To characterize where and how the benefits materialize
Claimed benefits of using
DevOps?
List of identified benefits
Demonstrated benefit or purpose
of using DevOps?
List of identified benefits
Type of study? (based on a
number of conditions presented
by Petersen et al. [48].)
-Used in practice (report makes explicit that industry has been the user of DevOps, e.g. it
should not be a student project)
-Novel solution (a solution or extension has been proposed)
-Empirical evaluation (a research method has been used)
-Conceptual framework
-Authors’ experience
-Opinion about something
Research method? (used in Eval-
uation/Validation Study [48]) i.e.,
this is the method authors state
they have used.
-Controlled experiment with practitioners [Evaluation method]
-Practitioner targeted survey [Evaluation method]
-Action research [Evaluation method]
-Ethnography [Evaluation method]
-Simulation as an empirical method [Validation method]
-Laboratory experiments (machine or human) [Validation method]
-Prototyping [Validation method]
-Mathematical analysis and proof of properties [Validation method]
-Academic case study (e.g., with students) [Validation method]
-Not specified
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 7
As mentioned in Section 3.3, a number of studies were rejected during the data extraction due to
the irrelevancy to DevOps and our study objectives. This elimination was impossible before full-text
reading, e.g., from the title and abstract. Therefore, all 118 studies were involved in data extraction,
of which 40 studies were finally selected. The majority of the selected papers are about DevOps
benefits. 55% of them (22 papers) have discussed DevOps benefits realized as the claimed ones and
5 studies provided relevant information in terms of e.g., type of study and research methods (see the
data extraction form in Table III) that we could specify them as demonstrated benefits.
It should be mentioned that only 3 papers [22,40,56] have had an overlap of DevOps principles,
practices, challenges and benefits that shows a lack of comprehension of DevOps in terms of
different characteristics in the selected studies. Table IV shows the frequency of the 40 studies
used in our research in accordance with each objective (DevOps principles, practices, challenges
and benefits).
3.5. Data analysis
We utilized content analysis [17] to structure the DevOps principles, practices, challenges and
benefits as follows:
Principles: For DevOps principles, we investigated the studies which have explicitly proposed
or presented e.g., general rules or guiding principles associated with DevOps (see Section 4.1.1).
Also, based on the identified descriptions from the literature, we mapped the practices to principles
to show how DevOps practices support the principles (see Table V).
Practices: For DevOps practices, we investigated the studies which have explicitly proposed or
presented activities to be executed in the context of DevOps. After identifying the practices and in
order to better illustrate them (e.g., in terms of the coverage of DevOps in the domain) they have
been categorized in accordance with the knowledge areas and corresponding sub-knowledge areas
presented by Software Engineering Body of Knowledge (SWEBOK) [3] as an international standard
for providing a comprehensive categorization of software engineering.
We specified each practice (P01 to P29) through open coding. In this process, the extracted data
has been labeled by codes. Each code represents the identified data as a DevOps practice. While
performing the open coding, we compared the concepts with each other to find similarities based on
the extracted information and then assigned them to the same label (see P01, P02, P04, P07, P10,
P11, P12, P16 and P17 in Table V). Figure 1shows how labeling is performed for P07 (continuous
integration) as an example.
Thereafter, the practices were placed under the SWEBOK knowledge areas based on the
categorization process. To this end, we considered 9 groups as follows: Management, Construction,
Configuration, Test, Process, Quality, Tools and methods, Requirement and Design. Then, we
assigned each DevOps practice to the corresponding group based on its type and extracted
information from the literature (see Appendix A). After the grouping, we again reviewed the
extracted data for DevOps practices and also the descriptions presented by SWEBOK for the sub-
knowledge areas. Afterwards, we decided about assigning the practices to relevant sub-knowledge
Table IV. Number of studies per objective
Reference #Study Claimed Benefit Challenge Practice Principle Demonstrated Benefit
[14,16,21,23,35,39,43,45,51,59,60] 11 X× × × ×
[7,20,29,36,58,63] 6 ×X× × ×
[24,26,27,42,50,55] 6 × × X× ×
[10] 1 × × × X×
[47] 1 × × × × X
[41,57] 2 X X × × ×
[28,32] 2 × × X×X
[61] 1 X×X× ×
[44] 1 X× × X×
[53] 1 X× × × X
[18] 1 ×X×X×
[25] 1 × × X X ×
[30,49] 2 X X X × ×
[13] 1 X X × × X
[22,40,56] 3 X X X X ×
Number of studies per objective 22 15 15 7 5
Total number of studies 40
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
8JABBARI ET AL.
[B] Study ID [F] Practices? Open Code
Virman2015 (111)
Not only share within component teams but integrate beyond component boundaries, at product
integration level. Further this stage of process optimization refers to achieving automation such
that as soon as developer delivers the change the build systems detects that (may even be a
scheduled event at end of day) and triggers a build carries out sanity tests and posts the build to a
repository. This has to be a repeatable continuous process all across the development cycle.
1. Continuous planning
2. Continuous integration
3. Continuous deployment
4. Continuous testing
Fitzgerald2015 (040)
Continuous integration requires a link between development and operations and is thus very
relevant to the DevOps phenomenon (Debois, 2009). Within continuous integration, a number of
further modes of continuous activities can be identified, namely continuous deployment and
continuous delivery (Humble, 2010; Lacoste, 2009). These concepts are related in that continuous
delivery is a prerequisite for continuous deployment, but the reverse is not necessarily the case.
That is, continuous deployment refers to releasing valid software builds to users automatically,
whereas continuous delivery refers to the ability to deploy the software to some environment, but
not automatically deploying to customers.
Continuous integration
Fitzgerald2014 (039)
Continuous integration requires a link between development and operations and is thus very
relevant to the DevOps phenomenon [16]. Within continuous integration, a number of further
modes of continuous activities can be identified, namely continuous deployment and continuous
delivery [28, 35]. These concepts are related in that continuous deployment is a prerequisite for
continuous delivery, but the reverse is not necessarily the case. That is, continuous delivery refers
to releasing valid software builds to users automatically, whereas continuous deployment refers to
the practice of deploying the software to some environment, but not automatically delivering to
customers.
Continuous integration
Farroha2014 (004)
The best practices in DevOps include active stakeholder participation. Without stakeholder
participation, expectations are not managed and assumptions are made that 9 times out of 10
results in a capability not requested by the stakeholder.
Automated testing facilitates success so that human focus can be on areas that cannot be
automated.
Integrated configuration management, integrated change management, continuous integration,
integrated deployment planning, continuous deployment, production support, application
monitoring, and automated dashboards. If a company uses these ten best practices their success
rate of adapting DevOps increases dramatically. (The goal is to enhance the ability of an
organization to rapidly adapt to market and environmental changes in productive and cost
effective ways.)
4. integrated change management
5. continuous integration
6. integrated deployment planning
7. continuous deployment
8. production support
9. application monitoring
Figure 1. A sample of labeling identified data
areas as well. For those practices not described in the literature, we also decided the assignment
using our judgement. The aim of this categorization is to provide an illustration of the DevOps
practices that shows the coverage of DevOps application in the software areas.
Challenges: Similar to the procedure for analyzing the extracted data of DevOps practices, we
also performed for the challenges. The same categorization (SWEBOK [3]) was used to present the
challenges categorized into each software area (see Table VI).
Benefits: For DevOps benefits, we considered two main types of benefits (i.e., claimed and
demonstrated benefits). The reason why we assumed these two types is to evaluate DevOps with
regard to the evidence presented for DevOps benefits. We extracted any advantage, outcome,
positive consequence, benefit, etc. discussed in the selected studies as a DevOps benefit. Then, we
divided them into the two main groups (claimed and demonstrated benefits) based on the evidence
presented in the literature e.g., type of study (see Table III).
After identifying the benefits and in order to provide a better illustration, we used SVM [34]
to categorize the extracted benefits according to the corresponding type of value component (see
Section 4.3).
To categorize the benefits into the value perspectives of SVM, we considered the perspectives
as four groups (Financial, Customer, Internal business process and Innovation and learning, see
Section 2.2). Then, according to the description provided for each perspective, we grouped the
identified benefits. Next, we consider corresponding value aspects and components related to each
perspective and based on their definitions, we categorized the DevOps benefits into the relevant
value perspective (see Section 4.3). As in SVM, the value perspectives have overlaps in terms
of value components, we have also captured the overlap during the categorization. Although the
categorization of DevOps practices, challenges and benefits is more theoretical and based on our
judgement it can be considered as an initial basis to provide a consolidated view on DevOps through
a systematic approach.
Relations: To determine dependencies among the practices and relations between the practices
and corresponding benefits, we used the concept of benefits dependency network [9,46]. The
structure of benefits dependency networks is shown in Figure 2.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 9
X
X
X
Practice
Benefit
Practices Benefits
X
X*
Figure 2. Structure of benefits dependency network
The dependency between practices indicates which practice has to be in place for another practice
to work. Also, the link between practices and benefits indicates that a practice or a chain of practices
leads to one or more benefits. By the use of the relations, we may derive what is needed to achieve
a certain benefit. For example (Figure 2), to achieve the benefit marked with “X” we need to
implement three practices. The dependency also shows in which order the practices have to be
implemented given their dependencies. In addition, we determine that another benefit “X*” is also
achieved through the implementation of the practices.
4. RESULTS
This section presents the results of our study to address each research question as follows:
4.1. RQ1: What are the principles and practices associated with DevOps in the peer-reviewed
literature?
4.1.1. DevOps principles. Five primary studies have presented the principles of DevOps with
the focus on teamwork, communication, and collaboration between development and operations
[10,18,22,44,56].
Farroha and Farroha [22] mention that the focus is on team and communication rather than tools
and processes to improve business value that points to simplicity and agility in technology, process
and human factors in addition to enabling trust and supporting innovation which has been also
highlighted by McCarthy et al. [40] as DevOps principles.
Virmani [56] states that DevOps extends agile principles to the entire software delivery pipeline,
in which the ability to run operations along with development is a must through a set of continuous
activities like planning, test, integration, deployment, and monitoring. Also, Fitzgerald and Stol
[25] mention that DevOps principles reflect the need of changing perspective on responsibility
and measurement, e.g., business metrics, time to deploy a new version, etc., which have been
also targeted in agile principles like “business people and developers must work together daily
throughout the project” and “working software is the primary measure of progress” [1].
Six principles (Pr1 to Pr6) identified in this study are as follows:
•Pr1: Sharing knowledge. Break down barriers between development and operations by
enabling trust, sharing tools and supporting communication, collaboration and integration
[10,18,22,25,40,44,56]. The focus is on teamwork, communication and collaboration
rather than tools and processes e.g., maximize team performance [22,40].
•Pr2: Automation. Automate build, deployment, and test to achieve fast software creation
through repeatable processes and operations resulting in e.g., short lead times, rapid delivery,
fast feedback from end-users, etc. [10,18,25,44,56].
•Pr3: Shared responsibility. Establish collaboration between all team members involved
in development and operations. Shared responsibility is an integral part of DevOps culture
[10,25].
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
10 JABBARI ET AL.
•Pr4: Continuous activity. Improve the business value of the software products and services
by ensuring continuous software development. Continuous activity can apply to the whole
software development life-cycle like continuous planning, continuous integration with the
main focus on early integration, continuous deployment, continuous testing and continuous
monitoring [22,56].
•Pr5: Measurement. Gain an understanding of e.g., the delivery capability and set goals to
improve it, as DevOps specifies that improvement can only be possible through measurement.
Measurement contains various measures from business metrics (like revenue) to test coverage
and time to deploy (e.g., a new version) [25]. Bang et al. [10] state that the measurement
perspective of DevOps covers the measurement of the process, value, cost and technical
metrics.
•Pr6: Composability. Follow certain design principles to achieve DevOps such as the minimal
functions with least dependencies, portability, predictable contract and maximized human
value [40].
4.1.2. DevOps practices. 29 practices have been identified during the investigation of the selected
studies.‡As mentioned in Section 3.5, the practices are categorized into the fundamental knowledge
area (KA) and sub-knowledge areas (sub-KA) based on SWEBOK [3] to provide a better illustration
of the areas covered by DevOps. Table Vshows the DevOps practices (P01 to P29). The last
column (Pr) indicates the mapping of DevOps practices to the principles done by using the extracted
definitions and our judgement. However, we did not find any explicit connection between the
identified practices and the principle Pr3 (i.e. shared responsibility). ”Shared responsibility” is
essential for DevOps as it enables and facilitates other activities like communication, decision
making and development and operations activities. Moreover, practices P08, P09, P13, P14, P15,
P16, P19, P20, and P21 were not been mapped to any of the principles. For some of these practices
e.g. P09 and P13 could be speculated to contribute to Pr4 (continuous activity), however since no
such indication was present in the primary study reporting them, we did not create such a mapping
between them.
As indicated in Table V, all knowledge areas are almost covered by the practices that shows the
adoption of DevOps has a potential implication for the whole process of software development.
However, only 8 practices (P01, P02, P04, P07, P10, P12, P16, and P17) identified have been
extracted from more than one study. The majority of them are from individual papers that can
indicate a lack of consensus in the community. We provided the definitions of each practice proposed
in the selected studies available in Table VIII, Appendix A.
4.2. RQ2: What are the challenges explicitly associated with DevOps in the peer-reviewed
literature?
Among the studies included in this review, 15 reported 41 challenges as shown in Table VI. The last
column (Pr) indicates the mapping of challenges to principles based on the extracted data from the
selected studies and our judgement.
The larger number of challenges, e.g., C01, C02, C03, C04, C05, C06, C08, C09, C13, C14, C17,
C18, C19, C22, C24, C25, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C38 and 41 are
about DevOps adoption [7,13,18,20,22,30,29,36,40,41,57,58,63].
In particular, lack of tools for automation, as one of the central DevOps principles (Pr2), has been
mentioned as a barrier to implement DevOps. Miglierina [41] specifies that while automation in
deployment, maintenance, and monitoring is the main requirement of DevOps, there is a series of
downsides (e.g., lack of infrastructure to bring the developers and operators closer) that hinders full
automation (C34, C35 and C36).
In this regard, Wettinger et al. [58] also mention that it is hard to implement holistic DevOps
automation for a specific application (C28) due to rapid changes in DevOps automation approaches.
‡An initial list of the practices has been early published to support our proposed definition for DevOps in DevOps
workshop at XP2016 [31].
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 11
Lack of consistent definition (C18) [20], KPIs (C08) [30], standards (C09) [41], maturity model
for DevOps (C19) [22] and lack of training and skill to implement DevOps (C31) [22] have also
stressed a shortage of infrastructure to adopt DevOps.
Table VI shows the identified challenges of DevOps (C01 to C41). Like the practices, the
challenges are categorized into knowledge areas (KA) and sub knowledge areas (sub-KA) of
SWEBOK [3].
4.3. RQ3: What are the benefits (claimed and demonstrated) explicitly associated with DevOps in
the peer-reviewed literature?
To investigate the benefits of DevOps, we have separated them into two main groups, namely:
claimed benefit (CB), which are proposed in the selected studies without e.g., any practical evidence
and demonstrated benefit (DB) come from practice or have been observed in practice demonstrated
through empirical investigation/evaluation. Then using the definitions from SVM as guidance we
have mapped the identified benefits to the value perspectives proposed by Khurum et al. [34]. This
provides a consolidated view of DevOps benefits as follows:
4.3.1. Financial. As mentioned earlier, the financial perspective is related to long-term strategic
objectives and concerned about revenue growth, costs, profit margins, cash flow, net operating
income and customer value analysis. Only one claimed benefit has been identified and mapped
to this perspective, as follows:
(CB01) Maximize investment outcome. Farroha and Farroha [22] consider DevOps as a business
strategy that enables creating an environment, in which operations can improve continually to
ensure that customers receive services and features continuously resulting in maximizing investment
outcome.
4.3.2. Customer/ perceived value perspective. This value perspective includes time (e.g., time to
market), quality, performance and service, cost, etc. to satisfy customers [34]. In total 12 claimed
benefits have been identified and mapped to this value aspect.
Table V. DevOps Practices
KA sub-KA Practice (P) Ref. Pr
Software
engineering
management
Project planning P01: Continuous planning [24,56] Pr4
P02: Continuous feedback [27,56] Pr4
Project enactment
P03: Continuous monitoring [56] Pr4
P04: Automated (application) monitoring [22,27,28] Pr2
P05: Automated feedback [55] Pr2
P06: Automated dashboard [22] Pr2
Software
construction
Practical considerations P07: Continuous integration [22,24,25,56] Pr4
Construction fundamentals P08: Prototyping application [28]•
Software
configuration
management
Release management and
delivery
P09: Integrated deployment planning [22]•
P10: Continuous deployment [22,24,56] Pr4
P11: Automated deployment [26,61] Pr2
P12: Continuous delivery [24,26,61] Pr4
P13: Cooperative application configuration [42]•
Management of the SCM P14: Staging application [28]•
P15: Integrated configuration management [22]•
Configuration control P16: Change management [22,30]•
Test Test techniques P17: Continuous testing [55,56] Pr4
P18: Automated testing [22] Pr2
Software Process Process definition P19: Process standardization [49]•
P20: Production support [22]•
Software quality Practical considerations P21: Use of data to guide QA [49]•
Software
engineering
tools & methods
Methods
P22: Infrastructure as code [50] Pr6
P23: Key performance metrics [27] Pr5
P24: Continuous application performance modeling [55] Pr4
Tools P25: DevOps maturity evaluation ontology [40] Pr5
P26: Elasticity practice [32] Pr6
Requirements Requirements fundamentals P27: Defining requirements [28] Pr6
Requirements process P28: Stakeholder participation [22] Pr1
Software design Structure and architecture P29: Designing architecture [28] Pr6
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
12 JABBARI ET AL.
Table VI. DevOps Challenges
KA sub-KA Challenge Ref. Pr
Software
engineering
manage-
ment
Project
planning
C01: Inability to resolve daily issues without an external party. [40] Pr3
C02: Lack of management structure between developers and operators, e.g.,
responsibility definition that might lead to extra workload for operators.
[22] Pr1
C03: Lack of skilled DevOps individuals. [22]•
C04: Central management versus delegating to single dev/product team
(e.g., for application lifecycle management).
[13] Pr3
C05: Conflicts and inefficiencies between developers and operators due to
communication, cooperation, and collaboration gap.
[30] Pr1,3
C06: Maintaining accountability while promoting shared responsibility
between development and operations.
[30] Pr3
Initiation & scope C07: Bringing in synergy in knowledge management and governance. [30] Pr1
Measurement C08: Lack of KPIs (i.e., success metrics) across DevOps [30] Pr5
Software
construction
Construction
fundamentals
C09: Lack of standards for growing cloud services from different providers
in e.g., cloud-based DevOps.
[41] Pr5
Software
configuration
management
Release management
and delivery
C10: Negatively affect the performance [of the client] and availability [of
the service] due to the frequency of delivery.
[49] Pr4
C11: Probability of having extra delays due to external resources. [63] Pr2,3
C12: The complexity of deployment due to increasing demand for DevOps
integration in a development organization.
[7] Pr2,4
C13: The complexity of deployment due to increasing demand for
continuous delivery in a development organization.
[7] Pr2,4
Management of SCMC14: The complexity of software deployment and configuration. [18] Pr2,4
Test Test techniques C15: Automation test results in a high load on the infrastructure. [36] Pr2,4
Test process C16: The complexity of validation test, e.g., for continuous integration [18,56] Pr2,4
Software
Process
Process
definition
C17: Bringing in synergy in a release cycle. [30] Pr2,4
C18: Lack of consistent definition for DevOps. [20]•
Implementation C19: Lack of maturity for DevOps process. [22] Pr5
Software
quality
Practical
considerations
C20: Lack of portability for e.g., legacy code written in e.g., C, C++, etc. [18] Pr6
C21: Negatively affect the performance of the client and availability of the
service due to the frequency of delivery.
[49] Pr5
C22: Lack of scalability. [29,36,41] Pr2
C23: Balancing speed & flexibility with quality &control. [13] Pr5
C24: Programmability to adopt DevOps in carrier networks. [36]•
C25: Different mentality for stability. For developers, constant changes are
a way of life, but for operators, instability is the result of external events.
[36] Pr3,4
C26: Lack of flexibility (elasticity) that e.g., the cloud seems to promise. [41]•
Software
engineering
tools &
methods
Methods
C27: Hard to integrate automation approaches, e.g., tools, open source
artifacts, due to a huge variety of options.
[57,58] Pr2
C28: Hard to choose and combine automation approaches due to rapid
changes (they appear, change, and disappear rapidly).
[58] Pr2
C29: DevOps knowledge is not systematically captured and managed. [58]•
C30: Hard to implement holistic automation for a specific application. [58] Pr2
C31: Lack of training for DevOps. [22] Pr1
Tools
C32: The difficulties in implementing tools for deployment. [18] Pr2,4
C33: Bringing in synergy in tools. [30] Pr2,4
C34: Lack of automation in deployment. [41] Pr2
C35: Lack of automation in maintenance. [41] Pr2
C36: Lack of automation in monitoring. [41] Pr2,5
Requirements Requirements
fundamentals
C37: Competing priorities (prioritizing the constraints by impact and having
authority to change or influence the change).
[22] Pr5
Software
design
Structure
and architecture
C38: The complexity of DevOps architecture design for a cloud application. [29] Pr2
Key Issues C39: Probability of committing more errors due to external resources. [63] Pr6
Software
maintenance
Maintenance
process
C40: Hard to maintain infrastructure setup due to having legacy code. [18] Pr2,4
C41: The complexity of troubleshooting (continuous changes in an
infrastructure and involvement of multiple tasks for developer and operator.)
[36] Pr3,4
1. (CB02) Process optimization. Virmani [56] states that with ongoing growth of the number of
customers it is a must to get internal delivery processes in line with ongoing market demands.
It is claimed that the introduction of DevOps helps to balance out time to market and resources
in order to optimize processes and activities in the software delivery lifecycle.
2. (CB03) Fast and frequent releases. The time-to-market reduction is claimed as a DevOps
benefit in a number of studies. According to Virmani [56], DevOps enables organizations
to reduce time to market through optimizing the software delivery processes. Therefore,
CB3 can be considered as a consequence of CB2. Also, it is claimed that adopting
DevOps reduces the time-to-market as it helps to achieve improved deployment frequency
[22,30,35,41,45,57,61].
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 13
3. (CB04) Scalability. Virmani [56] claims that the use of IaaC (Infrastructure as a Code) in
DevOps ensures that the entire automation is scalable.
4. (CB05) Portability. In addition to CB04, Virmani [56] claims that by the use of IaaC in
DevOps that maintains everything as code the entire automation is also portable.
5. (CB06) Cost saving. Virmani [56] says significant cost saving will be achieved through
automated deployment as teams can share a pool of resources and there is no need to reserve
them (e.g., hardware resources) [56].
6. (CB07) Balancing out cost and quality. Effectively balancing out cost and quality is claimed
by Virmani [56] as a DevOps benefit in an organization, as it can optimize the capability in
the release lifecycle and processes.
7. (CB08) Quick responses. DevOps enables to be responsive [22,57,61]. Wettinger et al.
[57,61] consider DevOps as a paradigm that can tightly integrate developers and operators to
continuously deliver new iterations of a specific application that makes it possible to meet the
expectation of fast feedback to problems and requests.
8. (CB09) Less failure rate. Park et al. [45] mention that DevOps is a response to the inter-
dependence of development and operations. It leads to lower failure rates in new releases by
focusing on communication and collaboration between developers and operators.
9. (CB10) Maximize maintainability. Erich et al. [21] point out deployment and maintenance
improvements achieved through a collection of practices for close collaboration between
development and operations personnel. Also, Park et al. [45] state that by the adoption of
DevOps maximal maintainability can be achieved.
10. (CB11) Improve reliability. Improving software reliability is also considered as an
advantage of DevOps [22,56]. Wettinger et al. [59] mention that by the use of some DevOps
tools like Chef and Puppet it is possible to increase reliability in processes management.
11. (CB12) Customer satisfaction. DevOps ensures customer satisfaction by continuously
increasing service and feature quality [22].
Three demonstrated benefits have been reported in the literature mapped to customer/perceived
value perspective as follows:
1. (DB01) Improve efficiency. DevOps leads to speed improvement and efficiency by the
combination of automated test and deployment [53].
2. (DB02) Speed up development. P´
erez et al. have introduced a tool to handle the gap between
development and operations in order to speed-up the development cycle [47].
3. (DB03) Better performance. Jayaram [32] presents preliminary empirical evidence to
indicate that DevOps improves the performance of an application.
4.3.3. Internal Business Process/ production value. This perspective is focused on the processes
creating the value for the customer and includes all activities providing value to the customer such
as productivity, the cycle time of different activities and costs. The corresponding claimed benefits
identified from the literature and mapped to this value aspect are as follows:
1. (CB13) Shortened development cycle. DevOps speeds up the processes from development
to operation [13,41,44]. Bruneo et al. [14] say DevOps can promote collaboration
between developers and operational teams that leads to shortening the loop between software
development, deployment, and operation.
2. (CB14) Flexible software development. DevOps enables development and operations teams
to be more flexible by providing close collaboration in implementing customer requirements
[13,22].
3. (CB15) Increase productivity. To maximize productivity and quality, DevOps attempts to
bridge the information gap between project teams and enforces rigorous processes that ensure
real-time communication [16].
4. (CB16) Time reduction. DevOps leads to time reduction from implementation to deployment
[35]. N´
emeth et al. [43] mention that by applying DevOps principles to service design, the
creation time will be reduced that supports continuous operation.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
14 JABBARI ET AL.
5. (CB17) Better requirement validation. DevOps also bridges the communication gap
between development and customers that leads to the continued involvement of the customer
during development process [39]. The idea of continuous involvement can help developers to
validate their understanding of requirements particularly in agile projects that requirements
frequently change [39].
6. (CB18) Higher quality. The merge of formal methods with e.g., Scrum development process
within DevOps context can address the communication gap between team members and
also with customers. In fact, DevOps can embrace development, quality assurance and
operations that promote higher quality. Olszewska and Wald´
en [44] claim that intra-team
communication provided by DevOps largely enhances the formal modelling capabilities that
actuate knowledge sharing, and raise understanding and awareness in the project.
7. (CB19) Identify bottlenecks. “Identifying bottlenecks and prioritizing the improvement areas
are identified as important in DevOps”, mentioned by Olszewska and Wald´
en [44].
8. (CB20) Automate test and integrated deployment. By DevOps the test environment should
reliably reflect the production, “DevOps allows automated testing of individual projects as
well as the integrated system deployments” [53].
9. (CB21) Frequently test design adjustment. By shifting to DevOps, runtime iterative and
monitoring-based test can be designed and executed that aligns with dynamic and agile needs
[51].
10. (CB22) Process massive data. “The heart of DevOps is the ability to process massive amount
of structured and unstructured data through using Big Data analytics approach” [40].
The five demonstrated benefits mapped to the customer/ perceived value perspective are as
follows:
1. (DB04) Autonomously operation. DevOps allows developers to work autonomously
resulting in being more willing to accept changes and providing more frequent updates [53].
2. (DB05) Communication overhead reduction. DevOps leads to “a reduction in
communication overhead for making coordinated changes” [53].
3. (DB06) Flexible architecture at lower cost and shorter time. Hosono [28] proposes
a framework, which supports DevOps practices, to provide comprehensive platforms for
developing and operating cloud applications. Hosono [28] through conducting case studies
shows that the framework allows developing architecture in the cloud at lower cost and shorter
time.
4. (DB07) Gradual application development. Hosono [28], through evaluating the proposed
framework (DevOps platforms for cloud applications), also states that the framework helps
gradual application development, e.g., agile developments. It has been also mentioned that
this framework is most effective for small and medium businesses (SMB) rather than large-
scale enterprise applications.
5. (DB08) Automated activities. Boschetti et al. [13] have proposed an automated platform over
Chef framework to automate activities for application development that guarantees consistent
versioning, packaging, deploying and executing.
4.3.4. Internal Business Process/ differentiation value. According to the SVM, differentiation value
overlaps with the perceived value of the value perspective ‘Customer’ in terms of value sub-aspects
and components. Therefore, the claimed benefits mapped to Customer/ perceived value, i.e., CB02
to CB11, have been also assigned to this category as differentiation values of the perspective
‘Internal Business Process’. Regarding the differentiation values, three claimed benefits have been
mapped:
11. (CB23) Improve deployment quality. Improvement in the quality of software deployment
will be achieved by DevOps [22].
12. (CB24) Response to market. The aim of DevOps is to bring agility into IT infrastructure
and service management [59]. Miglierina [41] mentions that together with novel software
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 15
development methods e.g., agile and DevOps, cloud application deployment and management
enables companies to respond to market by continuous delivery.
13. (CB25) Continuous delivery. Formal modelling functions within DevOps context can
promote various quality aspects and continuous delivery [44].
For the demonstrated benefits, because of the overlap between ‘differentiation’ and ‘perceived’
values described above, DB01 and DB02 are also mapped to the differentiation values.
4.3.5. Innovation and learning/ value of technology. Innovation and learning are concerned with
skills and capabilities needed to support the value-creating internal process. 15 claimed benefits
have been mapped to this value aspect as follows:
1. (CB26) No hardware. By adopting DevOps, there is no need for keeping dedicated hardware
anymore [56].
2. (CB27) Repeatability. By adopting DevOps, e.g., deployment can become a consistent
repeatable process [56].
3. (CB28) Predictability. DevOps enables companies to have more predictability in releases
as it helps to be continuously responsive to changes e.g., market changes [56]. Park et al.
[45] have mentioned that DevOps leads to predictability as a consequence of improving
deployment frequency like faster time to market, a lower failure rate of new releases, a shorter
lead time between fixes, etc.
4. (CB29) Maximize efficiency. Maximizing the efficiency of operational processes would
be expected by adopting DevOps [45]. Generally, DevOps leads to increased efficiency in
organizations as a whole [49,56].
5. (CB30) Communication and collaboration. DevOps aims to address the split between
developers and operators in order to automate e.g., the complete deployment process from
the source code to operational environment [60]. DevOps improves communication and
collaboration between development and operations [22,23,44,61].
6. (CB31) Standardization. Through DevOps, a stronger focus on metrics and process
standardization would be achieved [49].
7. (CB32) Maximize security. Like maximizing predictability and efficiency, DevOps can
maximize the security of operations [45].
8. (CB33) Code quality. DevOps can improve the quality of code through practices like a
pairing of developers [22].
9. (CB34) Self-organizing. DevOps practices support autonomy through self-organizing
principles to work autonomously [22,53].
10. (CB35) Interoperability. DevOps provides interoperability of software development and
operations that ensures automation, knowledge sharing and maximal data access [16].
11. (CB36) Transparency. Automated practices with minimal human-intervention make
developers and operators free to focus on tasks requiring highly specialized skills. Such tasks
provide and utilize information transparency and dissemination that is enabled by the use of
DevOps [16].
12. (CB37) Core business focusing. Together with Agile and DevOps, cloud applications enable
users to focus on its core business instead of being busy because of making things work [41].
13. (CB38) Reusable team. As a consequence of knowledge sharing provided by DevOps, the
expertise of each team member is known and can be used if needed, which points to the
concept of a reusable team [44].
14. (CB39) Better resource management. DevOps practices like automated deployment lead to
better resource management as teams share a pool of resources [56].
15. (CB40) Better knowledge management. “DevOps analytics has the ability to analyze both
explicit and implicit knowledge and tie them together to discover new contexts and new facts”
that leads to process and manage a massive amount of structured and unstructured data [40].
Also, one demonstrated benefit is identified and mapped to innovation and learning/value of
technology:
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
16 JABBARI ET AL.
1. (DB09) Better agility. Jayaram [32] presents preliminary empirical evidence to indicate
DevOps improves agility for the implementations.
5. DEVOPS BENEFITS DEPENDENCY NETWORK
In this section, we present a network of the DevOps practices specified in this study. Then, the
linking of the practices to the corresponding DevOps benefits is described.
DevOps practices dependency network: In order to illustrate the DevOps practices and probable
dependencies between them, we identified the relations (r1 to r24) between the specified DevOps
practices (P01 to P29) from the selected studies. For example, we defined r8as a relation from P02
(Continuous feedback) to P01 (Continuous planning) based on the statement presented by Virmani
[56] that is “DevOps allows you to do continuous planning by always having a continuous channel
of feedback.” To this end, we reread those articles, from which we had identified DevOps practices.
Then, according to the new findings, we could specify the relations between 23 (out of 29) practices.
Figure 3presents the DevOps practices dependency network based on the specified DevOps
practices and their corresponding relations. Table IX (Appendix B) shows the definition of the
relations.
Only a few studies are involved in investigating dependencies and relations between DevOps
practices. Some relations have been extracted directly from the selected studies [22,28,55,56,61]
and some other (dash colored orange in Figure 3) assumed by the authors during this study and
using two studies [30,42].
To investigate how these identified relations position the DevOps practices in terms of activity
type, we have mapped them to a workflow proposed as DevOps cycle [2] which has also been used
by one of the selected studies [22]. As shown in Figure 4, we have considered 6 types of activities:
plan, code, test, release, deploy and monitor.
r19 r11 r12 r13 r14 r20
r7 r1
r5
r17
r3
r18
r21
r10
r4
r16
r22
r9
r15
r24
r6
r8
r2
Integrated deployment planning
Continuous application
performance modeling
Change management
Continuous planning
Continuous
monitoring
Continuous
feedback
Cooperative application
configuration
Continuous
deployment
Stakeholder
participation
Defining
requirements
Designing
architecture
Prototyping
application
Staging
application
Automated
monitoring
Automated
dashboards
Continuous
delivery
P28 P27 P29 P08 P14 P04 P06
Infrastructure
as code
P22
Continuous
integration
P07
Automated testing
P18
Automated feedback
P05
Continuous
testing
P17
Automated
deployment
P11 P10
P12
P09
P24
P13
P01
Integrated
configurationmanagement
P15
P03 P02
P16
r23
PDevOps Practice
Relation based on literature
Relation based on assumption
Figure 3. DevOps Practices Dependency Network
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 17
r22
r17
r7
Deployment
r1
r14
r18
r4
r21
Monitor
r15 r20
Plan
r9
Release
Test
Code
r11
r12
r3 r2
r8
r16
r19
Dev Ops
r23
r24
r5
r13
r10
r6
P07
P22
P18 P17 P15 P13
P12 P14
P09
P11P10
P28P27P29
P24
P16
P08
P01 P02 P06
P05 P04P03
Figure 4. DevOps practices mapped to DevOps workflow
Figure 4shows that the mapping of the relations does not conform to the sequence provided
by the workflow. This evidently indicates that a difference between the sequential phase-based
definition (DevOps cycle [2]) and a sequence of practices used to adopt DevOps. In other words,
the application of DevOps is not compatible with a categorical view of phase-based activities (e.g.
grouping them into planing, coding, testing, etc.).
Therefore, instead of a phase-based view, we consider all individual sequences of the practices
which have constructed the DevOps practices dependency network. Figure 5shows 10 different sets
of the individual sequence of the practices, each of which can be considered as a means to apply
DevOps based on needs and desired benefits.
Now, we have the different sets of DevOps practices including their dependencies and relations
identified from the selected studies. The next step is to identify the corresponding benefits to e.g.,
recommend a kind of decision support to choose an appropriate set of DevOps practice.
Linking practices to benefits: To this end, we attempted to identify the relations between the
practices (especially those existing in the DevOps practices dependency network) and the benefits
(40 claimed and 9 demonstrated). In fact, we aimed to meet the overall question of this study (‘Which
DevOps practice and ultimately a sequence of DevOps practices give which benefit(s)?’).
Similar to the procedure for identifying the relations among the practices we specify the relations
between practices and corresponding benefits. For example, we linked CB03 (Fast and frequent
release) and CB08 (Quick responses) to P12 (Continuous delivery) based on the statement presented
by Wettinger et al. [61] that is “The main promise of DevOps is to enable continuous delivery of
software in order to enable fast and frequent releases. This enables quick responses to changing
requirements of customers and thus may be a critical competitive advantage.” Eventually, we
could identify the links between 12 DevOps practices (P01, P02, P03, P04, P05, P07, P10, P11,
P12, P16, P18, P24) and 13 DevOps benefits (12 claimed benefits CB03, CB06, CB08, CB12,
CB15, CB16, CB18, CB27, CB28, CB30, CB32, CB40 and one demonstrated benefit DB09)
[15,22,24,30,55,56,61]. As shown, only 7 peer-reviewed studies have discussed the link between
practices and benefits. Table X(Appendix C) presents the links identified between the practices
and benefits. Figure 6as an example also indicates the expected benefit(s) for one sequence of
development practices derived from the DevOps practices dependency network (Figure 3).
By specifying the practice-benefit association and using the suggested value category for
benefits, we could provide the link between the value perspectives (presented in Section 2.2) and
corresponding DevOps benefits and practices. Based on this linking, we have provided a holistic
view of the DevOps practices, benefits and values, in the form of a value-based practices dependency
network for DevOps illustrated in Figure 7.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
18 JABBARI ET AL.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
Continuous
integration
Continuous
integration
Continuous
integration
Continuous
integration
Continuous
integration
Continuous
integration
Stakeholder
participation
Infrastructure
as code
Infrastructure
as code
Automated
testing
Automated
testing
Automated
testing
Automated
testing
Automated
testing
Defining
requirements
Automated
deployment
Continuous
deployment
Continuous
delivery
Continuous
testing
Continuous
testing
Continuous
testing
Continuous
testing
Continuous
delivery
Continuous
deployment
Designing
architecture
Integrated deployment
planning
Continuous
deployment
Integrated configuration
management
Integrated configuration
management
Prototyping
application
Automated
feedback
Continuous
delivery
Continuous
monitoring
Staging
application
Continuous application
performance modeling
Continuous
feedback
Automated
monitoring
Continuous
planning
Automated
dashboard
Cooperative
application configuration
Continuous
delivery
P22 P12
P07 P18 P09
P22 P10 P12
P07 P11 P10 P12
P07 P18 P17 P10 P12
P07 P18 P17 P15 P13
P07 P18 P17 P05 P24
P27 P29 P08 P14 P04P28 P06
P18 P17 P15 P03 P02P07 P01
P03 P02 P16 P01P18 P17 P15P07
Continuous
integration
Automated
testing
Continuous
testing
Integrated configuration
management
Continuous
monitoring
Continuous
feedback
Change
management
Continuous
planning
Figure 5. 10 individual sets of DevOps practices
Each colored collection of benefit(s) in Figure 7shows which sequence of DevOps practices
should be applied. As an example shown in Figure 8, by applying each sequence of DevOps
practices, which ends up in blue color (P09, P13 or P24) we would expect to achieve the blue
collection of DevOps benefits.
6. DISCUSSION
6.1. State and maturity
DevOps principles and practices. The results of this study indicate the need for further studies and
specifying the principles and practices associated with DevOps structurally. The number of studies
discussing DevOps principles and practices is scarce. Only seven articles among the selected ones
have discussed DevOps principles [10,18,22,25,40,44,56].
CB03, CB08,
CB15, CB28,
CB30
CB03,
CB08,
CB12,
CB15,
CB16,
CB18,
CB28,
CB30,
CB32,
CB40,
DB09
1: P18P07 P17 P15 P03 P02 P16 P01
CB32 CB08, CB18,
CB32 CB12 CB15, CB30,
CB40, DB09
CB08, CB16,
DB09
Figure 6. Expected benefit(s) associated to a sequence of DevOps practices
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 19
r19 r11 r12 r13 r14 r20
r7 r1
r2
r5
r17
r3
r18
r21
r10
r4
r16
r22
r9
r15
r24
r6
r8
r2
r2
CB03
CB06
CB08
CB12
CB15
CB16
CB18
CB27
CB28
CB30
CB32
CB40
DB09
DevOps Practices Relations DevOps Benefits Value Perspectives Value Components
Value-based DevOps Practices Dependency Network
CB03. Fast and frequent
releases
CB06. Cost saving
CB08. Quick responses
CB12. Customer satisfaction
CB15. Increase productivity
CB16. Time reduction
CB18. Higher quality
CB03. Fast and frequent
releases
CB06. Cost saving
CB08. Quick responses
CB27. Repeatability
CB28. Predictability
CB30. Communication and
collaboration
CB32. Maximize security
CB40. Better knowledge
management
DB09. Better agility
Customer/
perceived value
Internal Business Process/
production value
Internal Business Process/
differentiation value
Innovation and learning/
value of technology
- Functionality
- Reliability
- Usability
- Maintainability
- Portability
- Process wrt. time
- Process wrt. quality
- Process wrt. cost
- Network externalities
- Complementary value
- Hedonic value
- Market requirements value
- Physical value wrt. time
- Physical value wrt. quality
- Physical value wrt. cost
- Functionality
- Reliability
- Usability
- Maintainability
- Portability
- Functionality
- Reliability
- Usability
- Maintainability
- Portability
- Process wrt. time
- Process wrt. quality
- Process wrt. cost
- Differentiation wrt. perceived
value (network externalities)
- Differentiation wrt. perceived
value (complementary value)
- Differentiation wrt. availability
- Differentiation wrt. sales
and promotion
- Differentiation wrt. cost
and price
- Human capital value
- Customer capital value
- Structural capital value
- Intrinsic value of technology
- Application value of technology
- Market value size
- Market value type
Integrated deployment planning
Continuous application
performance modeling
Change management
Continuous planning
Continuous
monitoring
Continuous
feedback
Continuous planning
Cooperative application
configuration
Continuous deployment
Continuous deployment
Continuous deployment Continuous delivery
Stakeholder
participation
Defining
requirements
Designing
architecture
Prototyping
application
Staging
application
Automated
monitoring
Automated
dashboards
Continuous delivery
Continuous delivery
Continuous delivery
P28 P27 P29 P08 P14 P04 P06
Infrastructure
as code
P22
Continuous
integration
P07
Automated testing
P18
Automated feedback
P05
Continuous
testing
P17
Automated
deployment
P11
P10
P10
P10 P12
P12
P12
P12
P09
P24
P13
P01
P01
Integrated
configurationmanagement
P15
P03 P02
P16
r23
Figure 7. Value-based DevOps practices dependency network
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
20 JABBARI ET AL.
r10
r3
P18
r18
r21
r22
r9
CB03
CB08
CB15
CB28
CB30
CB32
Integrated deployment
planning
Automated feedback
Continuous application
performance modeling
Continuous testing Integrated configuration
management
Cooperative application
configuration
DevOps Practices Relations DevOps Benefits
P05
P17 P15
P09
P24
Automated testing
P13
Figure 8. Expected benefits of the blue sequences of DevOps practices
For the practices, 15 studies of the selected ones have presented or discussed any activity in the
context of DevOps [22,24,25,26,27,28,30,32,40,42,49,50,55,56,61], from which we
identified the 29 practices. These studies mostly had a general discussion and do not investigate the
practices in detail. Some studies only mention a number of practices without any specific description
or definition as DevOps practices. They do not explain the practices, the context of the application,
required activities, etc. In fact, a majority of the practices have been presented by names only or at
best with a brief description (see Appendix A, Table VIII).
The immature literature on DevOps in particular regarding its principles and practices makes
it hard and ambiguous to understand the content of the identified DevOps practices. A majority
of discussions about DevOps adoption generally focus on high-level phases/activities such as
planning, integration, test, configuration, deployment, monitoring and feedback and mostly talk
about eliminating the gap between development and operations as the main goal of adopting
DevOps.
Challenges. In total, we identified 41 challenges for the DevOps adoption (see Table VI) that
might help us to better understand the state of DevOps adoption. How every actual context, in which
a challenge has arisen is missing that leads to an immature status of DevOps in the peer-reviewed
literature. Most challenges have been reported without detailed characteristics and discussions. For
example, C02 has mentioned a lack of management structure as a DevOps challenge but it has not
specified what structure means in this context. Also, the complexity of adopting DevOps is reported
in terms of deployment, configuration, integration, etc. but the corresponding conditions have not
been reported. Hence, further investigations are needed to specify DevOps challenges and identify
the most critical ones.
Despite a lack of mature studies, the identified challenges can indicate some major difficulties to
adopt DevOps. A lack of infrastructure for automation, as a central principle for DevOps, has been
highlighted as a key barrier. Moreover, a shortage of sufficient solutions to handle the difficulties of
communication and collaboration between development and operations is reported by a number
of studies (C02, C04, C05, C06 and C07). Although, due to the absence of investigations and
studies on the DevOps challenges in terms of specific DevOps practices, continuous deployment
and continuous delivery has been reported (C13, C14, C21 and C32). Therefore, there is a need
for further studies on investigating DevOps challenges that will be specifically related to DevOps
practices.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 21
Table VII. Type of studies discussed DevOps principle and practices
Study Type of study Research method
[8] Conceptual framework Prototyping
[10] Used in practice Industrial case study
[18] Used in practice not specified
Principle [22] Opinion about something not specified
[40] Novel solution not specified
[44] Novel solution not specified
[56] Opinion about something Industrial case study
[22] Opinion about something not specified
[24] Opinion about something not specified
[25] Opinion about something not specified
[26] Novel solution (opinion about something) not specified
[27] Opinion about something not specified
[28] Novel solution (authors’ experience) Prototyping
[30] Novel solution (authors’ experience) not specified
Practice [32] Authors’ experience Laboratory experience
[40] Novel solution not specified
[42] Opinion about something not specified
[49] Opinion about something not specified
[50] Novel solution (authors’ experience) not specified
[55] Used in practice Industrial case study
[56] Opinion about something Industrial case study
[61] Novel solution not specified
6.2. Methodological aspects
Moreover, the discussions provided in the selected studies are not consistent and the quality of
studies cannot support the proposed arguments. We assessed the quality of the selected studies used
for identifying DevOps principles and practices by extracting data regarding ‘type of study’ and
‘research method’ in accordance with a number of conditions presented by Petersen et al. [48] (see
Table III). Use of an empirical method has been reported by only two papers, which points to a clear
need for further empirical research on the topic. Table VII shows the type of studies, from which we
identified DevOps principles and practices.
A lack of well-defined DevOps practices in the selected studies prevents from specifying the
actual DevOps practices including prerequisites to apply, the details of e.g., required activities and
implications. As shown in Table VII, 11 papers (for DevOps practice) have not specified the research
method.
DevOps benefits. As shown in Section 4.3, the majority of DevOps benefits presented in the
selected studies are claimed ones and only a few benefits are demonstrated, which can indicate a
lack of empirical evidence on the topic. Also, only a few studies have investigated the link between
DevOps practices and expected benefits (see Table X).
DevOps Benefits Dependency Network. We derived a benefits dependency network for DevOps to
visualize our findings. The information used as inputs for the network is the extracted data related
to the relations among practices and their links to benefits.
The network might be used as a driver for further research studies on the application of DevOps
in terms of practical investigating the dependencies among DevOps practices and expected benefits.
Given the link between benefits and practices makes the desired benefits explicit from needed
practices, as the main aim of the dependency network. For example, one common challenge
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
22 JABBARI ET AL.
during planning a change is to decide on the order of implementing practices. Given the explicit
documentation of dependency links can address the mentioned challenge.
Although lacking clear definitions for DevOps practices and existing only a few demonstrated
benefits, the network presented in this paper is a starting point. We believe that this network is
a good way forward as it (a) is goal and benefit driven and (b) clearly indicates the progression
towards the benefits that should be achieved. Therefore, it may serve as a kind-of maturity model
for DevOps. The network would have to be continuously evaluated and updated with more evidence
becoming available as future works.
6.3. Comparison with related work
Nicolau de Franc¸a et al. [19] also conducted a literature study with a focus similar to our study,
although their study contains only a few peer-reviewed and more gray literature like websites,
technical reports and industrial sources. Overall, in comparison to Nicolau de Franc¸a et al. [19],
our review has the following distinctive features:
•We focused solely on the peer-reviewed literature and identified significantly more published
studies.
•Identified principles, practices and benefits and the relations among (a) DevOps practices and
(b) their links to DevOps benefits.
•We also synthesized the results in the form of a Benefits Dependency Network.
In terms of the approach used for conducting the review, we conducted a systematic literature
review with 40 primary studies. Nicolau de Franc¸a et al. on the other hand, conducted a multivocal
literature review with 43 studies (more than 69% of which are from gray literature). The brief
descriptions of practices found in the literature are insufficient to allow a meaning comparison of
the results with Nicolau de Franc¸a et al.’s review. For example, the details are insufficient to establish
whether we are refering to the same concepts. Therefore, we could only map the DevOps practices
identified in our study to the practices reported by Nicolau de Franc¸a et al. based on the naming
like P03 (Continuous monitoring), P07 (Continuous integration), P12 (Continuous delivery), P15
(Configuration management) and P18 (Automated testing).
Similarly, we also only made a simple comparison for the principles identified in the two studies.
Nicolau de Franc¸a et al. [19] have identified six principles, which overlap our findings. Three of
their six identified principles have identical names and meanings as the principles identified in this
review : “Principle 2 (Automation)” [19] to Pr2 (Automation), “Principle 5 (Sharing)” [19] to Pr1
(Sharing knowledge) and “Principle 6 (Measurement)” [19] to Pr5 (Measurement).
Furthermore, we can see some congruence between the remaining three principles: “Principle 1
(Social aspects)” [19] to Pr1 and Pr2, “Principle 3 (Quality assurance)” [19] to Pr6 and “Principle 4
(Leanness)” [19] to Pr4. However, the definitions of the principles aligned well based on the same
peer-reviewed references [10,21].
Given that both studies have been conducted independently, the alignment fo results adds
confidence in the validity of the findings. Also, this indicates that with regard to the principles
of DevOps, there is a similar view in both the peer-reviewed and gray literature.
7. THREATS TO VALIDITY
7.1. Generalizability (ability to generalize to different contexts)
This study has focused on a limited number of peer-reviewed studies available in six electronic
databases. However, the research on DevOps is not static and continues to evolve over time. In other
words, new results and materials related to the aims and objectives of this study may have been
potentially added to the body of knowledge. This is a threat to the validity of our study which is
beyond our control during the execution and reporting of this study. We have presented aggregated
characteristics of DevOps in terms of principles, practices, challenges, and benefits that may be
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 23
further extended as the research in this area evolves. The dependency network presented in this
paper potentially forms the basis for adding further evidence as it becomes available.
7.2. Limitations of the search (inability to identify what we intend to find from the search results)
To increase the confidence in the search process of this study, we expanded the search activity
over a broader number of available electronic databases (6 of the most well-known databases in the
field) and defined the search string as general as possible without any limitations and interventions
(i.e., “DevOps”) in order to avoid the risk of missing any potentially related paper to our research
work. Also, the first two authors concurrently conducted the search twice referred to the test-retest
procedure [38]. Moreover, to assure not missing any important studies we also conducted the search
on the full-text articles. It should be mentioned that we did not perform snowball sampling as it is
particularly useful if the keywords are not well-established for a topic. However, we used the term
DevOps and searched in the full-text of the articles. Therefore, we consider that not supplementing
the keyword-based search is not as a major threat to coverage of our study. Snowball sampling may
have helped us to identify research not covered in the databases used in our search. The threat is
however minimal since we used six databases that sufficiently cover major venues relevant for our
research. The reason why we also included index databases such as Scopus was to mitigate the risk
of missing any paper related to DevOps. In addition, as the search has been limited to the peer-
reviewed literature, we did not consider gray literature including web, book, technical report, etc.
Therefore, there is a risk of missing new results and materials related to our topic of interest, which
provided by the gray literature. This risk is further amplified since DevOps is still largely driven by
industry practitioners. To mitigate this risk, we have compared our findings with Nicolau de Franc¸a
et al.’s study [19], which included gray literature as well (see details in Section 6.3).
7.3. Probability of excluding relevant papers during data extraction
Initial inclusion/exclusion criteria have been applied to mitigate the risk of missing relevant studies,
thereafter; we ran two rounds of pilot studies to evaluate and validate the data extraction form.
Firstly, the first two authors have extracted data separately from five studies, randomly selected
from 118 papers, to evaluate the form. Based upon this evaluation the data extraction form was
revised. At the second round, all four authors have been involved to extract data individually from
two new randomly selected articles in common. After consensus building, the studies have been
distributed among the authors to extract in accordance with the contributions. In fact, full-text was
read for all 118 selected studies to mitigate the threat of excluding any relevant paper.
7.4. Potential for missing relevant information from included papers
To mitigate this threat, we followed the recommendation by Kitchenham and Brereton [37] to
include publisher databases and index databases in our search procedures. Also, through conducting
two separate rounds of pilot studies, as a test-retest strategy, we considered this potential risk by
evaluating, redesigning and improving the data extraction form and consensus building among the
four authors.
7.5. Risk of bias towards specific viewpoints due to nascence of the field
As presented in this study, a number of findings (i.e., DevOps principles, practices, benefits, and
challenges) are from individual papers that can indicate a lack of consensus in the community.
Therefore, the lack of literature and maturity on the topic can increase a risk of a bias towards
specific viewpoints in the community. However, it points to the need for further research work on
the topic, in particular, the potential gaps mentioned in this study. Furthermore, we have investigated
the type of study and the research method used in the selected studies, which helped us to assess the
reliability of the extracted data (see details in Section 6.1 and 6.2). This transparency in reporting
helps to identify future research opportunities to overcome the limitation of any biased results in the
current study.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
24 JABBARI ET AL.
7.6. Reliability of the study
One main aspect of reliability of secondary studies is repeatability of the study [62]. To this end,
we have taken the following measures to ensure the repeatability of our research work through
systematically documenting the search action by reporting the search strings, date of search, actual
search results (are available on request) per database, list of all the included papers, documenting
the process of data analysis, using established framework like SWEBOK and SVM for analysis, and
describing our research methods, data analysis and synthesis, interpretation, etc.
7.7. Objectivity (ability to objectively describe observations)
As mentioned above, multiple threats are possible during the search, study selection, data extraction,
and analysis. To mitigate these threats to validity, multiple authors have been involved in all steps of
the research work. The first two authors have been involved initially during search, study selection,
piloted the data extraction and reviewed the data analysis. Afterwards, all four authors were involved
to run another piloting to validate the data extraction form and establish common perception.
8. CONCLUSION
In this paper, we presented a systematic literature review focusing on characterizing DevOps through
identifying and specifying DevOps principles, practices, challenges and benefits. We also proposed
a holistic view of the findings by synthesizing the results and presenting the DevOps practices
dependency network based on the extracted data. Moreover, we investigated the relations and
dependencies between DevOps practices and their links to the corresponding benefits. To this end,
three research questions have been formulated.
RQ1: What are the principles and practices associated with DevOps in the peer-reviewed
literature?
To address the RQ1, we have extracted those principles and practices, which are explicitly linked
to DevOps from the literature. We specified 6 DevOps principles and 29 practices proposed for
DevOps. The principles focus on sharing knowledge and shared responsibility, which directly target
the relations between development and operations and the need for a culture change. They also
consider automation and continuity as the fundamentals of DevOps. More specific technical aspects
like measurement and composability have also been covered by the DevOps principles. We also
found that only a subset of papers described the practices and given the novelty of the concept more
detailed descriptions are needed.
RQ2. What are the challenges explicitly associated with DevOps in the peer-reviewed literature?
To complete our study on DevOps, we identified the DevOps challenges from the peer-
reviewed literature. 49 challenges have been identified and classified in accordance with SWEBOK
knowledge areas [3] to provide a better understanding of what areas are critical in adopting DevOps.
We concluded that the majority of identified DevOps challenges belong to the areas of tools and
methods, software quality, and software engineering management.
RQ3. What are the benefits (claimed and demonstrated) explicitly associated with DevOps in the
peer-reviewed literature?
We identified and specified the proposed benefits of DevOps and utilized the SVM [34] to
better illustrate them. We also differentiated between the claimed benefits and demonstrated ones
according to the evidence presented by the studies. In total, 40 claimed and 9 demonstrated DevOps
benefits have been specified. In addition, the link between practices to benefits has also been
investigated and the benefits dependency network proposed to indicate a consolidated view for
DevOps characteristics.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 25
The benefits dependency network is based on the current knowledge presented in the selected
studies. In fact, it forms the baseline for continuous updates as new evidence becomes available.
The advantage of the DevOps benefits dependency network is (a) its focus on benefits-driven
practice selection and (b) making the dependencies of practices explicit which may aid in the
planning of adoption strategies. Thus, it may function as a maturity model for DevOps that should
be continuously developed in future work.
Last but not least, to our knowledge, DevOps is perhaps immature in the peer-reviewed literature,
although the need of bridging the gap between development (Dev) and operations (Ops) is explicitly
highlighted by the literature. In the selected peer-reviewed studies, it is mostly suggestions and
claims, lack of empirical validation that shows the nascence of the field that points to the need for
more scientific evaluations (e.g., lack of studies on demonstrated benefits. As shown in Table IV,
only five studies discussed demonstrated benefits of DevOps).
Therefore, the following directions realized from this study as potential gaps for further research:
empirically study DevOps practices based on specific needs and expected benefits, investigate
DevOps challenges in terms of type of practice and its context, evaluate the DevOps benefits
dependency network proposed in this study and provide consistent DevOps characteristics in terms
of principles, practices, challenges and benefits through empirical research methods.
ACKNOWLEDGEMENTS
The work of Kai Petersen and Nauman bin Ali has been supported by ELLIIT, a Strategic Area
within IT and Mobile Communications, funded by the Swedish Government and from EASE, the
Industrial Excellence Centre for Embedded Applications Software Engineering.
REFERENCES
1. Principles behind the agile manifesto. Retrieved from http://agilemanifesto.org/principles.html, 2017, October 23.
2. A view of the devops cycle. Retrieved from http://readwrite.com/2014/01/01/three-reasons-your-startup-needs-
devops-or-else, 2017, October 31.
3. Alain Abran, James W Moore, Pierre Bourque, Robert Dupuis, and L Tripp. Guide to the software engineering
body of knowledge: 2004 version. IEEE Computer Society, 1, 2004.
4. Nauman Bin Ali. Is effectiveness sufficient to choose an intervention?: Considering resource use in empirical
software engineering. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software
Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8-9, 2016, pages 54:1–54:6, 2016.
5. Nauman Bin Ali and Kai Petersen. Evaluating strategies for study selection in systematic literature studies. In
Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement,
ESEM ’14, 2014.
6. Ayb ¨
uke Aurum, Claes Wohlin, and Andrew Porter. Aligning software project decisions: a case study. International
Journal of Software Engineering and Knowledge Engineering, 16(06):795–818, 2006.
7. Paula Austel, Han Chen, Parijat Dube, Thomas A. Mikalsen, Isabelle Rouvellou, Upendra Sharma, Ignacio
Silva-Lepe, Revathi Subramanian, Wei Tan, and Yandong Wang. A paas for composite analytics solutions. In
Proceedings of the 12th ACM International Conference on Computing Frontiers, CF’15, Ischia, Italy, May 18-21,
2015, pages 51:1–51:8, 2015.
8. Paula Austel, Han Chen, Thomas A. Mikalsen, Isabelle Rouvellou, Upendra Sharma, Ignacio Silva-Lepe, and
Revathi Subramanian. Continuous delivery of composite solutions: A case for collaborative software defined paas
environments. In Proceedings of the 2nd International Workshop on Software-Defined Ecosystems, BigSystem
2015, Portland, Oregon, USA, June 16, 2015, pages 3–6, 2015.
9. Amgad Ali Badewi, Essam Shehab, and Joe Peppard. Benefit realisation modelling for erp systems using system
dynamics. In Advances in Manufacturing Technology XXVII-Proceedings of the 11th International Conference on
Manufacturing Research (ICMR2013), pages 225–235, 2013.
10. Soon K. Bang, Sam Chung, Young Choh, and Marc Dupuis. A grounded theory analysis of modern web
applications: Knowledge, skills, and abilities for devops. In Proceedings of the 2Nd Annual Conference on Research
in Information Technology, RIIT ’13, 2013.
11. Sebastian Barney, Kai Petersen, Mikael Svahnberg, Ayb¨
uke Aurum, and Hamish T. Barney. Software quality trade-
offs: A systematic map. Information & Software Technology, 54(7):651–662, 2012.
12. Stefan Biffl, Aybuke Aurum, Barry Boehm, Hakan Erdogmus, and Paul Gr ¨
unbacher. Value-based software
engineering. 2006.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
26 JABBARI ET AL.
13. M. Boschetti, Vito Baglio, Pietro Ruiu, and Olivier Terzo. A cloud automation platform for flexibility in
applications and resources provisioning. In Ninth International Conference on Complex, Intelligent, and Software
Intensive Systems, CISIS 2015, Santa Catarina, Brazil, July 8-10, 2015, pages 204–208, 2015.
14. Dario Bruneo, Thomas Fritz, Sharon Keidar-Barner, Philipp Leitner, Francesco Longo, Clarissa Cassales
Marquezan, Andreas Metzger, Klaus Pohl, Antonio Puliafito, Danny Raz, Andreas Roth, Eliot Salant, Itai Segall,
Massimo Villari, Yaron Wolfsthal, and Chris Woods. Cloudwave: Where adaptive cloud management meets devops.
In IEEE Symposium on Computers and Communications, ISCC 2014, Funchal, Madeira, Portugal, June 23-26,
2014, pages 1–6, 2014.
15. Lianping Chen. Continuous delivery: Huge benefits, but challenges too. IEEE Software, 32(2):50–54, 2015.
16. Constantine Aaron Cois, Joseph Yankel, and Anne Connell. Modern devops: Optimizing software development
through effective system interactions. In 2014 IEEE International Professional Communication Conference, IPCC
2014, Pittsburgh, PA, USA, October 13-15, 2014, pages 1–7, 2014.
17. Daniela Cruzes and Tore Dyb˚
a. Research synthesis in software engineering: A tertiary study. Information &
Software Technology, 53(5):440–455, 2011.
18. Maximilien de Bayser, Leonardo G. Azevedo, and Renato F. G. Cerqueira. Researchops: The case for devops
in scientific applications. In IFIP/IEEE International Symposium on Integrated Network Management, IM 2015,
Ottawa, ON, Canada, 11-15 May, 2015, pages 1398–1404, 2015.
19. Breno Bernard Nicolau de Franc¸a, Helvio Jeronimo Jr., and Guilherme Horta Travassos. Characterizing devops by
hearing multiple voices. In Proceedings of the 30th Brazilian Symposium on Software Engineering, SBES 2016,
Maring´
a, Brazil, September 19 - 23, 2016, pages 53–62, 2016.
20. Andrej Dyck, Ralf Penners, and Horst Lichter. Towards definitions for release engineering and devops. In 3rd
IEEE/ACM International Workshop on Release Engineering, RELENG 2015, Florence, Italy, May 19, 2015, page 3,
2015.
21. Floris Erich, Chintan Amrit, and Maya Daneva. Cooperation between information system development and
operations: a literature review. In 2014 ACM-IEEE International Symposium on Empirical Software Engineering
and Measurement, ESEM ’14, Torino, Italy, September 18-19, 2014, page 69:1, 2014.
22. B.S. Farroha and D.L. Farroha. A framework for managing mission needs, compliance, and trust in the devops
environment. In Military Communications Conference (MILCOM), 2014 IEEE, 2014.
23. Nicolas Ferry, Franck Chauvel, Hui Song, and Arnor Solberg. Continous deployment of multi-cloud systems.
In Proceedings of the 1st International Workshop on Quality-Aware DevOps, QUDOS 2015, Bergamo, Italy,
September 1, 2015, pages 27–28.
24. Brian Fitzgerald and Klaas-Jan Stol. Continuous software engineering and beyond: trends and challenges. In 1st
International Workshop on Rapid Continuous Software Engineering, RCoSE 2014, Hyderabad, India, June 3, 2014,
pages 1–9, 2014.
25. Brian Fitzgerald and Klaas-Jan Stol. Continuous software engineering: A roadmap and agenda. Journal of Systems
and Software, 2015.
26. K. Gohil, N. Alapati, and S. Joglekar. Towards behavior driven operations (bdops). In Advances in Recent
Technologies in Communication and Computing (ARTCom 2011), 3rd International Conference on, 2011.
27. Wolfgang Gottesheim. Challenges, benefits and best practices of performance focused devops. In Proceedings of
the 4th International Workshop on Large-Scale Testing, LT’15, Austin, TX, USA, February 1, 2015, page 3, 2015.
28. Shigeru Hosono. A devops framework to shorten delivery time for cloud applications. IJCSE, 7(4):329–344, 2012.
29. Shigeru Hosono, Jiafu He, Xuemei Liu, Lin Li, He Huang, and Shuichi Yoshino. Fast development platforms and
methods for cloud applications. In 2011 IEEE Asia-Pacific Services Computing Conference, APSCC 2011, Jeju,
Korea (South), December 12-15, 2011, pages 94–101, 2011.
30. S.W. Hussaini. Strengthening harmonization of development (dev) and operations (ops) silos in it environment
through systems approach. In Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference
on, 2014.
31. Ramtin Jabbari, Nauman bin Ali, Kai Petersen, and Binish Tanveer. What is devops? a systematic mapping
study on definitions and practices. XP 2016, First International Workshop on Emerging Trends in DevOps and
Infrastructure, 2016.
32. K. R. Jayaram. Towards explicitly elastic programming frameworks. In 37th IEEE/ACM International Conference
on Software Engineering, ICSE 2015, Florence, Italy, May 16-24, 2015, Volume 2, pages 619–622, 2015.
33. Staffs Keele. Guidelines for performing systematic literature reviews in software engineering. In Technical report,
Ver. 2.3 EBSE Technical Report. EBSE. 2007.
34. Mahvish Khurum, Tony Gorschek, and Magnus Wilson. The software value mapan exhaustive collection of value
aspects for the development of software intensive products. Journal of Software: Evolution and Process, 25(7):711–
741, 2013.
35. Terhi Kilamo, Marko Lepp¨
anen, and Tommi Mikkonen. The social developer: Now, then, and tomorrow. In
Proceedings of the 7th International Workshop on Social Software Engineering, SSE 2015, 2015.
36. Juhoon Kim, Catalin Meirosu, Ioanna Papafili, Rebecca Steinert, Sachin Sharma, Fritz-Joachim Westphal, Mario
Kind, Apoorv Shukla, Felician N´
emeth, and Antonio Manzalini. Service provider devops for large scale modern
network services. In IFIP/IEEE International Symposium on Integrated Network Management, IM 2015, Ottawa,
ON, Canada, 11-15 May, 2015, pages 1391–1397, 2015.
37. Barbara Kitchenham and Pearl Brereton. A systematic review of systematic review process research in software
engineering. Information & Software Technology, 55(12):2049–2075, 2013.
38. Barbara A. Kitchenham. What’s up with software metrics? - A preliminary mapping study. Journal of Systems and
Software, 83(1):37–51, 2010.
39. Stephan Krusche and Lukas Alperowitz. Introduction of continuous delivery in multi-customer project courses.
In 36th International Conference on Software Engineering, ICSE ’14, Companion Proceedings, Hyderabad, India,
May 31 - June 07, 2014, pages 335–343, 2014.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 27
40. Matthew A. McCarthy, Lorraine M. Herger, Shakil M. Khan, and Brian M. Belgodere. Composable devops:
Automated ontology based devops maturity analysis. In 2015 IEEE International Conference on Services
Computing, SCC 2015, New York City, NY, USA, June 27 - July 2, 2015, pages 600–607, 2015.
41. Marco Miglierina. Application deployment and management in the cloud. In 16th International Symposium on
Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2014, Timisoara, Romania, September 22-25,
2014, pages 422–428, 2014.
42. Shaun Murphy, Scott Gallant, Chris Gaughan, and Manny Diego. U.S. army modeling and simulation executable
architecture deployment cloud virtualization strategy. In 12th IEEE/ACM International Symposium on Cluster,
Cloud and Grid Computing, CCGrid 2012, Ottawa, Canada, May 13-16, 2012, pages 880–885, 2012.
43. Felician N´
emeth, Rebecca Steinert, Per Kreuger, and Pontus Sk¨
oldstr¨
om. Roles of devops tools in an automated,
dynamic service creation architecture. In IFIP/IEEE International Symposium on Integrated Network Management,
IM 2015, Ottawa, ON, Canada, 11-15 May, 2015, pages 1153–1154, 2015.
44. Marta Olszewska and Marina A. Wald´
en. Devops meets formal modelling in high-criticality complex systems.
In Proceedings of the 1st International Workshop on Quality-Aware DevOps, QUDOS 2015, Bergamo, Italy,
September 1, 2015, pages 7–12, 2015.
45. Sun Park, ByungRae Cha, and JongWon Kim. Preparing and inter-connecting hyper-converged smartx boxes for
iot-cloud testbed. In 29th IEEE International Conference on Advanced Information Networking and Applications,
AINA 2015, Gwangju, South Korea, March 24-27, 2015, pages 695–697, 2015.
46. Joe Peppard, John Ward, and Elizabeth Daniel. Managing the realization of business benefits from it investments.
MIS Quarterly Executive, 6(1), 2007.
47. Juan F. P´
erez, Weikun Wang, and Giuliano Casale. Towards a devops approach for software quality engineering. In
Proceedings of the 2015 Workshop on Challenges in Performance Methods for Software Development, WOSP-C’15,
Austin, TX, USA, January 31, 2015, pages 5–10, 2015.
48. Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. Guidelines for conducting systematic mapping studies in
software engineering: An update. Information and Software Technology, 64:1–18, 2015.
49. James Roche. Adopting devops practices in quality assurance. Commun. ACM, 56(11):38–43, 2013.
50. Joel Scheuner, J¨
urgen Cito, Philipp Leitner, and Harald C. Gall. Cloud workbench: Benchmarking iaas providers
based on infrastructure-as-code. In Proceedings of the 24th International Conference on World Wide Web
Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume, pages 239–242, 2015.
51. Itai Segall and Rachel Tzoref-Brill. Feedback-driven combinatorial test design and execution. In Proceedings of
the 8th ACM International Systems and Storage Conference, SYSTOR 2015, Haifa, Israel, May 26-28, 2015, pages
12:1–12:6, 2015.
52. Jens Smeds, Kristian Nybom, and Ivan Porres. Devops: a definition and perceived adoption impediments. In Agile
Processes, in Software Engineering, and Extreme Programming, pages 166–177. 2015.
53. Mark Stillwell and Jos´
e Gabriel F. Coutinho. A devops approach to integration of software components in an
EU research project. In Proceedings of the 1st International Workshop on Quality-Aware DevOps, QUDOS 2015,
Bergamo, Italy, September 1, 2015, pages 1–6, 2015.
54. William Toll. Top devops conferences in 2016: 49 events for devops pros. ProfitBricks Blog,
https://blog.profitbricks.com/, 2015.
55. Tatiana Ustinova and Pooyan Jamshidi. Modelling multi-tier enterprise applications behaviour with design of
experiments technique. In Proceedings of the 1st International Workshop on Quality-Aware DevOps, QUDOS
2015, Bergamo, Italy, September 1, 2015, pages 13–18, 2015.
56. M. Virmani. Understanding devops bridging the gap from continuous integration to continuous delivery. In
Innovative Computing Technology (INTECH), 2015 Fifth International Conference on, 2015.
57. Johannes Wettinger. Streamlining devops automation for cloud applications using {TOSCA}as standardized
metamodel. Future Generation Computer Systems, 2015.
58. Johannes Wettinger, Vasilios Andrikopoulos, and Frank Leymann. Automated capturing and systematic usage of
devops knowledge for cloud applications. In 2015 IEEE International Conference on Cloud Engineering, IC2E
2015, Tempe, AZ, USA, March 9-13, 2015, pages 60–65, 2015.
59. Johannes Wettinger, Michael Behrendt, Tobias Binz, Uwe Breitenb¨
ucher, Gerd Breiter, Frank Leymann, Simon
Moser, Isabell Schwertle, and Thomas Spatzier. Integrating configuration management with model-driven cloud
management based on TOSCA. In CLOSER 2013 - Proceedings of the 3rd International Conference on Cloud
Computing and Services Science, Aachen, Germany, 8-10 May, 2013, pages 437–446, 2013.
60. Johannes Wettinger, Uwe Breitenb¨
ucher, and Frank Leymann. Compensation-based vs. convergent deployment
automation for services operated in the cloud. In Service-Oriented Computing - 12th International Conference,
ICSOC 2014, Paris, France, November 3-6, 2014. Proceedings, pages 336–350, 2014.
61. Johannes Wettinger, Uwe Breitenb¨
ucher, and Frank Leymann. Devopslang - bridging the gap between development
and operations. In Service-Oriented and Cloud Computing - Third European Conference, ESOCC 2014,
Manchester, UK, September 2-4, 2014. Proceedings, pages 108–122, 2014.
62. Claes Wohlin, Per Runeson, Paulo Anselmo da Mota Silveira Neto, Emelie Engstr¨
om, Ivan do Carmo Machado, and
Eduardo Santana de Almeida. On the reliability of mapping studies in software engineering. Journal of Systems
and Software, 86(10):2594–2610, 2013.
63. Liming Zhu, Donna Xu, An Binh Tran, Xiwei Xu, Len Bass, Ingo Weber, and Srini Dwarakanathan. Achieving
reliable high-frequency releases in cloud environments. IEEE Software, 32(2):73–80, 2015.
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
28 JABBARI ET AL.
A. DEVOPS PRACTICES DESCRIPTION
Table VIII. DevOps practices definition
Practice Definition Ref.
P01: Continuous planning “Continuous planning may be defined as a holistic endeavor involving multiple stakeholders from
business and software functions whereby plans are dynamic open-ended artifacts that evolve in response
to changes in the business environment, and thus involve a tighter integration between planning and
execution.”
[24]
P02: Continuous feedback “The way how teams work together and share the responsibility for the end users of their application.
It not only encourages the adoption of agile practices in operations work, it also allows developers to
learn from real world Ops experiences and starts a mutual exchange that breaks down the walls between
teams.”
[27]
P03: Continuous monitor-
ing
“With the capability to test early, through adopting continuous planning, integration, deployment and
testing, and on a production like system there is an opportunity to observe various quality parameters
throughout and hence ability to react to any surprises in timely manner.”
[56]
P04: Automated monitor-
ing
“Ops and Test Teams usually have a good understanding of performance, and they need to educate
developers on its importance in large-scale environments under heavy load. Providing automated
mechanisms to monitoring performance in all environments, from continuous integration (CI) and test
environments to the actual production deployment allows the shared language of performance to be
spoken.”
[27]
P05: Automated feedback “One way to achieve DevOps is continuous application performance (P25) modelling and prediction
combined with “automated feedback” of the models to the developer and their update via continuous
testing (P18).”
[55]
P06: Automated dashboard not defined in the paper! [22]
P07: Continuous integra-
tion
“Continuous integration basically refers to integrate early, don??????t keep changes localized to your
workspace for long, instead share your changes with team and validate how code behaves continuously.
Not only share within component teams, but integrate beyond component boundaries, at product
integration level.”
[56]
P08: Prototyping applica-
tion
“The goal of this step is to implement and build the application, and then test its functionalities. The
application implementers start implementation using the scalable prototyping. The scalable prototyping
initially displays the application logics, which are transferred from the logics defined in the resource
combination design.”
[28]
P09: Integrated deployment
planning
not defined in the paper! [22]
P10: Continuous deploy-
ment
“This is heart of DevOps and forms the critical piece of overall software delivery optimization. DevOps
principles recommend to automate the deployment and provisioning of hardware and various cloud
providers play a crucial role in this field.”
[56]
P11: Automated deploy-
ment
“A new domain-specific language called DevOpSlang in conjunction with a methodology to enable the
implementation of DevOps. The language serves as an efficient means of collaboration and provides the
foundation to automate deployment and operations of an application.”
[61]
P12: Continuous delivery “The main promise of DevOps is to enable continuous delivery of software in order to enable fast and
frequent releases. This enables quickresponses to changing requirements of customers and thus may be a
critical competitive advantage. Efficient collaboration and automation arethe key enablers to implement
continuous delivery and thus to react to changing customer requirements quickly.”
[61]
P13: Cooperative applica-
tion configuration
“Application configuration can be very complex and may need to be bonded with other configured
applications. The DevOps process captures the configurable elements of an application and exposes
them through a simple properties specification.”
[42]
P14: Staging application “The goal is to deploy the application, tune it and specify monitoring points for the production stage.
The application implementers can archive the application files into a packaged one with the scalable
prototyping and deploy it to the VMs which are created in prototyping application.”
[28]
P15: Integrated configura-
tion management
not defined in the paper! [22]
P16: Change management not defined in the paper! [22]
P17: Continuous testing “One way to achieve DevOps is continuous application performance (P25) modelling and prediction
combined with automated feedback (P05) of the models to the developer and their update via
“continuous testing”.”
[55]
P18: Automated testing not defined in the paper! [22]
P19: Process standardiza-
tion
not defined in the paper! [49]
P20: Production support not defined in the paper! [22]
P21: Use of data to guide
QA
not defined in the paper! [49]
P22: Infrastructure as code “IaC was introduced by the current DevOps trend and captures the complete provisioning and
configuration of various middleware components, most importantly IaaS VMs, operating systems, and
standard software, in provisioning code.”
[50]
P23: Key performance met-
rics
“(i.e., measure key performance metrics in CI, Test and Ops) With performance aspects being covered
in earlier testing stages, performance engineers on testing teams get time to focus on large-scale load
tests in production-like environments.”
[27]
P24: Continuous app per-
formance modeling
“One way to achieve DevOps combined with automated feedback (P05) of the models to the developer
and their update via continuous testing (P18).”
[55]
P25: DevOps maturity
ontology
“DevOps Maturity Ontology serves as a meta model, defining the concepts and relations related to the
DevOps Maturity evaluation.”
[40]
P26: Elasticity practice “Elasticity is the ability of a distributed application to increase or decrease its use of computing
resources to maintain a desired level of performance in response to changing workloads.”
[32]
P27: Defining requirements “The goal is to determine objectives of the application identify functional and NFRs and break them
down to functions or activities to be delivered.”
[28]
P28: Stakeholder participa-
tion
not defined in the paper! [22]
P29: Designing architecture “The goal of this step is to design the system architecture; determine both business logics and system
resources on the cloud and build middleware and resources.”
[28]
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
ON THE VALUE OF DEVOPS 29
B. RELATIONS OF DEVOPS PRACTICES
Table IX. Relations definition of DevOps practices dependency network
Relation Path Ref. Supporting verbatim quotes from the reference(s)
r1 P11 to P10 [61]“One major precondition to implement continuous delivery of software is to automate
the whole deployment process (r2), in a repeatable way (r1).”
r2 P10 to P12 [61]“One major precondition to implement continuous delivery of software is to automate
the whole deployment process (r2), in a repeatable way (r1).”
r3 P18 to P17 [22,56] Prerequisite to continuous testing is automation of test-case execution.
r4 P17 to P10 [56]“This whole principle of continuous testing not only moves the testing process to
early in cycle but also allows the tests to be carried out on production like system
(complemented by continuous deployment).”
r5 P22 to P10 [56]“Continuous deployment is heart of DevOps and forms the critical piece of overall
software delivery optimization. DevOps approach proposes that entire infrastructure
provisioning should be maintained as code in source code repository, a concept being
called Infrastructure as a code (IAAC).”
r6 P22 to P12 [56]“Continuous delivery transformation in this paper is explained with a real life case
study that how infrastructure can be maintained just in form of code (IAAC).”
r7 P07 to P11 [56]“Further this stage of process optimization refers to achieving automation such that
as soon as developer delivers the change the build systems detects that (may even be
a scheduled event at end of day) and triggers a build carries out sanity tests and posts
the build to a repository. This has to be a repeatable continuous process all across the
development cycle.”
r8 P02 to P01 [56]“DevOps allows you to do continuous planning by always having a continuous channel
of feedback.”
r9 P05 to P24 [55]“DevOps is defined as a set of practices and principles bridging the gap between
application development and operation stages. One way to achieve this is continuous
application performance modelling and prediction combined with automated feedback
of the models to the developer (r9) and their update via continuous testing (r10).”
r10 P17 to P05 [55]“DevOps is defined as a set of practices and principles bridging the gap between
application development and operation stages. One way to achieve this is continuous
application performance modelling and prediction combined with automated feedback
of the models to the developer (r9) and their update via continuous testing (r10).”
r11 P27 to P29 [28]The relation has been defined based upon the whole discussion in the paper.
r12 P29 to P08 [28]The relation has been defined based upon the whole discussion in the paper.
r13 P08 to P14 [28]The relation has been defined based upon the whole discussion in the paper.
r14 P14 to P04 [28]The relation has been defined based upon the whole discussion in the paper.
r15 P03 to P02 authors
assumption
This relation has been defined based upon the prerequisite needed for feedback, which
is monitoring. The assumption inspired by [22]
r16 P15 to P03 [56]“With the capability to test early and on a production like system there is an opportunity
to observe various quality parameters throughout and hence ability to react to any
surprises in timely manner.”
r17 P07 to P18 [22]There is a flow from continuous integration to automated test indicated by Figure 5 in
this paper.”
r18 P17 to P15 [22]There is a flow from test to configuration indicated by Figure 5 in this paper.
r19 P28 to P27 authors
assumption
Stakeholder participation to define requirements.
r20 P04 to P06 authors
assumption
Automated dashboard contains a combination of automated monitoring and feedback.
r21 P18 to P09 [22]“This approach requires that the various testing needs to be automated and integrated
into each team’s product.”
r22 P15 to P13 authors
assumption
When a request is submitted, a set of information is included with that request in order
for application integrators and testers to realize the desired usage definition [42], that
shows configuration comes after application integration and test.
r23 P02 to P16 authors
assumption
Based on the DevOps flow [22,30], when change management is being done after
feedback and before planning.
r24 P16 to P01 authors
assumption
Based on the DevOps flow [22,30], when change management is being done after
feedback and before planning.
C. LINKING DEVOPS PRACTICES TO THE BENEFITS
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
30 JABBARI ET AL.
Table X. DevOps practices and identified benefits
Practice Benefit(s) Ref. Supporting verbatim quotes from the reference(s)
P01 CB08,
CB16,
DB09
[56,15]“Businesses plans have to be agile i.e., able to adjust quickly to the changing market conditions. DevOps
allows you to do that by always having a prioritized product backlog, continuous channel of feedback
with customers and ability to prioritize the product backlog all the time, directly taking business angle in
consideration. Continuous planning as a holistic endeavor involving multiple stakeholders from business
and software functions whereby plans are dynamic open-ended artifacts that evolve in response to changes
in the business environment, and thus involve a tighter integration between planning and execution.
In addition to iteration and release planning, product and portfolio planning activities would also be
conducted”.
P02 CB12 [56]“Enterprises can no longer afford to make a customer keep waiting for 6 months or 1 year for a release to
come and then solicit feedback from customer on how software behaves. Customers expect continuous
engagement so that they can provide continuous feedback. In order to meet the challenges of today,
enterprises need to be lean and agile in all the phases of software development life cycle.”
P03 CB08,
CB18,
CB32
[22,56]“With the capability to test early and on a production like system there is an opportunity to observe
various quality parameters throughout and hence ability to react to any surprises in timely manner.
Compliance and security are more critical in this era of big data and the 9 V??????s. To integrate
compliance and security throughout a deployed cloud application, automated testing for non-compliance
and policy needs to be leveraged. Additionally, the enterprise should pre-build, test and certify
services in a Services Catalog. To ensure compliance and security rules are adhered to, continuous
monitoring/alerting and automation to detect and mitigate critical issues must be implemented.”
P04 CB32 [22]“To ensure compliance and security rules are adhered to, continuous monitoring/alerting and automation
to detect and mitigate critical issues must be implemented. Finally, to track breaches in compliance, the
application will need to do the following: Automatic reporting for compliance violations Terminate access
when exceeding a threshold Initiate alarms when new policy is not accepted.”
P05 CB30 [55]“One way to achieve this (the bridging of the gap between app development and operation) is continuous
application performance modelling and prediction combined with automated feedback of the models to
the developer and their update via continuous testing.”
P07 CB03,
CB08,
CB15,
CB28,
CB30
[24]“While some form of automation is typical, the frequency of integration is also important in that it should
be regular enough to ensure quick feedback to developers. Finally, continuous integration failures are
important events which may have a number of ceremonies and highly visible artifacts to help ensure
that problems leading to these failures are prioritized for solution as quickly as possible by whoever is
deemed responsible. Continuous integration has increased in importance due to the benefits that have
been associated with it.”
P10 CB27 [56]“Deployment becomes a consistent repeatable process.”
P11 CB06 [56]“Significant cost saving as teams can share the pool of resources. As deployment is automated in almost
no time there is no need to keep resources reserved ?????? as soon as done with testing, team can release
the resource.”
P12 CB03,
CB08
[61]“The main promise of DevOps is to enable continuous delivery of software in order to enable fast and
frequent releases. This enables quick responses to changing requirements of customers and thus may be a
critical competitive advantage.”
P16 CB15,
CB30,
CB40,
DB09
[30]The relations have been defined based upon the whole discussion in the paper.
P18 CB32 [22]“Compliance and security are more critical in this era of big data and the 9 V??????s. To integrate
compliance and security throughout a deployed cloud application, automated testing for non-compliance
and policy needs to be leveraged. Additionally, the enterprise should pre-build, test and certify
services in a Services Catalog. To ensure compliance and security rules are adhered to, continuous
monitoring/alerting and automation to detect and mitigate critical issues must be implemented.”
P24 CB30 [55]“One way to achieve this (the bridging of the gap between app development and operation) is continuous
application performance modelling and prediction combined with automated feedback of the models to
the developer and their update via continuous testing.”
Copyright c
2018 John Wiley & Sons, Ltd. J. Softw. Evol. and Proc. (2018)
Prepared using smrauth.cls DOI: 10.1002/smr
