Content uploaded by Ruzanna Chitchyan
Author content
All content in this area was uploaded by Ruzanna Chitchyan on Jan 30, 2020
Content may be subject to copyright.
Do we really know what we are building? Raising awareness of potential
Sustainability Effects of Software Systems in Requirements Engineering
Leticia Duboc
La Salle
University Ramon Llull
Barcelona, Spain
l.duboc@salle.url.edu
Sedef Akinli Kocak
Vector Institute for Artificial Intelligence
Toronto, Canada
sedef.kocak@vectorinstitute.ai
Jari Porras
Lappeenranta University of Technology
Lappeenranta, Finland
jari.porras@lut.fi
Stefanie Betz
Hochschule Furtwangen University
Furtwangen, Germany
besi@hs-furtwangen.de
Ruzanna Chitchyan
University of Bristol
Bristol, UK
r.chitchyan@bristol.ac.uk
Norbert Seyff
FHNW
Switzerland
norbert.seyff@fhnw.ch
Birgit Penzenstadler
Chalmers University of Technology
Gothenburg, Sweden
birgitp@chalmers.se
Ola Leifler
Link¨
opings Universitet
Link¨
oping, Sweden
ola.leifler@liu.se
Colin C. Venters
University of Huddersfield
Huddersfield, UK
c.venters@hud.ac.uk
Abstract—Integrating novel software systems in our society,
economy, and environment can have far-reaching effects. As a
result, software systems should be designed in such a way
as to maintain or improve the sustainability of the socio-
technical system of their destination. However, a paradigm
shift is required to raise awareness of software professionals on
the potential sustainability effects of software systems. While
Requirements Engineering is considered the key to driving this
change, requirements engineers lack the knowledge, experience
and methodological support for doing so. This paper presents
a question-based framework for raising awareness of the po-
tential effects of software systems on sustainability, as the first
step towards enabling the required paradigm shift. A feasibility
study of the framework was carried out with two groups of
computer science students. The results of the study indicate
that the framework helps enable discussions about potential
effects that software systems could have on sustainability.
Keywords: sustainability, software, socio-technical systems, re-
quirements engineering
1. Introduction
Software underpins all aspects of societal life from com-
merce, communication, education, to energy, entertainment,
finance, governance and defence etc. As a cornerstone of
various socio-technical systems, the software is also a key
driver in their sustainability: their capacity to endure within
the current economic, environmental, social, technical,
and individual settings. These are commonly referred to
as the five dimensions of sustainability [24]. As a result,
the sustainability of a socio-technical system should become
a prime concern for the Software Engineering discipline. To
demonstrate how software can affect the sustainability of a
social system within which it is placed, let us look at the
example of the Airbnb platform in New York.
Motivating Example: Airbnb offers a peer-to-peer
short-term accommodation booking platform [1]. It allows
property owners to rent out homes or rooms to visitors.
Recent research [32] shows that in New York, homeowners
who rent their house frequently can earn 55% more than
the median long-term rental in the same neighbourhood
(affecting the individuals and the economy). As a conse-
quence, it is estimated that Airbnb has removed between
7,000 and 13,000 units of housing in New York from the
long-term rental market, leading to an increase of 1.4% in
the median long-term rent (affecting the society and the
economy). Research has also shown that in New York, 72%
of the population in the neighbourhoods at the highest risk
of Airbnb-induced gentrification are non-white, increasing
the race separations across the city (affecting the society).
The main reason for developing the Airbnb platform was
to provide an immediate benefit to travellers and property
owners through a direct software function. Over time, the
system offers additional benefits to owners (more earnings)
and travellers (cheaper accommodation, authentic local ex-
perience). It has also led to economic and social impacts
which were not considered at the time of its deployment.
Usually, these types of effects are only noticed in hindsight
and considering such possibilities ahead of software deploy-
ment is considered to be an expensive undertaking.
This paper advocates that as major drivers of change
within society, software systems must be designed to main-
tain the sustainability [3] of the wider socio-technical system
in which they are integrated. As such, it is the responsibility
of requirements engineers to analyse the sustainability im-
pacts of the software across all dimensions and inform the
software design process of the potential (un-)desired con-
sequences. This requires a paradigm shift in requirements
engineering practice, in which software and requirements
engineers take explicit responsibility for the technological
solutions that they introduce into society. For that, they can
draw upon the lessons of tackling wicked problems from
a holistic perspective of systems thinking [9] instead of
insisting on the narrow computational thinking mindset of
“solving a problem for a customer” [14].
Systems Requirements Engineering (RE) is the key to
this change of paradigm [3] and must start by raising
awareness of the relationships between a software system
and sustainability. As such, our key contributions are:
1) A question-based “Sustainability Awareness Frame-
work” for raising awareness of the impacts that a
software system could have upon sustainability.
2) An evaluation of the proposed question-based
framework based on two instances of its application
as part of a teaching curriculum.
The paper is organised as follows: Section 2 ex-
plains sustainability-related concepts; Section 3 describes
our framework; Section 4 outlines the study design; Section
5 presents its results; Section 6 summarises the related work;
finally Section 7 examines of the study.
2. Sustainability and Related Concepts
Modern society’s reliance on software systems has re-
sulted in the emergence of sustainability as a growing area of
interest in the field of software and requirements engineer-
ing [31]. In the context of this paper, sustainability is defined
as the capacity of a socio-technical system to endure [3].
Two closely related concepts are sustainable use and
sustainable development. Hilty and Aebischer [16] define
sustainable use of a system Swith regard to a function
Fand a time horizon T, which in essence means to “use
Sin a way that does not compromise its ability to fulfil
Ffor a period of T”. As a result, the concept of sus-
tainable use is relative as it is dependent on how S,F
and Tare defined in context. The Brundtland Commission
defined sustainable development [7] as ”meeting the needs
of the present without compromising the ability of future
generations to meet their own needs”. The word ‘need’ is
central to this definition and includes a dimension of time:
present and future, as well as acknowledging the concept of
changing stakeholder requirements. While it may appear that
sustainable development is a special case of sustainable use,
it has an element of distributive justice where “the essential
needs of the world’s poor are given overriding priority.
It is argued that the controversy around how sustain-
ability is defined [30] is the result of how people think
about the different perspectives of systems Sand functions
Fto be sustained over different time horizons Tby not
explicitly declaring them in the discourse of designing a
technological artifact [16]. To address this, it is suggested
that any actions referring to sustainability should answer the
following questions: What is to be sustained? For whom?
For how long? And at what cost?
The Karlskrona Manifesto [5] provides a focal point
for establishing a common ground for the requirements
engineering community to engage with sustainability by
advocating a set of fundamental principles and commit-
ments that underpin sustainability design. It includes the
importance of recognising that sustainability is an explicit
consideration, even if the primary focus of the system under
design is not sustainability. It also advocates that sustainabil-
ity must be viewed as a construct across five dimensions -
environmental, economic, individual, social and technical -
and consider the potential long term effects of systems.
3. Sustainability Awareness Framework
The sustainability awareness framework is composed of
a diagram and five-question sets for guiding semi-structured
interviews, as discussed below.
3.1. Sustainability Awareness Diagram
To visualise the effects that a software system could have
on the sustainability of its socio-technical environment, we
use an adapted radar chart (in line with [3]), which we
refer to as the Sustainability Awareness Diagram (SusAD).
The diagram is a visualisation tool which breaks down the
radar chart graph into the five interrelated dimensions of
sustainability. Each dimension is further divided into three
order of effects, denoting the effect that a software system
can cause across time; These are: immediate (i.e., caused
by the direct function of the system or its development),
enabling (i.e., arising from the application of a system over
time), or structural (i.e., referring to persistent changes
that can be observed at the macro level) [16]. These are
illustrated starting with the immediate effect in the centre,
the enabling in the middle, and the structural at the external
layer of the diagram (see Figure 1).
Let us exemplify the use of the diagram through the
Airbnb example presented in Section 1, and now illustrated
in Figure 1. As its immediate technical use effect, Airbnb
allows property owners to rent out their homes or rooms.
As a result of persistent rental via Airbnb, homeowners earn
55% more than the median long-term renting, which is an
enabling effect upon individuals. Increased median long-
term rent due to reduced long-term rental accommodation
stock is a structural economic effect of this platform. Fi-
nally, the gentrification of primarily non-white localities and
increased race separation is its’ structural social effect.
A SusAD diagram would normally have several chain-
of-effects, showing how one aspect of the software system
use (e.g., renting rooms out) causes other consequences
(e.g., removal of non-white communities and deepened
social segregation). Filling the diagram out without any
Figure 1. Simplified SusAD diagram for AirBnB system
guidance, however, can be a daunting task, as it is not easy
to elicit the required information. This is especially the case
as the requirements engineers are not trained to consider
sustainability issues [11]. The question-based framework
proposed below aims to facilitate SuSAD diagram construc-
tion and use to further discussions of sustainability-related
impacts between socio-technical system stakeholders.
3.2. Questions Framework
1) Instructions for the interviewer detailing the in-
terview process, e.g., consent to record and collect
data, need to consider chains-of-effects, etc.
2) Questions sheets for each sustainability dimension,
containing questions in plain text, examples, re-
minders and checkboxes. The sheet also suggests
prompts to encourage the interviewee to think fur-
ther, and examples to clarify some of the questions.
E.g. prompt is: “You mention how the system gives
the same treatment to people, what about taking
actions to ensure the outcome for each person
is comparable?”. A clarifying example would be:
“Systems sometimes enable the co-creation or co-
destruction of value when a customer interacts with
the business. For example, [...] when a customer
cannot self-serve as expected, her experience is
affected [...]. Does the system enables this kind of
co-creation or co-destruction of value?”
3) One note-taking form per dimension for the in-
terviewer to capture and record key effects and
dependencies throughout the conversation.
4) Questions for the interviewee, to help respondents
to follow the interview process (same as asked by
the interviewer, but without prompts).
When creating the questions sheets, we did not aim to
have an exhaustive list of topics or questions to address
every aspect of sustainability (which is quite impossible).
Instead, we aimed to give requirements engineers a starting
point for discussing possible sustainability effects. Thus, we
chose to cover only five topics for each dimension, although
additional (system and domain-specific) topics could well
arise for each dimension as the interview progresses. Our
starting sample of topics is listed in the Table 1.
Social (1) Sense of Community; (2) Trust; (3) Inclusiveness and
Diversity; (4) Equality; (5) Participation and Communica-
tion;
Individual (1) Health; (2) Lifelong learning; (3) Privacy; (4) Safety;
(5) Agency;
Environmental (1) Material and Resources; (2) Soil, Atmospheric and Water
Pollution; (3) Energy; (4) Biodiversity and Land Use; (5)
Logistics and Transportation;
Economic (1) Value; (2) Customer Relationship Management (CRM);
(3) Supply chain; (4) Governance and Processes; (5) Inno-
vation and R&D;
Technical (1) Maintainability; (2) Usability; (3) Extensibility and
Adaptability; (4) Security; (5) Scalability;
TABLE 1. TOPICS COVERED BY QUESTIONS IN EACH DIMENSION
Figure 2 exemplifies the questions for social dimensions
. The forms can be found in [13].
3.3. Extreme Scenarios and Chains of Effects
The questions (exemplified in Figure 2) are intended to
help uncover possible immediate and longer-term effects. In
order to encourage identification of such impacts, the frame-
work complements questions with a simple note-taking form
(shown in Figure 3) which explicitly draws the attention of
the interviewer to noting down the chains-of-effects.
Yet, interviewees might not consider long-term, com-
pounded impacts. To foster this, the framework suggests
posing an imaginary “extreme” scenario, where the intended
software system is accepted and used by millions of people
worldwide for a long period of time. The interviewee is
then invited to reflect on the impact that such a wide-
spread, long-term use of the system may have. For example,
“Imagine that many people worldwide are using this system
for decades. Think about how one thing may lead to another.
We call this a chain of effects. If people feel closer to their
neighbours, they may choose to buy from local shops or
choose proximity products, which can then foment local
businesses, and finally better distribute wealth.”
4. Research Methodology
This section describes how we have designed the Sus-
tainability Awareness Framework and evaluated the feasibil-
ity of its use. The emphasis is on the clarity and utility of
the question sets for eliciting potential effects of software
systems on sustainability. For this, the following research
questions were addressed:
RQ1: Does the framework encourage insightful dis-
cussions about the potential effects of software
systems on sustainability?
Figure 2. Front page of the question sheet for the social dimension
Figure 3. Extract of the notes taking form
RQ2: Does the framework help to identify the po-
tential chains-of-effects of software systems on
sustainability?
RQ3: How practical is the proposed approach?
4.1. Design of Question Sets
To elicit the question sets, we used an adaptation of
the Delphi method [17], [22]. Here, the members of the
Karlskrona Alliance on Sustainability Design [4] acted as
the panel of experts, as they have worked on topics of sus-
tainability for over six years, focusing on various domains,
ranging from social sustainability to energy, and impacts of
technology transfer.
The facilitator (first author of this paper) set out an
online document and invited panel members to contribute
views on factors that affect the five dimensions of sustain-
ability, and questions that a requirements engineer should
consider regarding these factors. Two example software
systems – Airbnb and a procurement system – were used
to ground the discussions. Airbnb was chosen as it is a
generally well known and commonly used system, whereas
the procurement system was studied by the panel of experts
in a previously reported work [3]. The panel then worked
through three rounds of activities to converge on the final
question sets: The first (contribution) round started with the
panel members providing their views by directly editing a
document and populating questions. In this round of ques-
tion elicitation, the panel members were asked to write down
their own contributions, without any other concern. The
facilitator closed this round when all the contributors felt
they had listed the most important issues. She then reviewed
all the questions, removed repetitions and rephrased the
questions for better readability. She also consulted selected
literature (previously suggested by the panel) to refine the
questions. These materials then constituted the result of the
first round.
At the second (review) round, the panel was requested to
review and comment on all of the results of the first round.
Two weeks were allocated for this round, enabling panel
members to contribute their views asynchronously. This re-
sulted in a number of issues raised with regards to previously
expressed views/proposed questions (e.g., noting unclear
statements, pointing out further implications of the noted
event/question, restating leading questions, disagreements
with the questions, etc.). The facilitator closed this round
when all panel members stated that they had completed their
reviews.
The third (consensus) round in the question elicitation
process started by the panellists reflecting on the feedback
given by others, and reviewing their views in this light. The
process continued with the clarifications and resolution of
the issues raised. This round was carried out through online
small group meetings, where two to four panellists met to
discuss the raised concerns. The round terminated when all
raised issues were resolved, and all panel members were
satisfied with the derived questions set.
4.2. Research Design for Question Evaluation
The feasibility study consisted of using the Sustainability
Awareness Framework with two groups of students from
different universities and countries in the spring of 2019. At
both universities, the students were provided with the same
sets of instructions, question sheets, and surveys.
Set Task: Students were instructed to carry out at least
one interview per sustainability dimension, for a software
system of their choice (i.e., whichever system they may
wish to select from their own environment). They were
to follow the process and use the material described in
Section 3.2 and also to fill out a SusAD diagram with the
most interesting (in their view) chain-of-effects. We note
that (i) since there are five sustainability dimensions, at
least five interviews are to be carried out by each student
group; (ii) we generally expect that each dimension would
be represented by a different (at least one) stakeholder.
Upon interview completion, the feedback was collected from
each interviewee, as well as from the interviewer about the
question-set for that specific dimension.
At the California State University, Long Beach
(CSULB), thirty-one students worked in nine groups, in a
writing-intensive third-year course on ICT for Sustainability.
Of these, thirty students were studying computer science
and one environmental management and engineering. The
assignment was introduced in class and explained with a
number of example diagrams as well as a mock interview
with a research assistant.
At Lappeenranta University of Technology (LUT), six-
teen students worked in nine 1-2 person teams. The task was
carried out as part of an MSc level course on Sustainability
and IT. Here all students had an IT background. The task
was introduced in class and explained with several examples
on the topic.
In both settings, the students had two weeks to carry
out the interviews, create a summary SusAD diagram and
to write a report, aggregating the information from their
interviews and discussing their reflections. This work was
carried out by the students independently, and they could
approach the researchers in case of questions.
4.2.1. Data Collection. We collected data from the feasi-
bility study by means of two surveys. The first one was
answered after each interview and contained the feedback
of the interviewer and the interviewee about the questions.
To collect the impression of the interviewee, students were
instructed to ask two additional questions at the end of
the interview and to summarise their answers in the online
survey. Since the student groups had to carry out at least
one interview per dimension, we received 57 responses: 23
from CSULB and 34 from LUT.
The second survey was answered after the group filled
out the SusAD diagram; it was meant to gather the collective
feedback of the group regarding the SusAD framework.
It was expected that each group would submit a single
response, but some students preferred to submit individual
responses. In total, 26 responses were received: 18 from
CSULB and 8 from LUT.
The students were also required to deliver a SusAD for
what they considered to be the most interesting chains-of-
effects and a textual list of the remaining ones.
All surveys were collected via Google forms. The dia-
grams and reports were collected via Dropbox at CSULB
and Moodle (a learning management system) at LUT. The
survey data was anonymous but reported on the system that
the group had been working on. Since the groups were
working on a different system, we were able to relate the
survey responses to the group work submission.
4.2.2. Data Analysis. We start with an explanation of (a)
how survey questions are mapped to Research Questions,
then we describe (b) how the quantitative values have been
calculated, followed by an explanation of (c) how the qual-
itative data was used in the analysis.
Mapping survey questions to RQ: The two surveys [13]
contained 35 questions in total. After most questions, stu-
dents were asked to provide free text explanations of their
choices. We grouped the survey questions in two categories:
the first one is composed of questions that directly contribute
to answering the RQs. These are shown in Table 2 and
mapped to the RQs in Table 3. The second group contained
qualifying questions that helped us to interpret the answers
to the first category. Due to space constraints, these are not
listed but are discussed in Section 5. For clearer traceability,
each survey question was mapped only to the RQs that it
most contributed to.
Note that the questions come from two surveys: (i) con-
ducted after the interview and (ii) conducted after depicting
the SuSAD. This allows us to have three data points for
drawing conclusions on some of the questions: the feedback
of interviewees, interviewers and the group as a whole.
(a) After each interview Type (b) After the SusAD Type
Interviewer Group
a.1 Have you understood the questions? Likert b.1 Did you get access to the right stakeholders? [1
per dimension]
y/n/p
a.2 Has the interviewee had difficulties in answering
the questions?
(Likert) b.2 Have the questions helped to identify possible
effects?
y/n
a.3 Did the questions enable discussions with the in-
terviewees?
Likert b.3 How did you like the process of asking the ques-
tions, noting down the key points and sowing them
back to the interviewee?
Likert
a.4 Did you get insightful answers using the questions
in this particular domain?
Likert b.4 Did the questions help to fill out the SusAD? Likert
a.5 Have the interviewees been able to think of chains
of effects for the extreme scenario?
Likert b.5 Was the resulting SusAD readable? Likert
(a) Interviewee
a.6 Were the questions easy to understand? text
a.7 Have the questions been useful for triggering relevant discussions on the possible effects of software systems? Why? text
TABLE 2. SURVE Y QU ES TI ON S THAT D IR EC TLY CO NT RI BUT ED T O RQS
Research question Survey ques-
tions
RQ1: Does the framework encourage insightful discussions about
the potential effects of software systems on sustainability?
Aim: see if questions help to uncover new, previously unconsid-
ered info.
a.3, a.4, a.7
RQ2: Does the framework help to identify potential chain of
effects of software systems on sustainability?
Aim: see if questions provide sufficient informaiton for chains of
effects and SuSAD diagram.
b.2, a.5
RQ3: How practical is the proposed approach?
Aim: look into the phrasing, access to expertise, the structure,
and the process of the proposed approach
a.1, a.2, a.6,
b.1, b.3, b.4,
b.5
TABLE 3. MAPP IN G BE TW EE N RQS AN D SU RVEY Q UE ST IO NS
Calculating quantitative results: We used three types of
closed-questions in the surveys: binary (yes/no), tertiary
(yes/no/partially) and a 5-points Likert scale. In order to
analyse them and define their contribution to the RQs,
we mapped responses to numerical values, calculating the
averaged and normalising on a scale from 1 to 5 as follows:
•Binary responses (y/n) were re-scaled as averages
over a 5-point scale: yes mapped to 5, no mapped
to 1.
•Tertiary responses (y/n/p) were normalised to 3 point
range in the range 1 - 5 , where No maps to 1,
Partially maps to 3, and Yes maps to 5.
•The 5-points Likert scale questions had two cases:
some were asked with a “positive phrasing” (e.g.
“Have you understood the questions?”), meaning
that a higher value would support RQ. These were
simply mapped from 1 to 5. Others were asked with
a “negative phrasing” (e.g. “Has the interviewee had
difficulties in answering the questions?”), meaning
that a higher value would oppose the RQ. To calcu-
lated the contribution of a negative question, we have
used the six-complement (i.e. a ”2” would become
a ”4”). In order to differentiate them, “negatively-
phrased”, the type of this questions are marked with
brackets; for example “(Likert)”.
For answering the research questions, we mapped av-
erage values from survey questions to a scale of support
to the Research Questions (RQs). There are five points
in the scale to which the values are mapped (using equal
Scale Value range
Strongly oppose (SO) 1 - 1.7
Oppose (O) 1.8 - 2.5
Inconclusive (I) 2.6 - 3.3
Support (S) 3.4 - 4.1
Strongly support (SS) 4.2 - 5
TABLE 4. MAPP IN G OF T HE VAL UE S TO TH E SC AL E OF S UP PO RT
intervals): Strongly Oppose (SO), Oppose (O), Inconclusive
(I), Support (S), and Strongly Support (SS) — see Table 4.
Analysing qualitative results: Since nearly every survey
question also asked to provide free text explanations for
the choice of the scale value, a substantial amount of text
was also collected for qualitative analysis. This text was
coded using a set of codes defined for each survey question,
following the qualitative content analysis approach [18].
Two researchers created the codebook, and collabora-
tively analysed the free-text responses from CSULB, af-
ter which the coding was validated by two additional re-
searchers. The codebook was reused for analysis of survey
data from LUT. Double-coding was used if a given free-text
response related to several code categories. For example,
the answer “I would need to interview more people with
different expertise” would be coded as “more interviews”
and “different expertise”.
Table 5 shows an extract from the codebook, which
contains the RQ it refers to, the survey question, whether the
contribution of the code towards the question is “positive” or
“negative”, the code, and the number of occurrences for both
universities. This qualitative data was used for interpreting
the respondents’ choices.
RQ SQ cont. codes occur.
A B
RQ1 a.7
pos. useful to expand on topic 10 12
neg. useful some not relevant for system 2 3
pos. useful for future 2 5
pos. useful because detailed 2 2
neg. useful insufficient expertise 1 2
pos. useful because vague 1 1
pos. useful to take action 0 1
pos. useful 0 4
pos. useful privacy 1 0
TABLE 5. EXAM PL ES O F CO DE S FO R SU RVEY Q UE ST IO NS
4.3. Threats to validity
Threats to validity hamper the ability to draw conclu-
sions from the evidence [33]. For the Delphi study, the
main risks come from the fact that the members of the
panel were drawn from the same collaboration group. Thus,
the breadth of the views to be represented in the question
sets is biased towards the group’s own view. Furthermore,
the anonymity of the panel members was not preserved
(as they know each other). This could cause a number of
additional biases, where attitudes towards the individuals
could have influenced the agreement or disagreement with
their provided views. In the future, this might be mitigated
by integrating input from experts from outside the group
and validating the question set through wider participation.
For the feasibility study, one of the main risks is the re-
active bias, as the students might answer the questionnaire
positively to meet the expectations of their teachers (i.e.
halo effect). Additionally, there are several confounding
factors which may affect the outcome that was not taken
into account, such as differences in knowledge regarding
sustainability issues of the students and the level of expertise
of the interviewees. Since we worked with two different
groups of students from two different universities, these
factors cannot be ruled out completely. However, we en-
deavour to ensure a similar perspective on sustainability
and knowledge of the questions and the SuSAD method by
delivering the same introductory sessions and instructions to
both groups. Another main risk is the possible bias caused
by result interpretation. We applied researcher triangulation
and mixed qualitative and quantitative methods to minimise
this risk. Finally, we do not attempt to generalise the findings
from these two application cases; we only demonstrate the
feasibility of using the SuSAD for relating requirements
engineering process to topics of sustainability.
5. Results
RQs data Int‘ers Int‘ees Group
a.3 a.4 a.7 -
ASS SS SS -
BI S SS -
RQ1
Supported
a.5 b2
AI SS
BS SS
RQ2
Supported
b.1
a.1 a.6 a.2 Soc. Ind. Eco. Env. Tec. b.3 b.4 b.5
AS S S S SS I S SS O S S
BS S O O I O I S S S S
RQ3
Supported
TABLE 6. SUMMARY OF THE RESULT S
This section provides the results of our data analysis and
how they answer the RQs. Table 6 summarises the extent to
which the different surveys provided evidence for answering
our RQs based on the quantitative data and based on the
normalised answers to the survey questions. Next, we detail
the answer to each research question. Average quantitative
results that led us to the final conclusions about the support
to a given concept are shown in brackets.
RQ1: Does the framework encourage insightful dis-
cussions about the potential effects of software systems
on sustainability?
Three survey questions contributed to this RQ; the an-
swers we received are summarised below. The first survey
question asked whether the SusAD questions enabled dis-
cussions with the interviewee (a.3) and subjects supported
(3.76) this notion, with strong support from CSULB and
inconclusive support from LUT. Analysing the qualitative
codes for that question (shown with occurrences in paren-
thesis), we observed that some students stated that the ques-
tions led to more questions (17%, “lead to more questions”
A=7 B=3), and helped to elaborate the answers (10%,
“elaboration” A=2 B=4), e.g. one respondent said: “They
enabled,[...] the interviewee could direct the direction of
topic and voice his personal opinions without influence from
us”. Furthermore, the questions were reported to be a good
support (9%, “good support from questions” A=7 B=3),
and to encourage an interviewee who is knowledgeable
(12% “knowledgeable interviewee” A=2 B=3) or passionate
(5% “enable passionate interviewee” A=2 B=1) intervie-
wee, e.g. “This topic is something the interviewee was
very passionate about”. We note a difference in the textual
answers received from two universities, which could be due
to the cultural difference in communication in California
vs Finland: while in CSULB only one interviewee was
described as terse, in the LUT, six interviewees and one
interviewer received such description. This could also be
observed in the difference in the number of codes generated
for the data from these two universities. (Due to space
constraints, we will no longer show the codes and occur-
rences related to qualitative findings, but all explanations to
students’ choices have been analysed in the same way as
above.)
The second survey question asked whether insightful
answers (a.4) had emerged using the questions in this
particular domain. Again, the students supported (3.84)
this notion, with strong support by CSULB and LUT. The
most cited reason for getting good insights was the inter-
viewee opening new perspectives about the domain (14%),
followed by having a lot to discuss (7%): “The questions
explored areas I would not have thought of on my own.
The questions focused on the key concepts of a system.”
and “The applicable questions where very insightful and
invoked lots of back and forth discussion.” The most fre-
quent reason for not getting much insight was insufficient
domain knowledge (7%). Students got the best insights into
the individual dimension, followed by environmental and
technical. No dimension was particularly problematic. To
get more information on insights, we also asked students
whether anything unexpected came up. Only 26% reported
unexpected occurrences, the most common being new per-
spectives (8%) and effects of the system (8%).
Finally, the third survey question asked the interviewees
perceptions on whether the questions had been useful for
triggering relevant discussions on the possible effects of
software system (a.7). Interviewees from both universities
strongly supported (5.0) this notion. E.g. one student
mentioned “Yes, we discussed many topics triggered by the
questions asked.”. A majority of students confirmed that the
interviewees found the questions helpful to expand on the
topic, to think towards the future, and to discuss in more
detail. Helpful pointers towards exploring environmental as-
pects and privacy were mentioned. The reasons for reporting
less usefulness were that some questions were not relevant
for that particular system (9%) and that the interviewees did
not consider themselves sufficiently knowledgeable (5%).
Students from LUT often focused on discussing the future,
while group A answered more diversely and made more use
of the proposed questions.
Intermediate conclusion: The answers suggest that
these two studies support (4.07) RQ1. That is, that the ques-
tions enabled relevant discussions, both for the interviewers
and interviewees, and led to insightful findings.
RQ2: Does the framework help to identify the poten-
tial chain of effects of software systems on sustainability?
Two survey questions contributed to this RQ. We sum-
marize their answers below, adding insights from comple-
mentary questions to contextualize the answers.
The first survey question asked whether the SusAD ques-
tions helped to identify possible effects (b.2), students for
both universities supported (4.01) this notion, 35% stating
that the questions helped very much, 46% that the helped,
and 19% that helped a little bit1. Students from LUT had
more difficulty than students from CSULB. In order to get
more information, we asked how easy it was to identify the
effects of each dimension separately. The most problematic
one was the economic dimension, followed by the social,
environmental, individual and technical, in this order. We
also asked what else would have helped to identify possi-
ble effects. The two primary responses were around using
additional research outside of the interviews, such as online
sources (15%), and more interviewees (12%). Furthermore,
side conversations, data trends, and more knowledgeable
interviewees would have helped individual respondents.
When asked whether interviewees had been able to think
of chains of effects for the extreme scenario (a.5), the over-
all answer was inconclusive (supported by LUT students but
inconclusive for CSULB). To explore this further, students
were asked for how many topics (in Table 1) interviewers
were able to think of chains-of-effect: 78% thought of chains
of effect for up to three key topics, 8.5% for 4 - 5 topics,
and only 3.5% for more than five topics.
To see if the extreme scenario helped interviewees to
think of chains-of-effect, we asked whether the students
had encouraged the interviewee to think about the extreme
scenario and about effects across dimensions. We observe
that those with difficulties in identifying chains-of-effect
were less encouraged to think about the extreme scenario
and cross-dimensional effects. The opposite is also true,
as shown in Table 7. In addition, the more the intervie-
1. For historical reasons (to allow a comparison with a previous work
outside of the scope of this paper) this question allowed a yes/no answer
and was complemented to get the degree to which the questions helped.
For this reason, the calculation of the results was slightly different, with a
“no”=1, “a little bit”=3”, “helped”=4 and a “very much”=5.
wee was encouraged to think of an extreme scenario and
across dimensions, the greater the number of topics (s)he
identified chains-of-effect for. This correlation is shown in
Table 8. These suggest that the extreme scenario and the
encouragement given by the interviewee are indeed useful
to identify chains-of-effect. Finally, we also asked for how
many topics (in Table 1) interviewers were able to think of
chains-of-effect. 78% of the interviewees thought of chains
of effect for up to three key topics, 8.5% for four-five topics,
and only 3.5% for more than five topics. Interestingly, the
more the interviewee was encouraged to think of an extreme
scenario and across dimensions, the greater the number of
topics he or she identified chains-of-effect to. This was
also observed by students, who stated that “Giving them
to consider of chains of effects allows for their thought
process to expand past just one dimension” and “All these
things are interrelated and are necessary to examine when
researching a topic like this.” Finally, around 30% of the
students admitted to not having asked the interviewee to
consider the extreme scenario. The primary reasons for not
doing so varied greatly. The most cited were that it was that
is was difficult to include the questions (5%) and that the
extreme scenario was not relevant for the system (5%).
Ability to think of
chains-of-effects
Encouraged to think about
the extreme scenario
Encouraged cross di-
mensional thinking
1 - 2 2.39 1.78
4 - 5 4.03 3.19
TABLE 7. CORRELATION OF ENCOURAGEMENT AND EASE TO THINK
OF C HA INS O F EFF EC T (SMALLER NUMBER =GR EATE R DI FFIC ULT Y &
LESS ENCOURAGEMENT)
Normalized number
of topics
Encouraged to think about
the extreme scenario
Encouraged cross di-
mensional thinking
1 2.8 2.1
2 3.4 2.7
3 4.2 3.7
4 5.0 4.4
TABLE 8. CORRELATION OF ENCOURAGEMENT AND NUMBER OF
TOPICS WITH CHAINS-OF-E FFE CT (S MA LL ER N UM BE R =LE SS
ENCOURAGEMENT)
Intermediate conclusion: The answers suggest that
these two studies support (3.76) RQ2. That is, the questions
help to identify effects and chains-of-effects, highlighting
the importance of the extreme scenario and the encourage-
ment to think across dimensions.
RQ3: How practical is the proposed approach?
Seven survey questions contributed to this RQ.
The first survey question refers to whether the intervie-
wee understood the questions (a.1). Students from both
universities supported (4.08) this idea. However, about 1/4
of the students pointed out that there were questions with
unclear definitions, which points out the need to review
and refine the questions. The questions that caused greater
confusion were about the supply chain (23%) and agency
(11%). No dimension was particularly problematic. Also,
10% of the students felt that some questions were not rele-
vant to the system at hand. This was expected; in creating
a general framework, we knew we could neither be specific
to a domain nor comprehensive.
The second survey question refers to interviewee’s view
on whether the questions were easy to understand (a.2).
Interviewees from both universities supported (3.64) this
notion. One interviewee mentioned “Yes, the questions got
me thinking about change and the decisions we have to make
for a sustainable future.” Furthermore, responses indicate
that some questions were perceived as vague (12%), again
showing the need for reviewing and refining the sheets.
Other interviewees felt that some questions were not relevant
for the system under analysis (9%), or they needed time to
be interpreted (7%).
The third survey question refers to the interviewer’s
perceptions on whether the interviewee could understand
the questions (a.6). The answer to this question was in-
conclusive (-2.76), with the data showing that the students
from LUT opposed the notion while no conclusion could
be drawn for CSULB. One possible contributor to the
worse perception in LUT was that the students received
the questions in English, but conducted the interviews in
Finnish. It is also interesting to note the following dif-
ferences: interviewees themselves felt they understood the
questions, while interviewers thought otherwise. The greater
difficulties reported by students were that some questions
had no relation with the system (15%); e.g. one student said
“Difficult to get a conversation going about the topic, the
interviewee did not consider there to be much to discuss”.
Other reported difficulties were lack of knowledge of the
interviewee (10%) and the wording of the questions (9%).
To get a deeper understanding of their answers, we asked
whether students had interviewed an expert or a surrogate.
Overall, about 70% of the interviewees were surrogates,
and 30% had knowledge or expertise on the topic. We
found little correlation between the level of expertise and
the observed difficulty in understanding the questions.
The fourth survey question refers to whether the students
felt that they got access to the right stakeholders (b.1).
The overall result was inconclusive (3.3), with a ‘support’
from CSULB and an ‘inconclusive’ from LUT. Of the five
dimensions, both groups found it easiest to get access to
relevant stakeholders for the individual and technical dimen-
sions. Experts for the economic dimension proved hardest
to obtain for both groups.
The fifth survey question refers to whether the students
liked the process of asking the questions (b.3), noting down
the key points and showing them back to the interviewee.
Data was inconclusive (2.83), with students from CSULB
opposed to the practice and LUT supporting it. The reasons
why subjects liked and disliked the process varied greatly.
The reasons for positive ratings were that subjects liked
interviewing (12%) and found it to be a good practice (8%),
while the main reason for disliking it was that they felt it
was redundant (12%). In order to put these answers into
context, we should note that 34% of the students in CSULB
and 50% of the students in LUT had not used the form at all
as they did not find it useful (12%), had forgotten to bring
it (8%), or had made their own sheet (8%). Using the form
could also have contributed to the number of chains-of-effect
that the interviewee was able to think of since the students
were expected to show the form back to the interviewer in
order to ask them about the extreme scenario.
The sixth survey question asks about whether the SusAD
questions helped to fill out the Sustainability Awareness
Diagram (b.4). Both universities supported this notion.
Some students found the questions helpful for the diagram
(9%) and to extract key points (7%): “Answers were straight
forward, so the points were easy to establish on the SusAD.”
Only two respondents mentioned a negative impact in that
certain questions could lead to bias (2%) and that they were
too direct (2%).
The last survey question asks whether the resulting
SusAD was readable (b.5). Students from both universities
supported this idea (3.92). The three main explanations
were that the diagram was readable (23%), the students
decided only to include key points (15%), and that they
were able to make links (12%).
Finally, the last point worth commenting is the time
of the interviews. Most of them took between 15 and 30
minutes (52%), followed by 30 to 60 minutes (27%), and
by less than 15 minutes (12%). Even though the interviewees
who identified most chains-of-effect had also talked for
longer, no clear correlation was found between the time of
the interview and the number of chains-off-effect identified.
Intermediate conclusion: The answers suggest that
these two studies support (3.42) RQ3. That is, on their own
perceptions, both interviewers and interviewees understood
the questions, thought they helped to fill out the SusAD,
and that the latter was readable.
Overall, the feasibility studies support the three research
questions. Questions can be understood, enable discussion
and interesting insights, help to identify chains-of-effect
(especially when the extreme scenario is used), and finally
are reasonably practical in terms of time, the wording of the
questions and the resulting diagram. Yet, there is still room
for improvement in the process and on some questions.
6. Related work
While traditional RE methods and tools do not explicitly
facilitate the discussion of sustainability-related concerns,
research suggests that existing RE techniques, approaches
and methods can serve as a starting point for practitioners
to integrate sustainability into their practice [8]. Chitchyan
et al., [12] identified several techniques that help to support
sustainability in RE and demonstrated the application of
some of these techniques using two case studies. Similarly,
Mireles et al., [20] proposed a conceptual framework for the
classification of sustainability-aware requirements methods
to support practitioners in the selection of an appropriate
method to address stakeholders’ needs.
A number of studies have attempted to integrate sus-
tainability into specific methods and techniques. Seyff et
al., [28] extended the WinWin Negotiation Model to con-
sider the impact of requirements on sustainability. The
results of the study suggested that while the approach
stimulated the discussion on the multiple dimensions of
sustainability, stakeholders found it challenging to identify
the impacts of a given requirement on sustainability and
were not able to identify the long-term effects. Cabot et
al., [8] proposed using i* for modelling early requirements
as a way to visualise the impact of alternative options on
sustainability goals and to analyse the conflicts between
sustainability and other problem-specific objectives. Simi-
larly, Mussbacher and Nuttall [21] argue that goal models
are an ideal candidate to model the assessment of alter-
natives for sustainability as they express the hierarchy of
needs from high-level goals to specific activities for various
stakeholders,also allowing reasoning about alternatives and
their impact on high-level goals. In contrast to Cabot et
al. [8], they proposed a method to combine Goal-oriented
Requirement Language (GRL) because of its support for
indicators with a quantitative sustainability assessment ap-
proach based on time cost that would be applicable to a
broad range of development projects. However, no formal
evaluation of either method was conducted to demonstrate
their efficacy.
Brito et al., [6] proposed a concern-oriented require-
ments approach that allows both the modelling of sustain-
ability concepts and their relationships and the management
of conflicting situations triggered by impacts among sus-
tainability dimensions or between those and other system
concerns. In contrast to the previous studies, Penzenstadler
et al. [23] explored how the concept of leverage points could
be used to make sustainability issues more tangible in sys-
tem design in a public transportation system [29]. However,
both approaches do not discuss the role of requirements
engineering for sustainability engineering.
In addition, a number of alternative approaches have
been proposed, including the use of a recommender system
to overcome the barriers of incorporating sustainability into
the software engineering process [25], the application of a
sustainability requirement pattern to guide the specification
of sustainability requirements [26], a tool for requirement
engineers to analyse the requirements’ impacts on system
sustainability [2], and a meta-model which integrates sus-
tainability dimensions with the other quality attributes [27].
7. Discussion and Conclusions
This paper highlights that the software developed today
does not exist in isolation but forms part of the socio-
technical system within which it gets used. Thus, the paper
advocates that requirements engineers must tackle concerns
of sustainability of such socio-technical systems, starting
from the time when software requirements are elicited and
specified. A number of issues have implicitly motivated this
work and remain topics for an open discussion:
SusAD as a Systems Thinking activity: The proposed
Sustainability Awareness framework, essentially, incorpo-
rates simplified elements of Systems Thinking [19] activities
into the RE process. Like the systems thinking disciplines,
our work advocates consideration of the holistic system
within which the software-to-be will function, attending not
only to the functional and non-functional properties of the
software system but also to the indirect, longer-term impacts
that its use could cause and considering the risks and uncer-
tainties that this may engender. However, we are also well
aware that the discipline of software engineering already
suffers from high costs and late delivery problems, and ad-
ditional “whole systems” analysis could prove too costly and
complex to be useful. In truth, this very problem stifles the
use of techniques such as Soft Systems [10] methodology or
Critical Systems Thinking [15] in the software engineering
domain. To avoid both unbounded complexity and cost, our
approach supports the exploration of potential sustainability
impacts through both simple guiding question sets and im-
pact recording tools and an elicitation scenario. This allows
to keep the focus on identification of sustainability effects
across three orders of effect, yet provides a boundary to the
otherwise potentially overwhelming systems thinking and
analysis task.
Systems vs Software Requirements Engineering: It has
long been recognised that any systems engineering project
requires a requirements engineering activity. Indeed, require-
ments engineers working within the systems engineering
domains (such as construction, and chemical process en-
gineering) are well attuned to the systemic impact analysis.
Yet, this is too often amiss within SE, where RE has too
often limited itself to software requirements engineering,
disregarding the wider implications that the software could
cause. We hope that this paper has motivated sufficiently
clearly (using the Airbnb platform example) the need to
tackle such disregard of the socio-technical impacts of soft-
ware systems. In this paper, we provide a simple framework
to take the first steps in addressing this omission.
Requirements Engineers as leads for Sustainability En-
gineering: Researchers have previously argued [3] that re-
quirements shape the software systems, which, in turn, shape
the socio-technical systems within which they reside. As
such, if so engineered, software systems could become
the drivers towards sustainable societal, environmental, and
economic settings. Thus, the present work endeavours to
support requirements engineers in taking on the role of
sustainability engineers, through timely consideration and
fostering informed choices in tackling the challenge of the
socio-technical systems requirements engineering.
In conclusion, we found evidence that the Sustainability
Awareness Framework presented in this paper provides a
simple and accessible framework to elicit awareness of the
impacts that software systems could have. I.e., it could
be used by students independently and without previous
knowledge. Carver argues that students are a close enough
representation for practitioners [?]. Having evaluated the
questions that guide such awareness-building activity with
two sets of student groups, we find sufficient evidence that
the questions and elicitation scenario provide the desired
support. Yet, much work remains to be done, including
improving the clarity of the questions, supporting impact
visualisation (currently done manually), and considering
the specialisation of the questions per relevant application
domains, studying the framework across different cultures,
etc.
Acknowledgments
The authors would like to acknowledge EPSRC grant
EP/R007373/1, Digitaldialog 21 funded by the Ministry for
Science, Research and Art Baden-Wrttemberg. Each named
author made a significant contribution both in terms of ideas,
discussions, evolution, and physical writing to be named
on the paper. All authors participated in the design and
development of the framework applied in the study. The
authors would also like to thank Dr. Christoph Becker for
insights that led us to change the focus from an analytic to
an awareness framework, and Ms. Nanae Aubry for helping
with part of the coding of the feedback forms.
References
[1] Airbnb, inc. http://www.airbnb.com, 2019. Accessed: 2019-03-01.
[2] A. D. Alharthi, M. Spichkova, and M. Hamilton. Susoftpro: Sustain-
ability profiling for software. In IEEE 26th International Require-
ments Engineering Conference (RE), pages 500–501, Aug 2018.
[3] C. Becker, S. Betz, R. Chitchyan, L. Duboc, S. Easterbrook, B. Pen-
zenstadler, N. Seyff, and C. Venters. Requirements: The key to
sustainability. IEEE Software, 33(1):56–65, 2016.
[4] C. Becker, R. Chitchyan, L. Duboc, S. Easterbrook, M. Mahaux,
B. Penzenstadler, G. Rodriguez-Navas, C. Salinesi, N. Seyff, C Ven-
ters, C. Calero, S. Akinli Kocak, and S. Betz. Karlskrona manifesto
on sustainability design, 2015. https://www.sustainabilitydesign.org/.
[5] C. Becker, R. Chitchyan, L. Duboc, S. Easterbrook, B. Penzenstadler,
N. Seyff, and C. Venters. Sustainability design and software: The
karlskrona manifesto. In Proceedings of the 37th Intl Conference on
Software Engineering-Volume 2, pages 467–476. IEEE Press, 2015.
[6] I. S. Brito, J. M. Conejero, A. Moreira, and J. Arajo. A concern-
oriented sustainability approach. In 2018 12th International Confer-
ence on Research Challenges in Information Science (RCIS), pages
1–12, May 2018.
[7] Gro Harlem Brundtland, M Khalid, S Agnelli, et al. Our common
future. New York, 1987.
[8] J. Cabot, S. Easterbrook, J. Horkoff, L. Lessard, S. Liaskos, and
J. Mazon. Integrating sustainability in decision-making processes:
A modelling strategy. In 31st International Conference on Software
Engineering - Companion Volume, pages 207–210, May 2009.
[9] P. Checkland. Systems thinking. Rethinking management information
systems, pages 45–56, 1999.
[10] P. Checkland and Poulter J. Learning for Action: A Short Definitive
Account of Soft Systems Methodology and its Use, for Practitioners,
Teachers and Students. John Wiley and Sons Ltd, 2006.
[11] R. Chitchyan, C. Becker, S. Betz, L. Duboc, B. Penzenstadler,
N. Seyff, and C. Venters. Sustainability design in requirements engi-
neering: state of practice. In Proceedings of the 38th Intl Conference
on Software Engineering Companion, pages 533–542. ACM, 2016.
[12] R. Chitchyan, S. Betz, L. Duboc, B. Penzenstadler, S. Easterbrook,
C. Ponsard, and C. Venters. Evidencing sustainability design through
examples. In Proceedings of the 4th Intl. Workshop on Requirements
Engineering for Sustainable Systems, 2015.
[13] Duboc L.; Betz S.; Penzenstadler B.; Akinli Kocak S.; Chitchyan R.;
Leifler O.; Porras J.; Seyff N.; Venters C. Supplementary materials,
2019. https://www.sustainabilitydesign.org/susad-forms/.
[14] S. Easterbrook. From computational thinking to systems thinking: A
conceptual toolkit for sustainability computing. In ICT for Sustain-
ability 2014 (ICT4S-14), Stockholm, Sweden, 2014.
[15] M.. Goodchild and D. Janelle. Toward critical spatial thinking in the
social sciences and humanities. GeoJournal, 75(1):3–13, Feb 2010.
[16] L. Hilty and B. Aebischer. Ict for sustainability: An emerging research
field. In ICT Innovations for Sustainability, pages 3–36. Springer,
2015.
[17] J. Landeta. Current validity of the delphi method in social sciences.
Technological Forecasting and Social Change, 73(5):467 – 482, 2006.
[18] P. Mayring. Qualitative content analysis: theoretical foundation, basic
procedures and software solution. 2014.
[19] D. Meadows. Thinking in Systems: A Primer. Chelsea Green
Publishing, 2008.
[20] G. A. G. Mireles, M. . Moraga, F. Garca, and M. Piattini. A classifica-
tion approach of sustainability aware requirements methods. In 2017
12th Iberian Conference on Information Systems and Technologies
(CISTI), pages 1–6, June 2017.
[21] G. Mussbacher and D. Nuttall. Goal modeling for sustainability: The
case of time. In IEEE 4th International Model-Driven Requirements
Engineering Workshop (MoDRE), pages 7–16, Aug 2014.
[22] C. Okoli and S. Pawlowski. The delphi method as a research tool:
an example, design considerations and applications. Information &
Management, 42(1):15 – 29, 2004.
[23] B. Penzenstadler, L. Duboc, C. C. Venters, S. Betz, N. Seyff,
K. Wnuk, R. Chitchyan, S. M. Easterbrook, and C. Becker. Soft-
ware engineering for sustainability: Find the leverage points! IEEE
Software, 35(4):22–33, July 2018.
[24] B. Penzenstadler, A. Raturi, D. Richardson, and B. Tomlinson. Safety,
security, now sustainability: The nonfunctional requirement for the
21st century. IEEE software, 31(3):40–47, 2014.
[25] K. Roher and D. Richardson. A proposed recommender system for
eliciting software sustainability requirements. In 2013 2nd Inter-
national Workshop on User Evaluations for Software Engineering
Researchers (USER), pages 16–19, May 2013.
[26] K. Roher and D. Richardson. Sustainability requirement patterns. In
2013 3rd International Workshop on Requirements Patterns (RePa),
pages 8–11, July 2013.
[27] T. Saputri and S. Lee. Incorporating sustainability design in re-
quirements engineering process: A preliminary study. In Asia Pacific
Requirements Engineering Conference, pages 53–67. Springer, 2016.
[28] N. Seyff, S. Betz, L. Duboc, C. Venters, C. Becker, R. Chitchyan,
B. Penzenstadler, and M. Nbauer. Tailoring requirements negotiation
to sustainability. In 2018 IEEE 26th International Requirements
Engineering Conference (RE), pages 304–314, Aug 2018.
[29] UK Commission for Integrated Transport. Transport and climate
change: Advice to government from the commission for integrated
transport. http://www.cambridgeenergy.com/archive/2007-02-08/
commission-integ-trans.pdf, 2007.
[30] C. Venters, C. Jay, L. Lau, M. Griffiths, V. Holmes, R. Ward, J. Austin,
C. Dibsdale, and J. Xu. Software sustainability: The modern tower
of babel. CEUR Workshop Proceedings, 1216:7–12, 2014.
[31] C. Venters, Seyff N., Becker C., Betz S, Chitchyan R., Duboc L.,
Mcintyre D., and Penzenstadler B. Characterising sustainability
requirements: A new species, red herring, or just an odd fish? In
IEEE/ACM 39th Intl Conference on Software Engineering, pages 3–
12. Institute of Electrical and Electronics Engineers Inc., 6 2017.
[32] D. Wachsmuth and A. Weisler. Airbnb and the rent gap: Gentrification
through the sharing economy. Environment and Planning A: Economy
and Space, 50(6):1147–1170, 2018.
[33] C. Wohlin, P. Runeson, M. Hst, M. Ohlsson, B. Regnell, and
A. Wessln. Experimentation in Software Engineering. Springer
Publishing Company, Incorporated, 2012.
