Reviewing qualitative research approaches in the context of critical infrastructure resilience

Abstract

Modern societies are increasingly dependent on the proper functioning of critical infrastructures (CIs). CIs produce and distribute essential goods or services, as for power transmission systems, water treatment and distribution infrastructures, transportation systems, communication networks, nuclear power plants, and information technologies. Being resilient becomes a key property for CIs, which are constantly exposed to threats that can undermine safety, security, and business continuity. Nowadays, a variety of approaches exist in the context of CIs’ resilience research. This paper provides a state-of-the-art review on the approaches that have a complete qualitative dimension, or that can be used as entry points for semi-quantitative analyses. The study aims to uncover the usage of qualitative research methods through a systematic review based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). The paper identifies four principal dimensions of resilience referred to CIs (i.e., techno-centric, organisational, community, and urban) and discusses the related qualitative methods. Besides many studies being focused on energy and transportation systems, the literature review allows to observe that interviews and questionnaires are most frequently used to gather qualitative data, besides a high percentage of mixed-method research. The article aims to provide a synthesis of literature on qualitative methods used for resilience research in the domain of CIs, detailing lessons learned from such approaches to shed lights on best practices and identify possible future research directions.

Full text

Vol.:(0123456789)
1 3
Environment Systems and Decisions (2021) 41:341–376
https://doi.org/10.1007/s10669-020-09795-8
REVIEW
Reviewing qualitative research approaches inthecontext ofcritical
infrastructure resilience
R.Cantelmi1,2· G.DiGravio1· R.Patriarca1
Accepted: 22 December 2020 / Published online: 24 January 2021
© The Author(s) 2021
Abstract
Modern societies are increasingly dependent on the proper functioning of critical infrastructures (CIs). CIs produce and dis-
tribute essential goods or services, as for power transmission systems, water treatment and distribution infrastructures, trans-
portation systems, communication networks, nuclear power plants, and information technologies. Being resilient becomes
a key property for CIs, which are constantly exposed to threats that can undermine safety, security, and business continuity.
Nowadays, a variety of approaches exist in the context of CIs’ resilience research. This paper provides a state-of-the-art
review on the approaches that have a complete qualitative dimension, or that can be used as entry points for semi-quantitative
analyses. The study aims to uncover the usage of qualitative research methods through a systematic review based on PRISMA
(Preferred Reporting Items for Systematic Reviews and Meta-Analyses). The paper identifies four principal dimensions of
resilience referred to CIs(i.e., techno-centric, organisational, community, and urban) and discusses the related qualitative
methods. Besides many studies being focused on energy and transportation systems, the literaturereview allows to observe
that interviews and questionnaires are most frequentlyused to gather qualitative data, besides a high percentage of mixed-
method research. The article aims to provide a synthesis of literature on qualitative methods used for resilience research
in the domain of CIs, detailing lessons learned from such approaches to shed lights on best practices and identify possible
future research directions.
Keywords Resilience management· Resilience engineering· Critical infrastructure· Infrastructure resilience· Review
1 Introduction
Nowadays, due to the frequent occurrence of natural dis-
asters or catastrophic events related to human activities, a
global awareness on the strategic importance of Critical
Infrastructures (CIs) has increasingly grown in academic
and policy-making environments (Kete etal. 2018). CIs are
large-scale, man-made systems that function interdepend-
ently to produce and distribute essential goods (such as
energy, water, and data) and services (such as transporta-
tion, banking, and healthcare). An infrastructure is defined
as critical if its incapacity or destruction has a significant
impact on health, safety, security, economics, and social
well-being of a state (Council Directive 2008/114/EC of 8
December 2008).
CIs are various by nature, e.g. physical-engineered,
cybernetic, organisational, etc., and by environment (geo-
graphical, natural), and operational context (political/legal/
institutional, economic, etc.) (Zio 2016). Examples of CIs
are those providing services of energy (including genera-
tion, transmission, distribution and storage, in regard with
electricity, oil and gas supply); transportation (including
rail, roads, aviation and waterways); information and tel-
ecommunication (including information systems, industrial
control systems (SCADA), Internet, fixed and mobile com-
munications and broadcasting) (seee.g. Bunney etal. 2016;
Hughes etal. 2016; Leu and Peter 2016; Pagán-Trinidad
etal. 2019)). A failure in such an infrastructure can be even
more critical because it can produce cascading failures,
sending ripple effects at regional, national, or international
scale. These failures are often originated by natural hazards,
* R. Patriarca
riccardo.patriarca@uniroma1.it
1 Department ofMechanical andAerospace Engineering,
Sapienza University ofRome, Via Eudossiana, 18,
00184Rome, Italy
2 Land Armaments Directorate, Ministry ofDefence, Via di
Centocelle, 301, 00187Rome, Italy
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

342 Environment Systems and Decisions (2021) 41:341–376
1 3
such as earthquakes (Kachali etal. 2012b, a) and hurricanes
(Comes etal. 2013; Comes and Van De Walle 2014).
Managing CIs today means managing CIs’ resilience.
Starting from its Latin etymological root on the word resilio
(i.e. to leap back), resilience denotes the capacity of a sys-
tem to recover from challenges or disruptive events. There
are several definitions of resilience available in literature
grounded on the concept of robustness and adaptation in
line with a seminal ecology research (Holling 1973). Among
them, the definition by Bruneau etal. (2003) gives a wide
perspective for CIs’ management: “the ability of the sys-
tem to reduce the chances of shock, to absorb a shock if
it occurs and to recover quickly after a shock (re-establish
normal performance)”. This ability is recognised to result
from four properties (i.e. robustness, redundancy, resource-
fulness, rapidity), which are inter-related through technical,
organisational, and social aspects. These abilities encom-
pass slightly different perspectives which jointly offer the
opportunity to deal with micro-meso-macro level CIs’ man-
agement (Bergström and Dekker 2014): from pure techni-
cal artefacts, towards social structures made up by small
groups (Zemba etal. 2019), or large organisations (Wood
etal. 2019). Resilience is relevant for management because
it focuses on performance levels, as well as time and cost
required to reach them (Vugrin etal. 2010b). For example,
within civil infrastructures, resilience is defined as “the
capacity of a civil infrastructure system to minimise per-
formance loss due to disruption, and to recover a specified
performance level within acceptable predefined time and
cost limits” (Gay and Sinha 2013).
Nonetheless, as for these definitions, operationalising
the concept of resilience can become puzzling for CIs’
management, especially considering their interdepend-
ent nature which recalls the systems-of-systems treats.
Managing resilience calls for a reconsideration of avail-
able risk management methods and models in order not
to fall within the trap of reductionism and over-simplified
linear modelling techniques. New frameworks are indeed
needed to integrate different perspectives (e.g. topologi-
cal, functional, static, dynamic) and to ensure the capac-
ity of dealing with complexity and uncertainties (Kröger
and Zio 2011). Available literature shows that such
complexity is addressed by frameworks based on either
qualitative or quantitative approaches, or a combination
of the two. Quantitative data usually refer to historical
data, design specifications, climate models, or laboratory
experiments, whereas qualitative assessments come from
surveyed experts or operators, i.e. community leaders,
technical operators, managers, public decision makers
(Cegan etal. 2017; Kurth etal. 2017). A comparison on
the features of such approaches has been discussed widely.
For example, Linkov and Palma-Oliveira (2017) prepared
a workshop where military commanders and civilian
decision makers were brought together to explore how
they make resilience-driven choices on a daily and long-
term basis. According to the results of the study, a qualita-
tive approach allows for greater flexibility in applications
ranging from well-known hazards to highly uncertain
ones, thanks to subject matter experts’ (SMEs) judgments
(Münzberg etal. 2015). These latter can overcome those
frequent cases where retrieving reliable quantitative data
is challenging (Häring etal. 2017). Thereafter, qualitative
information might be integrated into dedicated models to
define representative and synthetic indexes.
This concept can be explored more systematically via a
tiered framework to resilience assessment, intended to ease
policy development and favour the adoption of resilience
practices (Linkov etal. 2018). This framework consists of
three different tiers at which a complex problem such as CI
resilience can progressively analysed. Each tier has its own
specific objectives, methods, and tools. Tier I involves the
use of existing data, expert judgement, and conceptual mod-
els, in order to provide a comprehensive understanding of
system’s functioning. At Tier II, decision analysis methods
(such as Multicriteria Decision Analysis) are utilised, e.g.
the Resilience Matrix (Fox-Lent etal. 2015) or the Analytic
Hierarchy Process (AHP) (Saaty 1990). Tier II encompass
methods intended to reveal the structure of the system, to
check scenarios, or to compare alternatives, that later on in
Tier III can be further specified. The last tier seeks to pro-
vide the highest fidelity in modelling real-world systems,
through, for example, system dynamics models, graph the-
ory, Bayesian networks, or agent-based models that allow
dedicated simulations. In summary, Linkov etal. (2018) is a
helpful paper for classifying resilience analytics; according
to this article, Tier I and Tier II can utilise both qualitative
and quantitative methods, while Tier III analysis is mainly
based on a quantitative approach.
Under current challenging times, where some CIs have
become even more critical for our society, the study of resil-
ience acquires a strategical role for decision-making, at any
modelling tier.Research on each tier progressed widely over
recent years, thanks to many scholars who offered multiple
opportunities for improving the management of CIs embrac-
inga resilience-oriented research dimension. Several reviews
are available in literature about resilience, but still restricted
to specific CIs (e.g. energy systems (Gasser etal. 2019),
cyber CIs (Mohebbi etal. 2020), remote sensing (Veettil
etal. 2020), supply chain (Golan etal. 2020)), or with an
explicit focus on quantification (see e.g. Hosseini etal. 2016;
Ingrisch and Bahn 2018; Shuang etal. 2019; Rehana etal.
2020)). However, to the best of our knowledge, there is no
explicit up-to-date literaturereview related to qualitative (or
semi-quantitative) methodologies as developed for managing
resilience of CIs.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

343Environment Systems and Decisions (2021) 41:341–376
1 3
The purpose of this study is to complement avail-
able literature in the context of CI resilience, through a
systematic review on qualitative aspects, i.e. reviewing
approaches explicitly focused on Tier I methods, or used
as entry points for Tier II approaches (Linkov etal. 2018).
This paper reviews current literature on qualitative
approaches used to manage and improve resilience of CIs,
through firstly bibliometric findings and then via an in-
depth descriptive content analysis. Our primary aim leads
to explore and in-depth understand the diverse approaches
available to support decision-making for CIs, and draw com-
mon concepts that emerge in case of the application of quali-
tative approaches. The research describes the hypotheses,
methodologies, and results of relevant papers, extracting
relevant knowledge from thematic full-text analyses which
is intended to support scholars dealing with CI’s resilience
mainly from a Tier I perspective (Linkov etal. 2018). While
the research inherently focuses on Tier I approaches, it also
partly deals with Tier II approaches, at least in those cases
where the semi-quantitative dimension of analysis is explic-
itly combined with qualitative data sources.
This research has been conducted taking Scopus as a
reference database, and then following a systematic review
process based on the well-established PRISMA (Preferred
Reporting Items for Systematic Reviews and Meta-Analyses)
framework (Moher etal. 2009b).
The remainder of the paper is organised as follows.
Section2 details our research method; Sect.3 illustrates
main bibliometric findings; Sect.4 discusses in detail each
included paper; Sect.5 offers points for discussion and
insights from the literature review. Finally, conclusions on
this work are summarised in Sect.6.
2 Methodology
The systematic approach followed in this research relies on
PRISMA (Moher etal. 2009a) and consists of 5 phases,
as sketched in Fig.1, conducted by authors of the article
through the usage of MS Excel and Mendeley.
Fig. 1 Literature search strategy
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

344 Environment Systems and Decisions (2021) 41:341–376
1 3
2.1 STEP 1: setting thesearch query
The first step of the review defined the scope of the search
query. A query has been progressively refined to include a
large set of contributions potentially linked to qualitative
approaches for CIs’ resilience. The final search query imple-
mented in Scopus query system included every paper mak-
ing use in title, abstract, or keywords of ‘resilience AND
“critical infrastructure”’, as well as any of the following
lemmas: interview, focus group, workshop, questionnaire,
surveys, or the term qualitative. The selection of terms was
inspired by (Patton 2002).
A specific inclusion criterion refers to papers indexed up
to March 2020. The query has been limited to the Scopus
database, since it is recognised as the biggest repository
of peer-reviewed literature with over 5000 publishers and
over 71 million records fairly balanced among technical and
social aspects of science (Elsevier 2018).
Following an inductive process on abstract reading, the
dataset has been enlarged to include other documents as
referred to EU/US-funded projects mentioned in the full
text of papers as assessed for eligibility, running an addi-
tional query with for title, abstract, or keywords including
‘resilience AND “critical infrastructure”’ AND the name of
the project (e.g. CIRMin, DARWIN, IMPROVER, NEXUS,
Resilience Shift, SMART Measure Resilience) (Adini etal.
2017). The two queries return, respectively, 132 and 16
documents.
2.2 STEP 2: refinement ofdataset
Overall, the queries returned 148 items matching the search
criteria. As a first step, a preliminary data refinement on the
analysis of titles has been conducted to eliminate duplicates:
3 duplicates have been identified and deleted, meaning that
145 documents progressed to the next phases.
2.3 STEP 3: screening
The output of this step consisted of selecting documents
relevant for the scope of the work. Each abstract has been
screened and several documents were excluded because they
were just mentioning the word “survey” as a synonymous
for “review” or “study”, not implying the actual usage of any
qualitative research method for managing resilience of CIs.
After this screening phase, 32 papers have been excluded,
meaning that 113 documents require full-text assessment
for inclusions. The analysis has been conducted by two
researchers independently (RC, RP), leading to an agreement
ratio of over 95%. Ambiguous situations have been solved
conservatively, keeping those papers for full-text reading.
2.4 STEP 4: eligibility assessment
This step represented the analysis of documents following
full-text reading. Through this step, 15 documents have been
excluded, being considered outside the scope of the analy-
sis, and showed a set of 98 documents to represent the final
dataset. The analysis has been conducted by two researchers
independently (RC, RP, over 95% agreement). The few inco-
herencies have been solved via a group discussion involving
also the third researcher (GDG). While reviewing the full
text, additional papers have been added to the dataset as
explained in Step 1 (as included in the 113 documents).
2.5 STEP 5: analysis ofpapers included inthefinal
dataset
The last step consisted of analysing the full text of the 98
papers. This analysis also aimed to refine the meta-data
on the paper to ensure meaningful bibliometric analyses,
and to follow an ad hoc protocol for systematic knowledge
elicitation. The protocol included aim of the paper; domain
being investigated; type of qualitative approach; causes and
threats; method or model used; metrics/indexes used or
defined. The dataset has been split between the researchers
and cross-checks have been defined to ensure higher consist-
ency. The analysis followed a deductive perspective with
an unconstrained categorisation: iteratively adding different
categories was considered possible within the bounds of the
protocols, in line with inductive content analysis (Elo and
Kyngäs 2008).
On the results of this categorisation, documents have been
presented following different dimensions of resilience for
CIs to facilitate the narrative dimension of the document
(see Sect.4). In practical terms, the adopted logic led to
identify four resilience dimensions, namely “techno-cen-
tric”, “organisational”, “community”, and “urban” within
which different qualitative approaches were utilised, several
threats or hazards were identified, and many issues were
discussed in order to improve resilience.
3 Bibliometric ndings
In this section, some bibliometric findings are reported.
Figure2 shows the evolution of the examined papers over
years, comparing as well open-access vs subscription-access
papers. In this context, the increase of open-access docu-
ments in recent years is noteworthy, even though still below
40%.
Figure3 considers the document types of the dataset and
shows a balanced proportion between conference proceed-
ings and journal articles. This means that when looking at
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

345Environment Systems and Decisions (2021) 41:341–376
1 3
Fig. 2 Evolution and access type of documents over years
Fig. 3 Document types in the
dataset
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

346 Environment Systems and Decisions (2021) 41:341–376
1 3
CIs’ resilience from a qualitative perspective, both journals
and conferences seem to be relevant publication fora. It is
also worth mentioning that many of the proceedings are
related to research conducted undervarious stages of funded
projects, proving that conferences represent a preferable way
for disseminating intermediateresults.
More specifically, the publications are spread across dif-
ferent journal and conferences with a few sources including
more than three papers, i.e. International Journal of Critical
Infrastructure Protection (4 documents), International Jour-
nal of Disaster Risk Reduction (4 documents), International
Journal of Critical Infrastructures (3 documents), Reliability
Engineering and System Safety (3 documents), Sustainabil-
ity (3 documents). On the other hand, the most recurrent
conferences are European Safety and Reliability—ESREL
Conferences (8 documents), Information Systems for Cri-
sis Response and Management—ISCRAM Conferences (7
documents), and International Development Research Cen-
tre—IDRC Conferences (3 documents).
More in detail, additional reflections can be proposed
about the content of the papers in the dataset. Figure4
reveals that more than half of them consist of interviews
(about 30%) and questionnaires (about 28%). It is also rel-
evant the percentage of approaches consisting of workshops
(15%) or focus groups (10%), methods where experts or
stakeholders can share their visions, opinions, ideas (focus
group) or can find solutions and reach conclusions (work-
shops). Moreover, specific knowledge elicitation methods
such as the Analytic Hierarchy Process (AHP) or the Del-
phi method have been utilised for prioritising concepts,
ideas, attributes, guidelines, etc. These approaches remain
of interest for qualitative research because of their knowl-
edge elicitation process, which is at Tier I, qualitative by
nature. Furthermore, simulation games have been discussed
about their potential in engaging people and understanding
human behaviours. Other sources refer to social networking,
or some type of observation (naturalistic), or ethnographic
research.
Besides the overall percentages shown in Fig.4, Fig.5
shows the number of different qualitative approaches utilised
Fig. 4 Percentage of used qualitative approach
Fig. 5 Number of different qualitative approaches utilised by each
paper
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

347Environment Systems and Decisions (2021) 41:341–376
1 3
among each paper. Although most papers (55%) apply a
single approach, it is worth mentioning that the other half
of papers utilises more than one approach, with about 20%
of the documents using 3, 4, or event 5 techniques jointly.
This observation might indicate the benefit of mixed-
method research, and for the emergingneed of knowledge
triangulation.
Furthermore, the histograms in Fig.6 show the different
domains covered in the papers, combined with the typology of
qualitative approaches utilised. The most frequent domain is
related to energy systems (frequently ascribed to power grids),
followed by transportation systems (e.g. roads, motorways,
railroads, or ports). It can be also highlighted a non-negligible
number of “not specified” documents because many papers
do not refer to a domain or CI and do not contextualise their
work, which therefore remains applicable to multiple domains.
Other relevant domains are linked to community and urban
infrastructures. Other CIs appearing in the dataset are supply
chain management and water facilities.
Lastly, a combined view on the results is proposed in
Table1, which summarises the previous results: by rows, the
figure offers the total of papers per domain, whereas, by col-
umns, the total of papers per approach.
4 Descriptive ndings
This section proposes four different dimensions relevant
for qualitative research applied to CIs, in line with recent
research (Labaka etal. 2014). These dimensions encom-
pass diverse, sometimes complementary aspects of resil-
ience, i.e. techno-centric, organisational, community, and
Fig. 6 Different domains combined with different qualitative approaches utilised
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

348 Environment Systems and Decisions (2021) 41:341–376
1 3
urban. The following sections are intended to provide a
synthesis of reviewed studies, highlighting methodo-
logical steps, and obtained results. While these dimen-
sions can be sometime overlapping, they are considered
helpful to support the narrative understanding of the
reviewedapproaches, as further detailed in theAppendix.
4.1 Techno‑centric resilience
This paragraph deals with papers encompassing a research
dimension that follows primarily the technical aspects of
resilience with respect to several complex systems from civil
infrastructures towards supply chains.
Among the wide range of CIs, e.g. water supply systems,
sewerage, transportation infrastructure, power grids, and tel-
ecommunications networks, Sircar etal. (2013) focus on UK
energy and transport infrastructures, via a set of scenario-
episodes to support key stakeholders in the examination of
their resilience. After an analysis conducted through the
Political, Economic, Social, Technological, Environmental,
Legal (PESTEL) lenses, tailored crisis episodes were added
to stress test the scenarios and to operate in a not “business
as usual” situation and a first set of key stakeholders from
local resilience forums, engineering firms, UK Government
departments, academia was interviewed to elaborate two tai-
lored episodes related to flooding and terrorist attack, and
other stakeholders were then invited to participate in focus
groups, in order to consider strengths and weaknesses of the
“2050 world”, guided by the three generations of resilience
(cf. introduction).
When considering strengths and weaknesses of a region,
it becomes crucial to focus on CIs’ spatial planning at differ-
ent administrative levels as proved by a research conducted
in Germany (Riegel 2014). Spatial planning considers and
integrates the needs of multiple stakeholders, through the
“mutual feedback” principle, which is an iterative top-down/
bottom-up process to develop visions and principles of spa-
tial development, to examine the compatibility of proposal
for plans and projects, and to monitor impacts of realised
developments on natural assets. A survey among all 111
German regional planning authorities examined the current
perception of CI protection to increase their resilience. Even
though the topic was generally known among the respond-
ents, the survey showed how regions were not prepared to
apply a systematic CI protection strategy as publicised by
national security authorities or by European institutions.
In Canada, Valiquette L’Heureux and Therrien (2013)
aim at assessing the main tendencies of the different dimen-
sions of resilience within three major CI networks, i.e. trans-
portation, energy, and telecommunication through a survey
among hundreds of CI managers from governmental, com-
munity, and private-sector organisations who were identified
screening official websites, annual reports, and official gov-
ernment information. The survey examined various aspects
of internal and external resilience adaptative management,
emergency preparedness, and post-crisis organisational
learning. Resilience was assessed through a wide range of
indicators using a Likert scale1 and the general attributes
were analysed to seek statistically significant differences
between groups of organisations by sector, type, and size
Table 1 Overview of results
1 A Likert scale is a psychometric scale commonly used to scaling
responses in a survey research. When responding to a Likert item,
respondents specify their level of agreement or disagreement on a
symmetric agree–disagree scale for a series of statements. Thus, the
range captures the intensity of their feelings for a given item.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

349Environment Systems and Decisions (2021) 41:341–376
1 3
(using Chi-square and Kruskal–Wallis). The empirical study
provides insights into the main challenges and barriers to
resilience: projective intelligence, decision-making abilities,
strategic positioning, monitoring of risks, crisis management
informational designs.
These challenges spans over technology-based, human
resource-based, procedure-based strategies to be imple-
mented for infrastructure resilience. The selection of the
most adequate strategy becomes a multi-dimensional
problem, as investigated through multicriteria decision-
making techniques such as the AHP. At a higher abstrac-
tion level, the results obtained through a pool of experts
from theRomanian Land Forces Academy suggested that
the best strategy to increase the resilience of a CI refers to
human resources, relying on an intensive training system
of employees, in order to empower them within the sys-
tem (Badea etal. 2017). Similarly, another research focused
on the analytic network process (ANP), a variant of AHP,
to prioritise factors affecting port resilience involving 11
experts including government officials, planners, and schol-
ars. The consistency of ANP answers was ensured using the
Delphi method leading to three major factors, i.e. ground
access system, travel time, shipping route density (Hsieh
etal. 2014).
Münzberg etal. (2017) aim at supporting building com-
munity resilience by introducing a spatial–temporal vul-
nerability assessment, based on various indicators, which
enables decision makers (i.e. crisis management groups,
management authorities, CI providers) to enhance their ini-
tial understanding of the impacts of a power outage. In this
paper, the Delphi has been also adopted to ensure a con-
sistent definition of an indicator for power outages, namely
the Relevance Criticality Weight, as assessed by decision
makers coming from different cities and disaster manage-
ment authorities. This indicator has been combined with the
Coping Capacity Resource (CCR) estimating the capacities
of a CI to continue the activities during a power outage for a
short time and finally integrated into a vulnerability model
based on Monte Carlo simulation.
Other papers usually refer to Critical Infrastructures Key
Resources (CIKRs) to describe infrastructure systems. In
Vugrin etal. (2010b), the U.S. Sandia National Laborato-
ries formulates a unifying framework which can be applied
to all the CIKR systems identified by the U.S. Department
of Homeland Security to study their resilience, explicitly
considering the cost of recovery efforts. In particular, the
discussed framework consists of an approach for quantitative
measurement of resilience costs, and a qualitative method
to evaluate features that determine systems’ resilience. The
quantitative measurement involves two components: sys-
temic impact (SI) which is defined as the difference between
a targeted system performance level and an actual system
performance after the disrupting event, and total recovery
effect (TRE) which is the amount of the resources expended
during recovery processes following the disruption. The
framework also introduces a qualitative analysis, which can
be performed to explain the quantitative measurement or can
take the place of quantitative results when data are not avail-
able. This analysis utilises three system capacities to explain
how inherent properties of a system can determine system
resilience, by reducing SI and TRE. These capacities are
absorptive, adaptive, and restorative. Better resilience sys-
tems can then be designed by developing resilience enhance-
ment features that improve one or more of these capacities:
for example, storage is an enhancement feature for absorp-
tive capacity; emergency generators enhance the adaptive
capacity; monitoring systems for break detection in power
grids increase restorative capacity.
In Vugrin etal. (2010a), the authors utilise aprevious
framework elaborated in Vugrin etal. (2010b), to analyse the
resilience of a particular CIKR like the U.S. petrochemical
supply chain during hurricane disruptions. In particular, the
Sandia centre performed a comparative analysis by simulat-
ing disruptions with the National Infrastructure and Simula-
tion Analysis Center (NISAC) Petrochemical Supply Chain
Model. This latter consists of two primary components: the
chemical data model (CDM), a database of domestic and
foreign chemical plants, chemical productions, commodity
flows, and chemical infrastructures, and the NISAC Agent-
Based Laboratory for Economics (N-ABLE) microeco-
nomics simulation tool that performs supply chain analy-
sis. Within the N-ABLE, each agent-based enterprise firm
is composed by supervisors, production workers, sellers,
buyers, and strategic planners. When applied to CDM, the
N-ABLE can predict economic impacts and loss estimates
to be used for measuring the systemic impact of a petro-
chemical supply chain affected by a hurricane. In addition,
N-ABLE can help to estimate the costs associated with the
recovery and adaptation processes, which are crucial for
calculating the TRE in a resilience analysis. The framework
allows to determine resilient costs starting from the calcu-
lated values of SI and TRE, but it also allows qualitative
assessments of attributes that enhance the supply chain’s
absorptive, adaptive, and restorative capacities.
In Petit etal. (2012), a Resilience Index (RI), useful for
assessing CIKR facilities, has beendeveloped. The RI can
assist owners/operators to compare their facilities to similar
sectors sites and can help them to make better risk-based
decisions. The data were collected through a questionnaire
of more than 1.500 data points per facility (i.e. commercial
buildings, electrical substations, transportation assets, and
dams). After a Quality Assurance review process, the RI
was developed through an aggregation of data collected into
four levels of information, by using multi-attribute theory,
an approach that helped to decompose resilience into its
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

350 Environment Systems and Decisions (2021) 41:341–376
1 3
individual attributes and then organised them into an organi-
sational tree.
Trucco and Ward (2011) study the propagation mecha-
nism and consequent dependencies of another CIKR, the
Fast-Moving Consumer Goods supply chain of perishable
goods in Italy. Fuzzy Cognitive Maps (FCMs) were utilised
defining 13 concepts that reflected attribute, characteristics,
qualities and paths of the modelled system, and tracing edges
that represented the interconnections between concepts. The
three independent FCMs for different temporal frames (i.e.
after one day, four days, and three weeks) were implemented
recurring to expert elicitation. Therefore, a specific ques-
tionnaire was distributed at least one week before, conduct-
ing a single face-to-face interview to define concepts and
to weight the relationships in the adjacency matrices of the
FCM approach. Regarding the case study, the results show
that the most crucial concepts are the staff availability, the
data network, and the electricity supply.
Also Seppänen etal. (2018) focus on the Infrastructure
Failure Interdependencies (IFIs) developing a qualitative
method, based on experts’ knowledge, for identifying and
describing the potential sector-specific and cross-sector IFIs
of CIs. The method, consisting of four phases (i.e. prepara-
tion, material collection, analysis, results), was developed
and applied with the Finnish regional preparedness commit-
tees. In the first phase, the planning group defined the threat
scenario. Then, company representatives from the studied
CIs (i.e. electricity distribution, telecommunications, and IT
infrastructures) were invited, together with representatives
from other CI fields (healthcare and government agencies)
in order to identify the broader interdependencies and to
elaborate a scenario with a severe storm and a pandemic
influenza. The second and third phase, respectively, material
collection, and analysis, were conducted in iterative man-
ner, performing a cycle which has been continued through
several workshops with up to fifty experts, until the col-
lected and structured material was satisfactory. In this case,
the difference between the discussed method and the Delphi
method is related to the anonymity of the experts: in the Del-
phi method it guarantees freedom of opinions and minimises
the effect of dominant characters, whereas the facilitated
face-to-face approach of the discussed method promotes the
cooperation among the actors who would cooperate in the
crisis situation. The discussions were recorded, and the col-
lected information was first structured in mind maps and
then the resulting documentation was used in the creation
of an ISFI matrix that describes the causal intra-sector and
inter-sector interconnections. Finally, the results were pre-
sented in system diagrams, a form that makes it easier to
understand the failure interfaces between the CIs and the
chains of dependencies.
As regards IFIs, Bloomfield etal. (2017) present a meth-
odology called Preliminary Interdependence Analysis (PIA),
useful for building, refining, and analysing models of inter-
dependent complex CIs. This method starts with a quali-
tative phase during which the scenario must be accurately
defined, and subject matter experts are asked to provide data
and information which will be utilised to parametrise the
selected model. Then, via a set of focused refinements, PIA
may evolve into a quantitative method for assessing the risk
due to interdependencies between CIs.
In order to analyse IFIs of specific supply chains and
demand nodes, Pfeiffer etal. (2017) present a Grassroots
Infrastructure Dependency Model (GRID-M). This model
helps public safety officials to make better approxima-
tions about disruptions by using pre-incident survey data.
GRID-M can also help the officials to gain near-real-time
situational awareness on the physical state of a node by
using the damage assessment application. GRID-M dis-
plays all outputs within a Geographic Information Sys-
tems environment with additional prepopulated layers such
as real-time traffic and demographic information of the
affected communities. This information can support the
prioritisation of infrastructures in planning, exercise simu-
lations, real-word situations, and restoration activities.
Beheshtian etal. (2017) use information extracted
from geodatabase, applying the concept of network resil-
ience to the motor-fuel supply chain (FSC) management
in New York City, affected by hurricanes. The authors
use a stochastic bi-stage optimisation model to analyse
practical strategies for the allocation of resources, called
Resilience-enhancing strategies (RES). For modelling the
transportation network, an application program interface
(API) implemented in Visual Basic was used to extract
data from the ESRI geodatabase and translate them to
a graph network consisting of nodes and arcs; then the
impact of hurricanes were modelled by considering three
characteristics: type and frequency; vulnerable loca-
tions; expected flooding intensity. The authors defined a
context-specific resilience index referred to the average
demand met across the gas stations, along with a vari-
able representing the unmet demand rate (UDR), which
allowed the authors to solve the model for the minimum
expected value of the UDR, minimising the FSC’s overall
inoperability and fuel distribution costs. Several experi-
ments were conducted, varying policy scenarios, physical
improvements, and budget limitations. The results showed
how the three pillars (absorption, adaptation, and restora-
tion) of the mentioned RES are interlinked and how dif-
ferent combinations of investment scenarios may provide
different levels of resilience.
As discussed earlier, several threats can undermine sys-
tems included in the techno-centric dimension of resilience.
Cutts etal. (2017), for example, deal with seismic risk. Dur-
ing the 2014 Cascadia Earthquake Readiness Workshop in
Washington, the attendees were coordinated by facilitators
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

351Environment Systems and Decisions (2021) 41:341–376
1 3
in order to identify the most important infrastructure, related
to three thematic areas (ports and waterways, critical energy
infrastructures, and emergency management) exposed to the
risk of an earthquake or tsunami and to brainstorm poten-
tial solutions. The participants produced a list of takeaways,
observations, recommendations, and highest priority infra-
structure improvements, to cope with possible earthquakes
or tsunami and to increase infrastructure resilience.
Feldpausch-Parker etal. (2018) present a case study
involving three different States (New York, Massachusetts,
and Vermont) of the U.S. subject to the Sandy storm of
2012. The aim of this study is to compare how climate and
energy are being linked in smart grid planning and develop-
ment. Discussions from 22 focus groups included 3–8 rep-
resentatives of a single organisation were recorded, profes-
sionally transcribed, and coded with NVivo 10.0 software
to characterise discursive patterns and linkages between cli-
mate change and energy by analysing text focused on both
climate mitigation and climate adaptation. Data analysis was
based on the principles of grounded theory, which offers
an internally logical set of techniques for collecting and
analysing qualitative data. The triangulation of knowledge,
via clarification questions and informant validations, sup-
ported a high level of accuracy, through a continual move-
ment between data collection and analysis. Comparatively,
the study found that the balance between the conversations
about adaptation versus mitigation was associated with the
severity of the storm’s impact in each of the three states.
Differences between stakeholders were also examined,
revealing that energy system operation experts (i.e. utilities)
preferred adaptation measures to mitigation ones, while the
opposite was for other energy system actors (i.e. authorities,
researchers).
Matsika etal. (2016) introduce terrorism, as a man-made
threat, requiring the development of risk assessment specifi-
cations for the RAMPART (Risk AssessMent toolbox for the
Prevention and reduction of terrorist Attacks on metRo and
light-rail criTical infrastructures) project. The paper focuses
on the incorporating resilience in the risk assessment tech-
niques. During a workshop with experts, six risk/resilience
assessment methodologies used in Canada, EU, and USA
were selected for detailed analysis and six key factors that
constitute a robust risk assessment methodology (RAM)
were identified. About one of these factors, “qualitative vs.
quantitative approach”, the paper argues that a risk assess-
ment in the public transport security should be done firstly
in a qualitative way because of the lack of statistics (espe-
cially concerning terrorist incidents), necessary to perform
a quantitative assessment based on mathematical formulas
and calculations.
McBurnett etal. (2018) aim at demonstrating the effec-
tiveness of simulation gaming for developing systems-think-
ing skills which are critical to understand the complex nature
of infrastructure management. Using Vensim software, the
authors implemented a system dynamic model to represent,
through a simulation game called LA Water Game, the
problem of maintaining the quality of Los Angeles water
distribution infrastructure over a 75-year period. The game
was performed in 16 workshops of over 200 participants
consisting of undergraduate and graduate students, faculty,
and active-duty military personnel. Data collection, through
participant observations and debriefing interviews, showed
the success of this particular teaching method: the players,
with cognitive and effective engagement and intrinsic moti-
vation, were able to identify the interdependencies between
game variables, the non-linear nature of infrastructure dete-
rioration, the stochastic emergency breaks, and the reinforc-
ing loops within the game. Therefore, simulation gaming
can be considered successful for training resilience skills.
Among techno-centric resilience, specific attention has
been devoted to those systems with a large Information
Technology dimension (Haass etal. 2015). For example,
Johnsen etal. (2009) aim at assessing the safety, security,
and resilience of Information and Communication Technol-
ogy (ICT) and Supervisory Control And Data Acquisition
(SCADA) systems used in the Norwegian oil and gas indus-
try through surveying 46 Norwegian offshore oil and gas
installations. In the survey, Yes/No answers were possible,
but respondents could also provide free-form comments.
The survey and the subsequent discussions provided some
important key results: poor risk awareness, lack of consistent
safety/security guidelines; absence of systematic knowledge
sharing, poor scenario training, and emergency prepared-
ness; lack of systems certification; lack of network barriers;
inadequate deployment of patches.
In Landegren etal. (2018), a research is conducted to
investigate the extent to which a simulation-based approach
can be applied to large socio-technical IT networks. These
networks (i.e. a municipal IT network and the SCADA sys-
tem of a wastewater network) are modelled using Monte
Carlo simulation to understand their recovery times. The
utilised model is hybrid and considers the technical network,
represented with graph theory, as well as the repair system,
represented with a queuing model with four types of enti-
ties: jobs, queues, stock, and repairers. Data were collected
through interviews with focus groups and through docu-
ment analysis to gather information about fault modes, their
relative probabilities, repair times, and resources needed
for repair. The results were evaluated through interviews
to check trustworthiness, usefulness, ability to increase
system resilience, improvement, and generalisation pos-
sibilities. The authors conclude that this approach can be
extended also including software and dependency failures
to hardware and operator failures and considering invest-
ments, price of network, and repair resources as suggested
by system experts’ feedbacks.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

352 Environment Systems and Decisions (2021) 41:341–376
1 3
Dealing with Industrial Control Systems (ICS), Lykou
etal. (2019) provide a review and analysis of available
cybersecurity Self-Assessment tools, which can be utilised
by ICT owners and CI operators. In order to identify weak-
nesses and cyber vulnerabilities and to establish targets for
continuing improvement, these tools provide an assessment
for many purposes: (1) they can be used by management
teams to gain a general understanding of security assurance
and make informed decisions; (2) they can be utilised as a
guide to assess the status of security for a system; (3) they
can enhance and support employees’ security awareness; (4)
they may be used to fulfil reporting requirements, to prepare
for audits or to identify resource needs.
Again on cyber vulnerabilities, Rajamaki etal. (2018)
present the work in progress in developing cybersecurity
and training in healthcare facilities. In particular, they focus
on the Proactive Resilience Educational Framework (Prosil-
ience EF), whose goal is to reduce cybersecurity vulnerabili-
ties and exposure in hospitals and to make hospitals more
resilient for cyber-attacks by identifying potential cyber-
security vulnerabilities and cyber threats; describing best
practices and training for targeted and untargeted attacks;
raising awareness of security and privacy of healthcare staff;
developing training schemes on cybersecurity in hospitals
for different categories of staff. The framework launches an
iterative process of awareness and training development with
relevant stakeholders (healthcare authorities, end users-hos-
pitals, industry members, cybersecurity training providers),
evaluating the framework via joint exercises and workshops.
Bernroider etal. (2016) aim at developing and test-
ing a framework that holistically measures the quality of
Information Security Management (ISM) in the context of
cybersecurity and allows for comparative assessments of
organisations in CI sectors, taking ideas from the Balanced
Scorecards (BSC) measurement system. Following a design
science approach, workshops, cyclic refinements of the
instrument, pre-tests, and framework evaluation within 30
critical infrastructure organisations were conducted, involv-
ing subject experts and Chief Security Information Officers
(CISOs) as the main stakeholders from the information secu-
rity domain. The quantitative evaluation served to bench-
mark, but to complement this assessment by explaining and
interpreting scores, open qualitative questions were neces-
sary to capture the special context situations. The authors
argue that the scorecards, used as quantitative estimators,
alone do not portray the complete security status.
Student etal. (2018) present an indirect measurement
method to improve preventative maintenance and increase
resilience of CIs as communication networks or electrical
transmission infrastructure. The authors start from consid-
ering that many instances of failed infrastructures are not
immediately discovered by CI operators or owners, but
rather, by the public who report the problems with rela-
tive and qualitative descriptions. The idea of the paper is
to enhance these qualitative descriptions by using a human-
in-the loop algorithm, derived from the concept of Agile
IoT, providing quantitative measurements (through action-
able intelligence from the general public who utilises own
devices or by using already deployed sensors, like traffic
cameras) which could help prioritising repairs to reduce the
likelihood of failures and to better allocate time and crew
resources.
In order to analyse resilience of a networked system that
depends from ICT, like the Internet, Garcia etal. (2010)
present a hardware-based emulation (emulation testbed).
The authors study the fidelity of the emulation by compar-
ing experimental results between two different emulation
configurations against the reference real configuration.
While confirming the efficiency and similitude of emulation
testbed, the authors recommend that the interpretation of
experimental results should not be based on absolute num-
bers, which are hardware dependent, but rather on system
behaviour and trends. This means that emulations are rep-
resentative of real systems from a qualitative point of view,
in terms of emerging behaviour, rather than a quantitative
perspective, in terms of absolute performance.
Reilly etal. (2018) explore how CI resilience can be
enhanced through the information-sharing practices of its
operators during each stage of an incident (mitigation, pre-
paredness, response, recovery). Effective crisis communi-
cation should manage information through the collection
and dissemination of crisis-related information, while also
managing its meaning to persuade the public in the hope
that they will plan for and respond appropriately to risks
and threats. Interviews, focus groups, and consultations were
conducted with 31 relevant stakeholders across Europe. CI
and emergency management professionals were asked about
current communication strategies, whether digital media
had been incorporated, how traditional and digital media
were used together, what audience they hope to reach using
different platforms. Interviews with journalists focused on
their experiences of social media in detecting and verifying
incidents, and whether they addressed ethical and legal chal-
lenges, using social media in relation to emergencies. The
public indeed expects CI operators to keep them informed
about progresses on restoration of CI infrastructures and
answer rapidly to queries, as proven by a large questionnaire-
based study (N = 403) and several semi-structured interviews
with multiple stakeholders (Petersen etal. 2020a).
Finally, Gheorghe etal. (2018) deal with the resilience of
particular ICT systems which are subject and object of crisis
and emergency situation management: Critical Space Infra-
structure (CSI), like satellites orbiting the Earth, may com-
promise, with their failures, the capacity of competent actors
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

353Environment Systems and Decisions (2021) 41:341–376
1 3
to manage the crisis. As regards the space governance, the
authors argue that space actors must agree on key resilience
measures, implement them and enforce them unilaterally on
third parties, such as corporations or other states, despite the
lack of jurisdiction due to the international character of the
space environment. Moreover, they foster the cooperation
among space actors and the interoperability of systems, to
ensure timely access to various resources or to substitute one
space system for another in terms of short-term provisioning
of critical space services.
4.2 Organisational resilience
When dealing with CIs, there are usually non-negligible
organisational components to be considered in the analysis,
i.e. components which embrace a social dimensions comple-
mentary to the more technical aspects of a system. This sec-
tion emphasises on the contributions which make an explicit
effort to understand, model, and measure the organisational
resilience for CIs (Pathirage 2010).
Referring to construction organisations, Sapeciay etal.
(2019) identify strategic resilience indicators through a tri-
angulation analysis of literature review, questionnaire sur-
vey, and in-depth interviews. Firstly, an extended literature
review revealed 72 indicators related to organisational resil-
ience, among which 27 indicators were selected according to
their frequencies, by means of Nvivo software. Secondly, an
electronic questionnaire was undertaken to elicit the views of
construction practitioners and resilience experts in New Zea-
land. The questionnaire was composed by closed-ended and
open-ended questions and aimed at identifying the profile
of respondents and organisations, at getting an overall view
of their organisational resilience practice, and at finding
out respondents’ opinions on key resilience indicators and
ranking the latter ones. Thirdly, semi-structured interviews
were conducted with 23 construction practitioners from
construction client and contractor organisations (i.e. mainly
project senior executives). The transcript interviews were
qualitative analysed by using Nvivo software, searching for
key themes. Triangulation of data improved reliability, by
reducing judgmental bias, and supporting validation of the
most significant indicators through interviews. Most experts
agreed that the top five indicators for assessing resilience
of the construction industry are, in rank order: leadership,
planning strategies, internal resources, decision-making, and
staff engagement.
Again in New Zealand, Brown etal. (2017) present a
method to benchmark the organisational resilience of CIs’
providers, i.e. the Benchmark Resilience Tool, based on 13
indicators of resilience. Respondents were asked to use a
Likert scale to rate how much they agreed with each state-
ment and had to answer other demographic and preparedness
questions (gender, age, organisation use of emergency plan,
etc.). The study assesses relative resilience strengths and
weaknesses of CI organisations, finding the “effective part-
nership” as the strongest indicator whereas “breaking silos”
(i.e. breaking barriers to the sharing of ideas and skills) and
“stress testing plans” (i.e. the capacity to actively practice
emergency, crises, or business continuity plans) as the weak-
est ones. Findings also show that senior managers have much
more positive perceptions of the resilience of their organisa-
tions compared with other staff workers.
Conducted by a private company, the study describedin
(Mendonça and Wallace 2015) provides a critique of stand-
ardised factors for organisational resilience, analysing the
restoration process of electric power in Manhattan after 11
September 2001. Qualitative and quantitative data were
collected to support triangulation of observations. Inter-
nal reports from the company and articles in the popular
press provided also contextual information on the compa-
ny’s response, while data regarding the timing, cues, and
key decisions were provided through the Critical Decision
Method (CDM). Data on participants were collected by
questionnaires and data on the performance of technologi-
cal systems (i.e. timing and location of restoration activi-
ties) were collected through logs of the performed work. The
Woods factors (i.e. buffering capacity, flexibility/stiffness,
margin, tolerance, cross-scale interactions) were analysed
and the study proposed another factor—boundary-spanning
capability—which may help to understand how cross-
organisational linkages can help to determine organisational
resilience.
Adini etal. (2017) present a work which is part of the
European research project DARWIN, whose aim is improv-
ing responses to crises affecting CIs or social structures by
developing resilience management guidelines (Cedrini etal.
2018). The first phase of DARWIN Project was to identify
concepts, practices, and approaches of resilience manage-
ment through a Systematic Literature Review and interviews
with relevant stakeholders involved in crisis management.
A final list of 56 concepts, practices, and approaches was
compiled, phrased in a uniform mode, and incorporated into
a computerised survey tool, using Survey Monkey. A 2-cycle
modified Delphi process was conducted to decide which
items of the list should have been included in the resilience
management guidelines. Reviewing the items that reached
the highest scores reveals that they comprise at least one of
the three common elements frequently found in definitions
of resilience: the need of flexibility, adjustability, and adapt-
ability; the need for sharing and understanding for the actors
involved; the focusing on CIs, considered vital for organisa-
tions and communities.
Förster etal. (2019) introduce another ongoing work
within the scope of the DARWIN project: the resilience
management guidelines. These latter, applied to the Air
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

354 Environment Systems and Decisions (2021) 41:341–376
1 3
Traffic Management (ATM) and the healthcare domain, are
intended to support organisations in critical situations and
evaluate their effectiveness by attribute- and performance-
based approaches, in line with, respectively, engineering
resilience and resilience engineering (i.e. resilience in tech-
nical systems and resilience in complex socio-technical sys-
tems). Simulation trials, performed by operational experts
in the form of gaming sessions, assessed the effectiveness
of the adopted operational procedures. Additional scenarios
were developed to emphasise the organisational interdepend-
encies between ATM and healthcare CIs. Finally, a debrief-
ing was conducted to assess the performance of the system,
indicating possible bottlenecks and identifying brittleness.
Field and Look (2018) aim at assessing organisations
resilience performance, providing a framework based on a
value model, against which each type of CI was compara-
tively assessed. A set of interviews was conducted to 50
industry experts from a variety of CIs’ sectors with direct
responsibility for the assess risk and resilience. Each infra-
structure type was reviewed in terms of value expectations
from its various stakeholders’ groups (i.e. end user/customer
to investors, suppliers, and constituent organisations). The
evidence for these assessments was based on the results of
the interviews in addition to performance metrics published
by government authorities, regulators, consumer organi-
sations. Key findings were the lack of incentives to work
proactively with other providers; the focus on response and
recovery instead of proactive mitigation measures; the non-
measurement of the impacts of disruption on UK society.
Based on the INTACT project co-founded by the Euro-
pean Union, Räikkönen etal. (2017) focus on Risk Man-
agement Measures (RMM). In particular, the study con-
tributes to the value creation of RMM, which is crucial for
decision-making and for the development of strategies to
prevent or reduce the impacts of extreme weather events. A
case study, regarding the electricity distribution network in
Finland, was presented to validate the proposed approach.
The stakeholder value of RMM during the entire life cycle of
CI was assessed, by applying AHP to the considerations and
evaluations made by 18 experts from the DSO (Distribution
System Operator), the regional rescue service, the city of
Tampere, the Finnish Red Cross, and an ICT company. The
AHP method ranked the following value criteria for compar-
ing the RMMs: benefits of the RMMs in economic, environ-
mental, and social terms; impact of the RMMs on reliability,
availability, and maintainability of electricity distribution
network; life-cycle cost (investment and operating costs) of
the RMMs. Then, the involved stakeholders were divided
into three groups and they identified alternative RMMs
which were categorised according to the phases of the disas-
ter management cycle (i.e. mitigation & prevention; prepar-
edness; response; recovery). The findings of the assessment
ranked the following RMMs in order of relevance: mutual
planning and training, underground cabling, ICT systems,
and forming and disseminating situational awareness.
Another paper deals with climate change (CC) and its
related natural hazards for CIs. Airports are CIs particu-
larly at risk from the potential consequences of CC with
impacts like sea-level rise, increased temperature, changes
in precipitations or in wind patterns. The work presented
in Burbidge (2016) starts from the studies carried out by
EUROCONTROL, the European Organisation for the
Safety of Air Navigation, to demonstrate that, although
awareness is growing, there are still significant barriers
to take actions (e.g. lack of reliable information, missing
guidance). Therefore, EUROCONTROL, together with
Manchester Metropolitan University, organised a work-
shop, attended by 30 participants representing industry,
regulators, and academia, which led to identify four key
priorities to overcome these barriers: better understanding
of the problem; assessing the problem; initiating actions to
adapt both operations and infrastructure; collaboration in
research and information sharing and communication of
best practices both within Europe and globally.
Taking into account the international framework to
promote disaster risk reduction throughout the educa-
tion sector, Bandecchi etal. (2019) focus on assessing the
school resilience, from an organisational point of view.
A survey, made by 7 different types of questionnaires
for different ages (from 3 to 19years) and respondents
(students, headmasters, professors, auxiliary personnel),
was conducted in 27 schools in Tuscany (Italy), located in
areas of high geo-hydrological and seismic hazards. The
questions were inspired by the Naylor examples of assess-
ment techniques and were structured in different ways to
identify criticalities in the community management pro-
cess: closed question, open-ended question, completing
table, matching exercise, cartoon strip sequence, graphic
organiser, sequencing, graphic open-ended question, and
graphic closed question. Shared results have been obtained
from a resilience analysis on US campus emergency man-
agement units, by means of a survey, and then investigated
in statistical terms (Murphy etal. 2019).
Shifting towards civil response management, Große
(2019) examines the Swedish Emergency Response plan-
ning process, trying to identify and characterise the sources
of uncertainty with respect to power shortages. This plan-
ning is a multi-agency activity that requires decomposition
and coordination of goals and means throughout a multi-
level approach. It involves local actors as municipalities and
power grid operators, regional actors like county adminis-
trations, and national actors as agencies. The author con-
ducts a literature and document review analysing guidelines,
national laws, and reports regarding the Swedish description
of the planning approach. Thereby, sources of uncertainty,
stemming from lack of knowledge, emerged and guided the
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

355Environment Systems and Decisions (2021) 41:341–376
1 3
meetings with planners at municipalities and County Admin-
istrative Boards (CABs). Semi-structured interviews were
then conducted, registered, and transcribed: a questionnaire
with open-ended questions, based on the sources of uncer-
tainty, was utilised to guarantee a similar structure for the
interviews and to allow participants to report on individual
experiences and perceptions. Follow-up questions were
asked for more richness of detail and clarity. This study
revealed three sources of uncertainty: the planning refer-
ence process in general, the decision-making process, and
the direction and guidance alongside these processes.
In the same national context, Olausson (2019) deals with
the resilience in the case of power shortage, focusing on
the Swedish STYREL policy. STYREL (a Swedish acronym
for “Steering of electricity to prioritised users during short-
term electricity shortages”) is a planning and prioritisation
process, involving national authorities, CABs, and munici-
palities. The aim of this process is to identify and prioritise
the vital societal functions that must be carried out during
a situation of power shortage, to foster sustainability and to
increase resilience for power supply. A survey including 21
coordinators at the regional level and semi-structures inter-
views at three different CABs were conducted to provide a
broad picture on the importance of the process, its useful-
ness, and the trust within the networks. However, according
to the frank discussion outlined by the author, there are no
guarantees that STYREL process, such as it has been carried
out, will make the electrical energy supply more resilient
and sustainable; nor does it seem that STYREL has created
any formal or established type of collaboration between pri-
vate and public sectors actors in practice.
Also Hiete etal. (2011) analyse the possible impacts of a
power outage, focusing on the German healthcare. As effects
and measures are strongly determined by the duration of the
power outage, three scenarios are developed reflecting three
different outage durations: below 8h, between 8 and 24h,
more than 24h. Discussions on these scenarios were fostered
among the participants and three sub-groups were formed
for the impact analysis. In a second step, Preparation, Miti-
gation and Recovery (PMR) measures were collected and
discussed, and results were sent to experts for commenting.
Finally, semi-structured interviews with additional experts
were conducted to have a more balanced view. The analy-
sis of impacts represents an important starting point for the
identification of crisis and continuity management measures
referred to the three different phases of the crisis manage-
ment process: prevention, crisis management, and recovery.
Another important result of the workshop was that good
cooperation between administrative authorities and health-
care providers as well as power suppliers may reduce nega-
tive impacts significantly. Cooperation includes sharing of
information, for example on the expected duration of the
outage, on the patients needing power-dependent medical
devices, and on respective resources.
Line (2013) focuses on power industry and examines how
distribution system operators (DSOs) align their policies
to the principles of resilience within information security.
The author performs a case study of incident management,
conducting 19 semi-structured interviews to survey cur-
rent practices and to identify the improvements needed to
enhance smart grids. The ISO/IEC 27,035:2011 incident
management process scheme, composed by five phases (plan
and prepare; detection and reporting; assessment and deci-
sion; responses; lesson learnt), is used to investigate findings
from the interviews to ICT and power automation systems
managers of six large Norwegian DSOs. The investigation
reveals that DSOs have quite some steps to go in the direc-
tion of being resilient organisations with respect to informa-
tion security: plans for responding to information security
incidents do not exist in all DSOs, training is an underpri-
oritised activity, noticing and evaluating minor incidents are
not common in practice.
The energy sector constitutes a societal economic driver
which requires to study its vulnerabilities, barriers, and
resilience (Pathirage and Al-Khaili 2016). Among the per-
sonnel of three different power plant facilities (Abu Dhabi,
Dubai, Sharjah), 42 questionnaires containing close-ended
questions, "yes/no" options, and few open-ended questions
were answered by professionals aged between 30 and 40
and semi-structured face-to-face interviews were undertaken
with the top management. Questionnaire results were shown
to respondents, who were asked to comment their own per-
spective, and then interpreted via a resilience management
lens. Data were elaborated through MS Excel and SPSS,
generating descriptive statistics. Findings indicate terror-
ism, atmospheric, and tectonic hazards as the main risks
of vulnerability, while the lack of or absence of national
government legislation, and awareness and education were
revealed as the main barriers.
Carpenter (2014) aims at creating a picture of the
strengths and weaknesses of the state of Georgia and
Savanah area, based on a review of existing resilience ini-
tiatives and interviews with CIs’ representatives. About 30
individuals among facility managers, safety specialists and
other representatives of Georgia-based CIs’ sites (i.e. state,
local, military, and private facilities) were interviewed.
Questions included current practices that enhance and
detract from resilience, barriers to resilience, development
of contingency plans, kinds of exercises conducted, iden-
tifications of less or more prepared sectors, identification
of interdependencies and cascading effects between sectors,
command structures, leadership, public outreach, how to
allocate resources to best improve resilience.
With a similar focus, the framework presented in
Labaka etal. (2014) provides a set of policies to be
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

356 Environment Systems and Decisions (2021) 41:341–376
1 3
implemented in order to increase CIs’ resilience level.
These policies were obtained through several research
methods: Group Model Building workshops; examination
of multiple case studies of different past major industrial
accidents; complemented by a Delphi method where 15
multidisciplinary experts from different sectors (academic,
transport, energy, and first responders) took part answering
two different questionnaires with different aims and con-
tents. The policies were classified based on the four resil-
ience dimensions (technical, organisational, economic, and
social) and on the two resilience types identified (internal
resilience referred to a specific CI and external resilience
associated to involved external agents such as govern-
ment, first responders, and society). Finally, 25 experts
were asked to provide the temporal order in which the
policies should have been implemented, to achieve a high
efficiency and effectiveness in their implementation.
Some papers start from disruptions in the payment sys-
tems and investigate the organisational aspects to increase
their resilience. Berggren etal. (2018) come back on gaming
simulation and aim at creating a game for participants from
food, fuel, and financial industry sectors, who could ben-
efit from major training to understand the challenges posed
by interdependencies like a major disruption in the pay-
ment systems. In particular, the authors focus on the design
choices for developing a mixed-methods approach to assess
teams’ resilient capability. The authors suggest that team
resilience can only be captured by a holistic mixed-method
that considers both “soft” aspects like team workload, col-
laboration, trust, shared awareness, collected through verbal
questions or queries, as well as "hard" measures of team
qualities and performances, considering some indicators like
payment options, good flows, trust, and security.
Also Van Laere etal. (2017) identify and discuss chal-
lenges faced in case of disruptions in the payment system.
The method is based on inductive qualitative research. Data
sources included documents of previous incidents, inter-
views with key representatives of each relevant sector and
two workshops with local and national actors. Results from
document study and interviews were utilised for building
two scenarios discussed during the following workshops,
whose outputs were analysed to identify seven challenges
for CIs’ resilience. The analysis proves resilience to be not
only a matter of technical measures (i.e. alternative pay-
ment solutions, rationing fuel or food, and offering services
to the vulnerable part of society), rather involving several
communicative challenges (i.e. maintaining trust, preventing
hoarding, avoiding panic).
Another paper deals explicitly with the organisational
aspects of the payment system: Johansson etal. (2018) aim
at understanding how local businesses in the sectors of bank-
ing, food, and fuel distribution are prepared to manage any
disruptions. Six semi-structured interviews were recorded
and transcribed, and a thematic analysis was then applied
to the transcribed material. The results show that food, fuel,
and bank sectors are not prepared for a long-period disrup-
tion in the payment system. The respondents trust others
(mainly IT Support) to solve the problems and they assume
that disruptions last few minutes or hours. There is no plan,
so the respondents are likely to shut down their business if
the payment system is not working for a longer period. These
concerns may be also emphasised in the current pandemic
era.
4.3 Community resilience
Following an even broader perspective, this paragraph illus-
trates those papers that deal with CIresilience at a com-
munity level. Resilience is interpreted here as the ability of
a community to handle surprises, avoid disasters/accidents,
and to be able to recover to a satisfactory state, i.e. nor-
mal societal operations (Johnsen and Øren 2015), and it is
now discussed in terms of its respective qualitative research
methods.
When dealing with a community dimension of resilience,
it is necessary to take into consideration the role of public
perception and public tolerance levels for minimum level of
service and rapidity of service restoration, as demonstrated
widely by IMPROVER project (Lange and Honfi 2017;
Rosenqvist etal. 2018; Storesund etal. 2018; Grosse 2019).
This latter indeed, adopting a wide variety of qualitative
research techniques (e.g. questionnaires, interviews and
workshops), provided evidence of the crucial role played
by public engagement in the management of CI resilience
(Petersen etal. 2020b). Among the experiments, it appears
relevant the case study on the transportation system related
to the Oresund crossing between Denmark and Sweden
(Petersen etal. 2018), or the Hungarian highway (Petersen
etal. 2019b).
Banks etal. (2016) focus on the resilience of a rural
community of central Appalachia, a U.S. remote mountain-
ous region prone to flooding. An inter-professional team of
nursing, architecture, and engineering students conducted a
comprehensive assessment of health and environmental liv-
ing conditions via an ethnographic data collection approach,
using both quantitative and qualitative sources of data:
interviews, surveying, open-ended questions, observations,
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

357Environment Systems and Decisions (2021) 41:341–376
1 3
measurement, and photography. Qualitative interview data
were organised and securely managed using NVivo software,
while quantitative data were organised into a database and
descriptive statistical analysis using SPSS 22. A model of
resilience for the rural Appalachian community was devel-
oped, depicting the cycle of facing hardships, rebounding,
supporting one another, and building community.
Also Mavhura (2017) focuses on building community
resilience in the Muzarabani district of Zimbabwe, often
subjected to drought and flood-related disasters. The study
applies systems-thinking approaches to examine how rural
livelihoods address such living difficulties. 40 interviews
were conducted among local authorities, traditional lead-
ers, villagers, and health officials, focusing on the five liveli-
hoods pillars: Natural, Economic, Human, Physical, Social.
Then, three separate focus groups were held on the same
capitals, comprising people who experienced the highest
magnitude floods in 2008. Finally, a survey involving 700
households was conducted through questionnaires self-
administered with the help of research assistants. The results
were analysed by using thematic analysis and descriptive
frequencies in Vensim software. In the study, resilience
emerges as sharing resources among flood and draught vic-
tims and shows existing absorptive and adaptive capacities
that smallholder farmers put in place to cope with natural
hazards. Ethno-based flood and drought warnings, tempo-
ral migration to highest zones, particular social net (Zunde
raMamb) and scheme (nhimbe) to share resources: these are
the constituents of absorptive capacities. Adaptive capacities
like practice of flood recession agriculture (mudzedze), dual
cropping system, traditional flood proofing structures (dura
and dara) suggest that the community has also the ability
to store and recall flood experience, to learn and reorganise
resources to address flood threats.
Simpson etal. (2010) focus on CIs’ measurements, using
recovery curves, after the impact of the hurricane Katrina
in 2005 over two coastal communities of the Mississippi
region. Data were collected by interviewing key informants
as public works administrators, government officials, and
emergency personnel. Photographic evidence, Geographic
Information Systems (GIS) data, and public documents were
also used to integrate interviews. Press releases and media
reports completed the missing data, although these releases
and reports are, according to the authors, inherently sub-
optimal sources: press releases are issued from corporate
sources and tend to put organisations in good light by mini-
mising failures, while media reports look for the emotional
and interesting issues. Therefore, the research underlines the
importance of data collection in emergency, establishing
recovery curves and standardised measurement technologies
for infrastructure, as helping tools to understand the effects
and assist community to prepare for next events.
Still on hurricane Katrina, Glavovic (2008) presents the
vicissitudes of the city of New Orleans. The author visits
the region six times after the disaster and conducted natu-
ralistic observations and interviews with planners, academ-
ics, and other people involved in the reconstruction efforts.
The research, conducted with qualitative methods, allowed
developing a conceptual framework to define principles for
guiding actions related to building more sustainable and
hazard-resilient communities. In this framework, the task
of the governance institutions is to mediate human access
to natural resources and prevent human beings impacts from
exceeding certain thresholds. Then, the authors provide a
list of principles aimed at involving local communities in
the decision-making processes and on the prioritisation of
ecological sustainability.
A similar focus on the reconstruction process has been
documented in Ong etal. (2016), but in Tacloban city (Phil-
ippines), after the typhoon Haiyan in 2013. The authorities
decided to implement a No-Dwelling Zone (NDZ) along the
coastline. Housing reconstruction and people relocation pro-
grams were started to move households to other safer areas.
Among these people, the authors conducted paper-based
questionnaire surveys to investigate disaster experiences;
assistance received; reconstruction or relocation experience;
household and community decision-making; demographic
profile. The responses were triangulated with focus groups
with beneficiary households and key informant interviews
were conducted with government and non-government
organisations. Finally, the collected data were examined by
using Wilcoxon–Mann–Whitney test for categorical varia-
bles, Spearman’s Rank Correlation for ordinal variables, and
One-way Analysis of Variance for comparisons of groups.
The questionnaire survey was conducted at three different
sites that used three different approaches: owner-driven on-
site reconstruction; community-driven off-site relocation;
contractor-driven off-site relocation. Household respondents
were asked to indicate their level of satisfaction regarding
the programs, based on a five-point Likert scale. In terms of
resilience management, results showed that on-site recon-
struction was delayed to insufficient assistance schemes like
materials and skill training, while off-site relocation was
delayed by prolonged land acquisition and subcontracting
issues. Satisfaction levels of respondents were affected by
disruption of CIs, such as water and utility services, lack of
livelihood opportunities as markets or business establish-
ments, and proximity to learning facilities.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

358 Environment Systems and Decisions (2021) 41:341–376
1 3
Herrmann Lunecke (2015) deal with the Chilean response
after the severe tsunami of February 2010. After examina-
tion of the Chilean law framework for coastal planning, the
study analysed tsunami mitigation measures and policies
developed at local level in recent reconstruction plans. Then,
a total of 50 semi-structured interviews and questionnaires
with key local and regional actors of public and private sec-
tors, as well as community leaders, were conducted to ana-
lyse tsunami impact mitigation measures. Processing and
analysis of the collected data were realised through a simple
content analysis and a frequencies analysis of each response
category. This study found that mitigation measures in Chil-
ean coastal urban planning after 2010 have been focused
at the local level on anti-tsunami engineering solutions,
whereas other policies like key infrastructures restriction in
tsunami flood zones or relocation of housing and key equip-
ment were not adopted, due to the fact that reconstruction
plans were non-binding master plans and citizens scarcely
participated in their development.
The research byRogers etal. (2019) deals with coastal
communities and describe NASA’s efforts on applying
remote sensing and modelling in areas exposed to high
potential risks. In particular, the NASA Disaster Program
held a workshop in Spring 2017 with participants from aca-
demic institutions, local and regional which gave birth to
the Mid-Atlantic Community and Area at Intensive Risk
(CAIR) team that demonstrated the ability to integrate sat-
ellite derived earth observations and physical models into
trusted and actionable knowledge for tactical and strategic
decision-making.
From a different threat, Cradock-Henry etal. (2018) ana-
lyse the emergence of new networks, agents, and institutions
after the 2016 earthquake, for the small coastal settlement
of Kaikōura in New Zealand. The paper utilises a qualita-
tive method based on 12 semi-structured interviews and two
focus groups, supported by literature review and document
analysis. The first interviewees were selected among existing
research collaborations with emergency management staff
and local council. Then, a snow-ball strategy was imple-
mented to identify further participants’ representative of the
range of affected interests, making this a convenient sam-
ple (e.g. local government, regional economic development
staff, food producers). Themes were identified and emerged
through the process of data analysis using deductive and
inductive methods, validated by inter-reliability analysis.
The paper focuses on two case studies that show the trans-
formation of the rural community after the earthquake.
In the first one, the shock induced by this event raised the
awareness of the rural community of how important it was
to strengthen local food networks, to build local capabilities,
to promote local agricultural products as one of the reasons
for tourism, the major economic driver of the district. The
second case study regards the activity of a team that was
formed after the earthquake to provide isolated rural house-
holds with life essentials and a readily accessible line of
communication, responding to immediate personal needs.
On the importance of CIs for local communities, Grosse
(2019) focuses on regional airports in Sweden, clarifying
their role for communities, in terms of regional develop-
ment and civil protection. Four semi-structured interviews,
employing a questionnaire with open-ended questions, were
conducted, and recorded among a selection of stakehold-
ers. The data collection was extended with a workshop of
14 delegates from public and private organisations, which
aimed at giving suggestions to develop a conceptual model
for assessing risks, the criticality of infrastructure in the con-
text of civil protection and the economic value of airports.
In this case, the participants emphasised the importance of
several transports by air for the society: ambulance flights,
transportation of criminals, flights for crisis management
such as fighting wildfires.
Another paper deals with the economic impacts on a com-
munity: Akhtar and Santos (2013) investigate the adverse
impacts of hurricanes to interdependent workforce sectors
in Virginia, using the dynamic inoperability input–output
model (DIIM), a risk-based transformation of the Leon-
tief’s classic input–output model (i.e. a system of linear
equations). To study the ripple effects which lead to seri-
ous economic repercussions and to identify the most critical
workforce sectors and prioritise them, two significant met-
rics were considered: inoperability level and economic loss
to industry sectors. Although this research is based on quan-
titative method, qualitative sources, as published surveys
of workforce absenteeism in the aftermath of hurricanes,
constituted the core of the data analysis, being the source for
the formulation of the workforce perturbation models. The
results provide guidance in disaster policy-making, particu-
larly in systems-based resource allocation, enhancing pre-
paredness to better manage the consequences of hurricanes
to workforce sectors.
4.4 Urban resilience
This paragraph is dedicated to the analysis of the papers
where resilience is addressed explicitly as referred to cit-
ies and urban development. On this context, resilience
has a twofold interpretation: an umbrella term for every-
thing to be addressed within a city, related to the aspects
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

359Environment Systems and Decisions (2021) 41:341–376
1 3
of sustainability, climate change adaptation, or disaster risk
reduction, an ability required within a stable risk assessment
process (Ferreira and Bellini 2018; Bellini etal. 2020). This
second dimension is the one considering of interest in this
review, since it is nested in the recovery phase after a haz-
ard impact, representing the “bouncing back” process phase
which is the literal translation of the term resilience that we
discussed above (Fekete and Bogardi 2018).
On this research stream, both Fekete and Bogardi (2018)
and Fekete and Fiedrich (2018) are chapters included in the
book entitled: “Urban disaster resilience and security”. In
the former, the authors underline the relevance of urban
areas which are laboratories for observing and conceptu-
alising resilience, because of the concentration and overlay
of human beings, values, ideas with structural and non-
structural objects, giving rise to socio-environmental or
socio-ecological systems. In this environment, CIs have the
potential to even aggravate a disaster situation for society,
but are also key components for recovering, since they are
a core part of resilience of an urban habitat. Again, Fekete
and Fiedrich (2018) emphasise the central role of urban
dimension where urban environment and resilient cities are
flagships of recent research to investigate not only worst-
case impacts of natural or man-made hazards but also to test
the effectiveness of measures. Urban areas are selected for
research and funding since density of people and human val-
ues are concentrated here and this is both an asset and risk
factor. Finally, the authors show how existing indicators for
resilience assessment can be improved or new indicators can
be created: by adjusting old indicators, by involving experts
or by using data sources or big data.
Gonzalez etal. (2017) report first-year findings of the
European Union’s Horizon 2020 project called Smart
Mature Resilience (SMR) (Iturriza etal. 2017; Marana
etal. 2019). This project aims to develop a maturity model
(a tool to assess current effectiveness of a group, support-
ing figuring out what needs to be improved) for society’s
resilience, focusing on the progress towards resilience of
seven cities, assimilated to the vertebrae of the strong Euro-
pean resilience backbone. In particular, the model consists
of four maturity stages: Starting, Moderate, Robust, and
verTebrate (SMART) corresponding to increasing cities’
capabilities towards higher resilience maturity levels. The
model was enhanced through a literature survey on resil-
ience, followed by an expert assessment by using Delphi
method, and a series of workshops with experts on CIs,
climate change, social dynamics (i.e. immigration, poverty,
population ageing) and city representatives. Through Group
Model Building, policies at strategic level were associated
to each stage of the maturity model and classified along
four resilience dimensions: Robustness of infrastructure &
Resources, Preparedness, Leadership & Governance, Coop-
eration and Learning. The paper also describes an interactive
questionnaire for risk assessment programmed in Excel (i.e.
Systemic Risk Assessment Questionnaire), whose questions
are dependent each other, capturing the interdependence
between risks, and whose output is a risk score which helps
in assessing a particular project or initiative and in prioritis-
ing those areas which require more attention. Finally, by
using a Group Explorer decision support system, different
views regarding the risks associated to CIs, climate change
and social problems, were collected during the workshops.
Monstadt and Schmidt (2019) shed light on the key role
of the urban governance in the German approach to the CIs’
protection. The study is based on literature and documentary
analyses, interviews with 48 experts and workshops with
practitioners from local utility companies and crisis man-
agement teams. The qualitative research pointed out some
critical issues for the urban governance level: the German
federal system (the Lander delegate the key operational
tasks to the municipalities), the European regulatory market
reforms (many public utilities have been privatised and many
networked infrastructures have been unbundled into disinte-
grated companies), and the budget reduction (the companies
are more likely to accept temporary revenue losses through
supply disruptions during crises than to invest in the preven-
tion of such events). These issues result in a lack of coordi-
nation and cooperation among the main stakeholders; scarce
information sharing and awareness of other infrastructure
locations and their potential vulnerability; different vision
about intervention priorities, mitigation strategies, staff
training, and crisis exercises.
Pescaroli (2018) explores the awareness of cascading
risk, the possible mitigation measures, and the current lev-
els of training among stakeholders of the city of London,
all intended as measures for increasing resilience in crisis
management. The research started from a workshop whose
participants came from emergency response organisations,
public utilities, businesses, and academia. The workshop
included questions with multiple items to be evaluated
through a standard Likert scale. To record the bottom-up
stakeholders’ perspective, each section of the survey also
included open questions for the answerers, asking sugges-
tions for mitigation measures, training strategies, general
comments on the workshop. The answers were analysed with
SPSS software, considering respondents’ experience, affili-
ation, and gender; possible correlations between answers
within sections and across sections were also searched.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

360 Environment Systems and Decisions (2021) 41:341–376
1 3
The findings confirm that the current crisis management
approach to cascading risk is inadequate. London’s stake-
holders are aware and concerned about cascading events and
interdependencies, but they recognise that these issues are
not sufficiently incorporated in the current policies, prac-
tices, and emergency management at large.
From a terrorist point of view, urban environment is a
perfect target, since cities can be seen as nodes where peo-
ple, ideas, value streams, and information meet. On this
context, Heino etal. (2019) aim at better understanding the
operational environment when a CI becomes a target of a
terrorist attack. Therefore, two scenarios, which reflected
the model for comprehensive security defined in the Finn-
ish Security Strategy for Society, were elaborated during a
workshop: water contamination and electricity disruption.
The workshop, organised at the Police University College in
Tampere, was facilitated by a modified Open Space method.
Unlike the original method, participants did not suggest top-
ics for discussion, but they were invited to discuss about for-
mation of situation awareness, competencies, and resources
in crisis management, crisis communication, and develop-
ment of a new tool in terms of continuity management. The
sixteen participants represented the key actors who would
be involved in the workshop scenarios in real life. In order to
enhance system’s resilience, the findings reaffirm the impor-
tance of a multi-agency situational awareness, shared among
key actors, as an essential element in decision-making and
the need for a more networked defence to face threats as
organised crime, hybrid action, and terrorism.
The study in Räikkönen etal. (2016) is based on the
research conducted in the INTACT and HARMONISE pro-
jects, both co-funded by the European Union (Doyle etal.
2018). The first project addresses the resilience of CIs to
the challenges posed by extreme weather events, while the
second one presents resilience enhancement methods for
large-scale urban built infrastructures. The aim of the paper
is to establish a systematic approach for conducting risk
assessment of urban CIs and for calculating and compar-
ing benefits and costs of measures. The proposed approach
is flexible, encompassing not only a rigorous quantitative
assessment, but also allowing for a semi-quantitative or qual-
itative assessment. For example, for CI and system model-
ling, since the technical information on CI core functions
and processes is specific for each CI and system, cooperation
with experts, who have the knowledge and the data access, is
needed. Moreover, for risk estimation and evaluation, there
are three different types of calculations: qualitative, semi-
quantitative, and quantitative. Some methods aim at general
mapping and understanding of potential consequences and
impacts, others are based on very detailed analysis in the
form of indexing and strict quantitative modelling; a mixture
of this approaches is used in some cases.
Finally, Lomba-Fernández etal. (2019) propose a guide
to help cities to become more resilient by considering urban
CIs as key elements to cope with Climate Change (CC)-
related crises and maintain citizens’ welfare. The research
consisted of two phases: the conceptualisation and the devel-
opment phases. In the first phase, a literature review was
carried out, analysing scientific and grey literature articles
and reports. In the second phase, a co-creation approach was
adopted, through focus group method, to elicit information
from experts in two cities in the Basque country (Spain). In
particular, two workshops in each city were organised with
30 multidisciplinary experts to identify resilience building
policies for improving urban CIs’ resilience level and to
carry out a detailed analysis for classifying the CIs against
CC impacts and for studying the interrelationships among
CIs. Then, two pilot tests, one in each city, were carried out
to review the guide and use it in a real context. Moreover,
additional interviews with the heads of the environmental
departments of eight towns which had not participated in
the development phase were carried out to provide relevant
feedbacks. The main result of this research is a guide to
help cities to analyse their current situation and understand
challenges and opportunities supporting the development of
resilience-strengthening strategies.
5 Discussion
This section summarises and discusses the outcomes of
current works on qualitative research for CIs’ resilience as
emerged from full-text analysis of reviewed documents. The
discussed approaches explore macro-themes that resilience
research on CIs should take into account, mainly in case of
Tier I (or Tier II) approaches. Nevertheless, many of these
discussion points remain valid even for more quantitative
investigations, as for Tier III approaches (Linkov etal.
2018).
5.1 Qualitative andquantitative, orqualitative
alone?
Adopting a qualitative rather than a quantitative approach to
study complex issues is not an easy choice: both approaches
have their pros and cons. As mentioned in the introduc-
tion, this is a debated topic, which also emerges from the
reviewed papers.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

361Environment Systems and Decisions (2021) 41:341–376
1 3
The choice to use qualitative or quantitative approach
should be driven by the needs and desires of the decision
makers who usually prefer having numbers on which base
their decisions. For some type of analyses, e.g. cost–benefit,
it is expected to have quantitative assessment (Matsika etal.
2016), while for others, there is much more flexibility. About
quantitative sources, often claimed to lack data, it has been
proposed to elicit knowledge from press releases and media,
which are however considered sub-optimal sources. Press
releases are issued from corporate documents and tend to
put organisations in good light by minimising failures, while
media reports look for the emotional and interesting issues
of stories (Simpson etal. 2010). This means it is a viable
option, but there should be a conscious usage of the respec-
tive storytelling. One possible option here, at least for some
type of disruption and events, could be to link the usage of
social media (Verma etal. 2019) (e.g. Twitter), possibly as
a social sensor and get information on this direction. Some
early results on power grids prove the potential benefits to
further research on the topic (LaLone etal. 2017; Heglund
etal. 2020).
Quantitative data offer an inherent benchmarking dimen-
sion, which anyway requires to be complemented by qualita-
tive assessment to ask the right questions for understanding
the context, and provide meaningful answers to interpret the
scores (Bernroider etal. 2016). However, qualitative assess-
ment needs also an attentive and accurate planning to avoid
unmanageable results. In retrospect, it is worth mentioning
how Bernroider etal. (2016) admit that questionnaire and
terminology might have been more precise in their study if
workshops and interviews should have been conducted first.
Then, the original questionnaire should have been adjusted
considering the respondents’ comments, allowing for a more
precise questionnaire to be distributed. There is always a
trade-off to understand the right sample size for preliminary
interviews and workshops, which could count on the so-
called knowledge saturation principle (Onwuegbuzie etal.
2011). Future research may thus investigate this principle
explicitly in the context of CIs, to prove its feasibility and
define guidelines to support analysts in conducting their
research.
Some scepticism has been documented about resilience
measurability in quantitative terms, because of CIs’ com-
plexity which cannot be covered simply by larger data
availability (Fekete and Fiedrich 2018). The same authors
warn about the shortcomings of exemplary social science
qualitative assessments, backed from their experience on
the field. Using participative social science methods such as
workshops, focus group discussions, and expert interviews,
Fekete and Fiedrich (2018) realise that the same experts
might offer contradicting arguments in assessments repeated
only few weeks after the first one.
Reliability of quantitative scoring must be treated as cau-
tiously as reliability upon individual qualitative results. Nev-
ertheless, combining the two dimensions appears a promis-
ing choice to build bridges between users, social and natural
scientists. Such mixed methods and approaches are often
balancing demands by different end users, and even if they
could disappoint some of them partly, they generally allow
for a bigger picture.
5.2 Involvement ofstakeholders
Another relevant point of discussion is the involvement of all
stakeholders, in both urban realities, and complex infrastruc-
ture systems, and organisational settings, characterised by
dense interdependencies and hybridity between social, natu-
ral, and technological worlds (Monstadt and Schmidt 2019).
For example, the importance of preparedness and contin-
gency thinking emerges explicitly in urban settings, along
with the need for a networked defence strategy among
stakeholders to face threats which lurk within the increased
diverse and sophisticate operating environment (e.g. organ-
ised crime, hybrid action, and terrorism) (Heino etal. 2019).
Resilience requires different strategies for urban reali-
ties (Dierich etal. 2019): while bigger cities usually have
centralised resources to face crises, smaller ones depend
even more strictly on the collective use of resources and,
therefore, emphasise cooperation and coordination, includ-
ing citizens’ participation (Lomba-Fernández etal. 2019).
Additional examples on the need for stakeholders’ involve-
ment refer to community resilience, where methods such as
the Resilience Matrix (Fox-Lent etal. 2015; Linkov etal.
2018) or its variant Population Resilience Matrix (Rand
etal. 2020) have been used to organise community goals
in a context of population displacement and infrastructure
reconstruction. About the Population Resilience Matrix, the
authors explicitly claim as critical the community engage-
ment for establishing legitimacy around acceptable bounds
of performance.
A similar need for stakeholder involvement has been doc-
umented at intra- and inter-organisational levels to support
stakeholders’ common knowledge and resilience to cascad-
ing failures (Seppänen etal. 2018). A noteworthy result in
this case is the benefits of cooperation between administra-
tive authorities, healthcare providers, and power suppliers
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

362 Environment Systems and Decisions (2021) 41:341–376
1 3
to plan for the negative impacts of a disruption (Hiete etal.
2011). At the extremes of this concept there lies the push
for mitigation measures to be included within legislation
(Pescaroli 2018), and the involvement at local level, where
private citizens may voluntarily self-arrange in groups to
define some resilience strategy (Bouchon etal. 2014).
However, such type of coordination is a rather complicate
and delicate task, and should be treated taking care of the
potential privatisation of some companies, as well as the
organisational cultures. A critical reflection in this sense
comes from the STYREL project, where the public–private
cooperation envisaged in the process revealed lack of trust
between actors, concerning both lack of resources and feed-
backs (Olausson 2019). Most reviewed papers shed light on
the importance of data availability and information sharing.
Few studies focus on extracting the failure dependencies on
experts’ knowledge because detailed information about the
CI failure interdependencies is considered highly sensitive
and private CI operators are reluctant to share information
with academic communities (Seppänen etal. 2018; Behesh-
tian etal. 2017).
Involvement means cooperation, information sharing,
transparency, and discussion to enrich one other perspec-
tive. Communication is crucial at all level of CIs’ manage-
ment (Antunes etal. 2017). Future research should explore
the contributing success factors to cooperation as well as it
should give evidence of those scenarios where cooperation
drove to positive outcomes, isolating the strategies that foster
success, and the ones that may create detrimental effects.
A promising research stream in this sense may refer to the
usage of methods built in the context of Resilience Engi-
neering used for socio-technical system safety, as for the
Functional Resonance Analysis Method, early applied for CI
resilience in the urban context (Bellini etal. 2016). At soci-
etal, or even organisational level, this may imply the push for
development of dedicated app to inform about vulnerabilities
and response strategies, as for the early results in the disaster
management domain (Petersen etal. 2019a).
5.3 Guidelines, butforwhom?
Several papers reveal the need of guidelines and criteria
for helping authorities at different scales to develop crisis
management, to analyse vulnerabilities, and to improve
CIs’ resilience (Herrera etal. 2017; Woltjer etal. 2018).
Of course, the societal scale at which guidelines are pro-
posed shapes the details and operating strategies suggested
(Petersen etal. 2020c).
At international scale, focusing on transportation infra-
structure, it is worth mentioning how the European Surface
Transport Operator (EUSTO) built common guidelines
for developing security plans for surface transport with an
EU dimension, involving National Contact Points of EU
members and surface transport stakeholders (Hedel etal.
2018). Still in EU, the AESOP (Association of European
Schools of Planning) guidelines offer details on identifying
target population, and emphasise the need for CI operators
to give evidence via news media of their positive working
relationships with their counterparts, always in light of the
regulatory context (Reilly etal. 2018).
At national scale, CIs require more regulatory efforts
by national policies and highlight the need for identifying
place-based vulnerabilities, locally differentiated prepared-
ness strategies, and the training of local utility companies
as well as crisis management teams (Monstadt and Schmidt
2019). Besides, it should also be fostered some public–pri-
vate partnerships, leadership, trusted and secured informa-
tion sharing, and exercises for social vulnerability assess-
ment (Carpenter 2014).
At regional scale, guidelines should support authorities
to integrate CI protection and reconstruction into the spatial
planning process. This target can be achieved via indicators
(e.g. population equivalents) for a systematic analysis of vul-
nerabilities, interdependencies, and criticalities of infrastruc-
tures and customers (Riegel 2014). This is a viable research
stream, as proved by the interventions defined at a local
level to increase community resilience in light of a multi-
level integrated management approach, particularly care-
ful to respect and empower local cultures and capabilities
(Glavovic 2008) or by modelling population displacement
as a function of infrastructure reconstruction decisions, in
order to implement best strategies for infrastructure recovery
(Rand and Fleming 2019). The local level also includes the
management of urban resilience and the treats a stakeholder
profile should respect to be involved in a resilience research
(Lomba-Fernández etal. 2019).
In summary, the reviewed papers show the need of guide-
lines, and the benefits arising from them at different scales
(Save etal. 2019), with no specific inter-level research avail-
able yet. Future research should thus start from these results
and propose different staging areas to harmonise the pri-
orities, and the constraints imposed by different competent
authorities acting at different scales (international, nation,
regional, local).
5.4 Continuous trusted learning
A continuous trusted learning process is crucial for both
organisations and communities. For these settings, it is
frequently reported a poor risk awareness, as well as the
absence of systematic knowledge sharing, and poor scenario
training and emergency preparedness (Johnsen etal. 2009).
Solutions about organisations available in the literature
refer to the preparation of inter-teams’ meetings, where
diverse personnel are involved (technicians, operators,
managers, etc.) to reinforce mutual understanding and foster
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

363Environment Systems and Decisions (2021) 41:341–376
1 3
participation, by complementing top-down decision-making
with bottom-up evidences. It is expected that the owner-
ship of solutions supports a learning-oriented healthy envi-
ronment (Pathirage and Al-Khaili 2016). Similar evidence
emerges in school emergency management. School staff do
not know fully the actions and the post-evacuation proce-
dures, and both staff and students do not share the correct
perception of natural hazards. At a management level, this
result calls for a reconsideration of the connection between
schools’ evacuation plans and cities’ civil protection plans
(Bandecchi etal. 2019). Risk awareness and learning-
oriented approach are essential for communities who face
natural hazards continuously, and have to deal with impro-
vised means (Mavhura 2017; Cradock-Henry etal. 2018).
As an example, a research on the Appalachian community
proved their resilience based on faith and spirituality, cul-
tural values and heritage, and social support despite stressors
like poverty, rural isolation, low educational level (Banks
etal. 2016). Similar results have been achieved by this type
of research conducted in Zimbabwe where the results pin-
point to lemmas used by the community to identify inherent
capacities and strategies (Mavhura 2017).
These observations show how helpful might be a pre-
cise type of qualitative research, designed for ethnographic
research to understand local absorptive and adaptive capaci-
ties. Understanding humans in their real-life environment
becomes a central point to incorporate anthropocentric
assessment and transfer them to other communities. Besides
validating the idea in many communities, future research
may extend the concept to organisational ethnography
(Yanow 2012).
5.5 From cybersecurity tocyber resilience
Except for some papers like Linkov etal. (2013) where the
Risk Matrix is adopted to develop metrics useful for assess-
ing resilience of cyber systems, as explored from the bib-
liometric findings in Sect.3, cybersecurity has been less
investigated than others via qualitative research methods.
Literature review shows that there is a focus on identifying
cyber vulnerabilities and preventing cyber-attacks, but much
less attention to mitigate their effects by improving cyber
resilience. For example, Lykou etal. (2019) discovered that
the majority of the available questions, among distributed
qualitative surveys, focus on protection measures and techni-
cal safeguards rather than examining response and recovery
strategies. Available Self-Assessment tools are only one—
limited—component to assess cybersecurity: they should be
revised to ensure the full resilience capabilities and custom-
ised based on the organisation profile (Lykou etal. 2019).
Cybersecurity requires an evolution of traditional risk man-
agement process, calling for a greater emphasis on shared
responsibility, leadership, and humans intended as resources
(Rajamaki etal. 2018). The same observations apply for
cyber resilience in space missions (Millwood 2019).
These early observations mean that cyber resilience
among modern CIs is an under-developed domain, which
requires the modern view on resilience inspired by other
more traditional CIs. Qualitative research here plays a cen-
tral role to prioritise activities and identify threats by means
of dedicate surveys that can increase the level and quality of
information security to next maturity levels.
6 Conclusions
Continuing failures and disasters remind us the need to
further advance the available scientific understanding, and
policy-making, ofresilience for CIs. The management of
resilience in modern CIs requires an understanding of CIs’
functioning, as well as the needs and the determinant fea-
tures of all the stakeholders involved. Modern CIs require
indeed methodologies able to capture diversity, hetero-
geneity, and inter-relatedness, providing meaningful and
interpretable representations also in light ofcyber-physical
systems’ interdependencies(Patriarca etal. 2021). Quali-
tative research has a high potential in such settings, as
shown by the results of the systematic review presented
in this paper. Through qualitative research, the analysis
of CIs can reflect more explicitly the complex and tight
coupled factors of individual, group, societal behaviours,
as they act jointly with technological artefacts.
While the empirical results discussed in the paper prove
the usability and usefulness of qualitative research to deal
with resilience management of CIs, there are still several
research areas to be developed, as linked to the notion of
knoweldgeintegration: (i) from a methodological perspec-
tive, integration with quantitative methods still adopting
a systemic perspective; (ii) from a managementperspec-
tive, integration across different micro-meso-macro scales;
(iii) from a strategical perspective, integration among data
from different stakeholders, ensuring trust and coopera-
tion; (iv) from a tactical perspective, integration and dis-
semination of knowledge to support a continuous trusted
learning.
In ourmodern, uncertain, and turbulent world, future
research on CIs’ resilience should prioritise integration
to support the survivability and development of future
organisations, communities, cities, and societies towards
next staging areas of evolution and adaptations.
Appendix1
See Table2
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

364 Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 Summary of reviewed papers
Authors Title Year Main dimension Domain Approach
Banks L.H. Davenport L.A., Hayes
M.H., McArthur M.A., Toro S.N.,
King C.E., Vazirani H.M
Disaster Impact on Impoverished Area
of US: An Inter-Professional Mixed
Method Study
2016 Community Community Interview; questionnaire; Ethnographic
research
Glavovic B.C Sustainable coastal communities in the
age of coastal storms: Reconceptualis-
ing coastal planning as ‘new’ naval
architecture
2008 Community Not specified Interview
Mavhura E Applying a systems-thinking approach
to community resilience analysis
using rural livelihoods: The case of
Muzarabani district, Zimbabwe
2017 Community Community Interview; questionnaire; Focus group;
Observation
Ong J.M., Jamero M.L., Esteban M.,
Honda R., Onuki M
Challenges in Build-Back-Better Hous-
ing Reconstruction Programs for
Coastal Disaster Management: Case
of Tacloban City, Philippines
2016 Community Community Questionnaire; focus group; interview
Herrmann Lunecke M.G Urban planning and tsunami impact
mitigation in Chile after February 27,
2010
2015 Community Urban infrastructures Interview; questionnaire
Simpson D.M., Lasley C.B., Rockaway
T.D., Weigel T.A
Understanding critical infrastructure
failure: Examining the experience of
Biloxi and Gulfport, Mississippi after
Hurricane Katrina
2010 Community Community Interview
Rogers L., Borges D., Murray J., Mol-
than A., Bell J., Allen T., Bekaert D.,
Loftis J.D., Wang H., Cohen S., Sun
D., Moore W
NASA’s Mid-Atlantic Communities
and Areas at Intensive Risk Demon-
stration:: Translating Compounding
Hazards to Societal Risk
2019 Community Community Workshop
Johnsen S.O., Øren A 10years from risk assessment to regula-
tory action—is complacency creating
a reactive and brittle regulatory
regime in Norway?
2015 Community Not specified Interview
Grosse C Airports as Critical Infrastructure:
The Role of the transportation-by-air
System for Regional Development and
Crisis Management
2019 Community Transportation systems; aviation Interview; questionnaire; Workshop
Verma R., Karimi S., Lee D., Gnawali
O., Shakery A
Newswire versus Social Media for Dis-
aster Response and Recovery
2019 Community Community Social network
Murphy S.A., Brown J., Shankar A.,
Lichtveld M
A quantitative assessment of institutions
of higher education disaster prepared-
ness and resilience
2019 Community Community Questionnaire
Petersen L., Sjöström J., Horvath E Evaluating critical infrastructure resil-
ience via tolerance triangles: Hungar-
ian Highway pilot case study
2019 Community Transportation systems Questionnaire
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

365Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Bunney S.L., Ward S., Butler D Can UK water service providers man-
age risk and resilience as part of a
multi-agency approach?
2016 Community Water facilities Interview
Cradock-Henry N.A., Fountain J.,
Buelow F
Transformations for resilient rural
futures: The case of Kaikōura,
Aotearoa-New Zealand
2018 Community Community Interview; focus group
Akhtar R., Santos J.R Risk-based input–output analysis of
hurricane impacts on interdependent
regional workforce systems
2013 Community Not specified Questionnaire
Pagán-Trinidad I., Lopez R.R., Diaz E.L Education and building capacity for
improving resilience of coastal infra-
structure
2019 Community Community Workshop
Bouchon S., Urquhart J., Gibson R.,
BaMaung D., Dimauro C., Trucco P
The role of public–private stakeholder
collaboration to achieve critical infra-
structures resilience: Main findings
from the EU-CIPS MIRACLE project
2014 Community Not specified Questionnaire; focus group; interview
Petersen L., Lange D., Theocharidou M Who cares what it means? Practical rea-
sons for using the word resilience with
critical infrastructure operators
2020 Community Not specified Workshop; interview;
Rosenqvist H., Reitan N.K., Petersen L.
Lange D
ISRA: IMPROVER societal resilience
analysis for critical infrastructure
2018 Community Logistics Focus group; questionnaire
Adini B. etal Striving to be resilient: What concepts,
approaches and practices should be
incorporated in resilience manage-
ment guidelines
2017 Organisational Aviation; healthcare Delphi method; questionnaire
Van Laere J., Berggren P., Gustavsson
P., Ibrahim O., Johansson B., Larsson
A., Lindqwister T., Olsson L., Wiberg
C
Challenges for critical infrastructure
resilience: Cascading effects of pay-
ment system disruptions
2017 Organisational Payment system Interview; workshop
Labaka L., Comes T., Hernantes J., Sar-
riegi J.M., Gonzalez J.J
Implementation methodology of the
resilience framework
2014 Organisational Not specified Interview; workshop; Delphi method;
questionnaire
Pathirage C., Al-Khaili K Disaster vulnerability of Emirati
energy sector and barriers to enhance
resilience
2016 Organisational Energy systems; supply chain manage-
ment; oil and gas infrastructures
Questionnaire; interview
Johansson B.J.E., Jaber A., Van Laere
J., Berggren P
The lack of preparedness for payment
disruptions in local community core
businesses
2018 Organisational Payment system Interview
Mendonça D., Wallace W.A Factors underlying organisational
resilience: The case of electric power
restoration in New York City after 11
September 2001
2015 Organisational Energy systems Interview; questionnaire
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

366 Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Brown C., Seville E., Vargo J Measuring the organisational resilience
of critical infrastructure providers: A
New Zealand case study
2017 Organisational Energy systems; water facilities; trans-
portation systems; telecommunication
networks; oil and gas infrastructures
Questionnaire
Field C., Look R A value-based approach to infrastruc-
ture resilience
2018 Organisational Not specified Interview
Kachali H., Stevenson J.R., Whitman
Z., Seville E., Vargo J., Wilson T
Organisational Resilience and Recovery
for Canterbury Organisations after the
4 September 2010 Earthquake
2012 Organisational Logistics; construction; IT networks Questionnaire
Burbidge R Adapting European airports to a Chang-
ing Climate
2016 Organisational Transportation systems; aviation Questionnaire; workshop
Hiete M., Merz M., Schultmann F Scenario-based impact analysis of a
power outage on healthcare facilities
in Germany
2011 Organisational Healthcare Questionnaire
Comes T., Bertsch V., French S Designing dynamic stress tests for
improved critical infrastructure
resilience
2013 Organisational Energy systems Questionnaire
Bandecchi A.E., Pazzi V., Morelli S.,
Valori L., Casagli N
Geo-hydrological and seismic risk
awareness at school: Emergency
preparedness and risk perception
evaluation
2019 Organisational Community Questionnaire
Große C Sources of uncertainty in Swedish
emergency response planning
2019 Organisational Energy systems Interview; questionnaire
Petersen L., Fallou L., Carreira E.,
Utkin A
Public tolerance levels of transportation
resilience: A focus on the Oresund
region within the IMPROVER project
2018 Organisational Transportation systems Questionnaire
Sapeciay Z., Wilkinson S., Costello
S.B., Adnan H
Building Organisational Resilience for
the Construction Industy: Strategic
Resilience Indicators
2019 Organisational Construction Questionnaire; interview
Förster P., Schachtebeck P.M., Feuerle
T., Hecker P., Branlat M., Herera I.,
Woltjer R
An Approach for Attribute- and Perfor-
mance-Based Evaluation of Interde-
pendent Critical Infrastructures
2019 Organisational Aviation; healthcare Simulation game; focus group
Carpenter A.M Critical infrastructure resilience: A
baseline study for Georgia
2014 Organisational Supply chain management; transporta-
tion systems; emergency response;
water facilities; healthcare; energy
systems; defence
Interview
Line M.B A study of resilience within information
security in the power industry
2013 Organisational Energy systems Interview
Kachali H., Seville E., Vargo J Recovery and resilience of industry and
geographic sectors after the 2010 and
2011 Canterbury earthquakes
2012 Organisational Community Interview
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

367Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Olausson P.M Planning for resilience in the case of
power shortage: The Swedish STY-
REL policy
2019 Organisational Energy systems Questionnaire; interview
Räikkönen M., Molarius R., Mäki K.,
Forssén K., Petiet P., Nieuwenhuijs A
Creating stakeholder value through risk
mitigation measures in the context of
disaster management
2017 Organisational Energy systems AHP
Berggren P., Van Laere J., Johansson
B.J
Using a mixed-methods assessment
approach in a gaming-simulation envi-
ronment to increase resilience
2018 Organisational Payment system Simulation game; questionnaire
Linkov I., Palma-Oliveira J.M An Introduction to resilience for critical
infrastructures
2017 Organisational Not specified Workshop
Cedrini V., Mancini M., Rosi L.,
Mandarino G., Giorgi S., Herrera
I., Branlat M., Pettersson J., Jonson
C.-O., Save L., Ruscio D
Improving resilience management for
critical infrastructures—strategies and
practices across air traffic manage-
ment and healthcare
2018 Organisational Aviation; healthcare Delphi method
Pathirage C The role of knowledge management in
effective disaster mitigation strategies:
Critical infrastructure
2010 Organisational Not specified Qualitative analysis
Save L., Branlat M., Hynes W., Bellini
E., Ferreira P., Lauteritz J.P., Gonza-
lez J.J
The Development of Resilience Man-
agement Guidelines to Protect Critical
Infrastructures in Europe
2019 Organisational Aviation; healthcare Workshop; interview; questionnaire;
simulation game
Woltjer R., Hermelin J., Nilsson S.,
Oskarsson P.-A., Hallberg N
Using requirements engineering in the
development of resilience guidelines
for critical infrastructure
2018 Organisational aviation; healthcare Delphi method; interview; workshop;
focus group
Herrera I., Grøtan T.O., Woltjer R.,
Nevhage B., Nilsson S., Trnka J.,
Adini B., Cohen O., Forsberg R.,
Jonson C.O
Applying resilience concepts in crisis
management and critical infrastruc-
tures—The DARWIN project
2017 Organisational Aviation; healthcare Interview
Petersen L., Lundin E., Fallou L.,
Sjöström J., Lange D., Teixeira R.,
Bonavita A
Resilience for whom? The general pub-
lic’s tolerance levels as CI resilience
criteria
2020 Organisational Water facilities Questionnaire; interview
Storesund K., Reitan N.K., Sjöström J.,
Rød B., Guay F., Almeida R., Theo-
charidou M
Novel methodologies for analysing criti-
cal infrastructure resilience
2018 Organisational Water facilities Focus group; interview; questionnaire
Lange D., Honfi D Novel techniques and approaches for
risk based application of resilience
concepts to critical infrastructure:
An introduction to the IMPROVER
project
2017 Organisational Community Questionnaire
Hsieh C.-H., Tai H.-H., Lee Y.-N Port vulnerability assessment from the
perspective of critical infrastructure
interdependency
2014 Techno-centric Transportation systems Interview; questionnaire; workshop;
AHP; Delphi method
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

368 Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Cutts M., Wang Y., Yu Q New Perspectives on Building Resil-
ience into Infrastructure Systems
2017 Techno-centric Transportation systems; energy
systems; supply chain management;
water facilities; telecommunication
networks
Workshop; focus group
Münzberg T., Wiens M., Schultmann F A spatial–temporal vulnerability
assessment to support the building of
community resilience against power
outage impacts
2017 Techno-centric Energy systems Delphi method; workshop; questionnaire
Petit F.D., Eaton L.K., Fisher R.E.,
McAraw S.F., Collins III M.J
Developing an index to assess the resil-
ience of critical infrastructure
2012 Techno-centric Not specified Questionnaire
Feldpausch-Parker A.M., Peterson T.R.,
Stephens J.C., Wilson E.J
Smart grid electricity system planning
and climate disruptions: A review of
climate and energy discourse post-
Superstorm Sandy
2018 Techno-centric Energy systems Focus group
Münzberg T., Wiens M., Schultmann F The effect of coping capacity depletion
on critical infrastructure resilience
2015 Techno-centric Energy systems Questionnaire
Bloomfield R.E., Popov P., Salako K.,
Stankovic V., Wright D
Preliminary interdependency analysis:
An approach to support critical-infra-
structure risk-assessment
2017 Techno-centric Energy systems; telecommunication
networks
Questionnaire
Sircar I., Sage D., Goodier C., Fussey
P., Dainty A
Constructing Resilient Futures: Integrat-
ing UK multi-stakeholder transport
and energy resilience for 2050
2013 Techno-centric Transportation systems; energy sys-
tems; supply chain management
Interview; focus group
Johnsen S., Skramstad T., Hagen J Enhancing the safety, security and
resilience of ICT and SCADA systems
using action research
2009 Techno-centric Supply chain management Questionnaire
Valiquette L’Heureux A., Therrien M.-C Interorganizational dynamics and char-
acteristics of critical infrastructure
networks: The study of three critical
infrastructures in the greater Montreal
area
2013 Techno-centric Transportation systems; energy sys-
tems; telecommunication networks
Questionnaire
McBurnett L.R., Hinrichs M.M., Seager
T.P., Clark S.S
Simulation Gaming Can Strengthen
Experiential Education in Complex
Infrastructure Systems
2018 Techno-centric Water facilities Workshop; simulation game; interview
Petersen L., Lundin E., Sjöström J.,
Lange D., Teixeira R
Creating comparable public tolerance
and technical performance measures
for critical infrastructure resilience
evaluation
2018 Techno-centric Water facilities Interview; questionnaire
Bernroider E.W.N., Margiol S., Taudes
A
Towards a general information security
management assessment framework
to compare cyber-security of critical
infrastructure organizations
2016 Techno-centric Cybersecurity Workshop; interview; questionnaire
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

369Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Leu P.O., Peter D Case study: Information flow resilience
of a retail company with regard to the
electricity scenarios of the sicher-
heitsverbundsübung schweiz (Swiss
security network exercise) SVU 2014
2016 Techno-centric Supply chain management Interview
Trucco P., Ward D A clustering approach to the operational
resilience analysis of Key Resource
Supply Chains (KRSC): The case of
Fast Moving Consumer Goods
2011 Techno-centric Supply chain management Interview; questionnaire
Lykou G., Anagnostopoulou A., Ster-
giopoulos G., Gritzalis D
Cybersecurity self-assessment tools:
Evaluating the importance for secur-
ing industrial control systems in criti-
cal infrastructures
2019 Techno-centric Cybersecurity; IT networks Questionnaire
Landegren F., Höst M., Möller P Simulation based assessment of
resilience of two large-scale socio-
technical IT networks
2018 Techno-centric IT networks Focus group; interview
Hedel R., Boustras G., Gkotsis I., Vasi-
liadou I., Rathke P
Assessment of the European pro-
gramme for critical infrastructure pro-
tection in the surface transport sector
2018 Techno-centric Transportation systems Workshop; interview; questionnaire
Adelmeyer M., Teuteberg F Cloud computing adoption in critical
infrastructures—Status Quo and ele-
ments of a research agenda
2018 Techno-centric IT networks Interview
Reilly P., Serafinelli E., Stevenson R.,
Petersen L., Fallou L
Enhancing critical infrastructure resil-
ience through information-sharing:
Recommendations for European criti-
cal infrastructure operators
2018 Techno-centric Not specified Interview; focus group
Rajamaki J., Nevmerzhitskaya J., Virag
C
Cybersecurity education and training in
hospitals: Proactive resilience educa-
tional framework (Prosilience EF)
2018 Techno-centric Healthcare; cybersecurity Workshop
Seppänen H., Luokkala P., Zhang Z.,
Torkki P., Virrantaus K
Critical infrastructure vulnerability—A
method for identifying the infrastruc-
ture service failure interdependencies
2018 Techno-centric Energy systems; telecommunication
networks
Workshop; questionnaire
Badea D., Bârsan G., Virca I., Iancu D Quantitative and qualitative differences
worth considering in approaching
critical infrastructures resilience
2017 Techno-centric Not specified AHP
Student L.R., Goubran R., Kwamena F Computer vision-assisted human-in-
the-loop measurements: Augmenting
qualitative by increasing quantitative
analytics for CI situational awareness
2018 Techno-centric energy systems; telecommunication
networks
Social network
Vugrin E.D., Warren D.E., Ehlen M.A.,
Camphouse R.C
A framework for assessing the resil-
ience of infrastructure and economic
systems
2010 Techno-centric Community Qualitative analysis
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

370 Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Vugrin E.D., Warren D.E., Ehlen M.A A resilience assessment framework for
infrastructure and economic systems:
Quantitative and qualitative resilience
analysis of petrochemical supply
chains to a hurricane
2011 Techno-centric Supply chain management Qualitative analysis
Comes T., Van De Walle B Measuring disaster resilience: The
impact of hurricane sandy on critical
infrastructure systems
2014 Techno-centric Transportation systems; energy sys-
tems; supply chain management
Qualitative analysis
Hughes L., de Jong M., Wang X.Q A generic method for analyzing the
risks to energy systems
2016 Techno-centric Energy systems Interview
Beheshtian A., Donaghy K.P., Geddes
R.R., Rouhani O.M
Planning resilient motor-fuel supply
chain
2017 Techno-centric Supply chain management Social network; interview
Matsika E., O’Neill C., Battista U.,
Khosravi M., Laporte A.D.S., Munoz
E
Development of Risk Assessment
Specifications for Analysing Terrorist
Attacks Vulnerability on Metro and
Light Rail Systems
2016 Techno-centric Transportation systems Workshop
Haass M.J., Warrender C.E., Burnham
L., Jeffers R.F., Stevens-Adams S.M.,
Cole K.S., Forsythe C
Toward an Objective Measure of Auto-
mation for the Electric Grid
2015 Techno-centric Energy systems Interview
Riegel C Infrastructure resilience through
regional spatial planning—prospects
of a new legal principle in Germany
2014 Techno-centric Urban infrastructures Qualitative analysis;
Gheorghe A.V., Georgescu A.,
Bucovețchi O., Lazăr M., Scarlat C
New Dimensions for a Challenging
Security Environment: Growing
Exposure to Critical Space Infrastruc-
ture Disruption Risk
2018 Techno-centric Space systems Qualitative analysis
Pfeiffer K.B., Burdi C., Schlueter S Local supply chains: The disaster man-
agement perspective
2017 Techno-centric Energy systems; water facilities; supply
chain management
Questionnaire
Bellini E., Gaitanidou E., Bekiaris E.,
Ferreira P
The RESOLUTE project’s European
Resilience Management Guidelines
for Critical Infrastructure: develop-
ment, operationalisation and testing
for the urban transport system
2020 Urban Transportation systems Focus group
Ferreira P., Bellini E Managing interdependencies in critical
infrastructures—a cornerstone for
system resilience
2018 Urban Transportation systems Interview
Pescaroli G Perceptions of cascading risk and
interconnected failures in emergency
planning: Implications for operational
resilience and policy making
2018 Urban Not specified Workshop; questionnaire;
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

371Environment Systems and Decisions (2021) 41:341–376
1 3
Table 2 (continued)
Authors Title Year Main dimension Domain Approach
Monstadt J., Schmidt M Urban resilience in the making? The
governance of critical infrastructures
in German cities
2019 Urban Urban infrastructures Interview; workshop
Heino O., Takala A., Jukarainen P.,
Kalalahti J., Kekki T., Verho P
Critical infrastructures: The operational
environment in cases of severe disrup-
tion
2019 Urban Energy systems; water facilities Workshop
Dierich A., Tzavella K., Setiadi N.J.,
Fekete A., Neisser F
Enhanced crisis-preparation of critical
infrastructures through a participatory
qualitative-quantitative interdepend-
ency analysis approach
2019 Urban Energy systems; water facilities; emer-
gency response
Interview; focus group
Gonzalez J.J., Bång M., Eden C.,
Gimenez R., Hernantes J., Howick
S., Maraña P., Pyrko I., Radianti J.,
Rankin A., Sarriegi J.M
Stalking resilience: Cities as vertebrae
in society’s resilience backbone
2017 Urban Urban infrastructures Workshop; Delphi method; questionnaire
Lomba-Fernández C., Hernantes J.,
Labaka L
Guide for climate-resilient cities: An
urban critical infrastructures approach
2019 Urban Urban infrastructures Focus group; workshop; interview
Fekete A., Fiedrich F Synthesis 2018 Urban Urban infrastructures Workshop; focus group; interview
Räikkönen M., Mäki K., Murtonen
M., Forssén K., Tagg A., Petiet P.J.,
Nieuwenhuijs A.H., McCord M
A holistic approach for assessing impact
of extreme weather on critical infra-
structure
2016 Urban Not specified Qualitative analysis
Fekete A., Bogardi J.J Considerations about urban disaster
resilience and security—Two concepts
in tandem?
2018 Urban Urban infrastructures Qualitative analysis
Marana P., Eden C., Eriksson H.,
Grimes C., Hernantes J., Howick S.,
Labaka L., Latinos V., Lindner R.,
Majchrzak T.A., Pyrko I., Radianti J.,
Rankin A., Sakurai M., Sarriegi J.M.,
Serrano N
Towards a resilience management
guideline — Cities as a starting point
for societal resilience
2019 Urban Urban infrastructures Questionnaire
Bellini E., Nesi P., Pantaleo G., Venturi
A
Functional resonance analysis method
based-decision support tool for urban
transport system resilience manage-
ment
2016 Urban Transportation systems AHP; interview
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

372 Environment Systems and Decisions (2021) 41:341–376
1 3
Funding Open Access funding provided by Università degli Studi di
Roma La Sapienza.
Compliance with ethical standards
Conflict of interest All authors declare that they have no conflict of
interest.
Open Access This article is licensed under a Creative Commons Attri-
bution 4.0 International License, which permits use, sharing, adapta-
tion, distribution and reproduction in any medium or format, as long
as you give appropriate credit to the original author(s) and the source,
provide a link to the Creative Commons licence, and indicate if changes
were made. The images or other third party material in this article are
included in the article’s Creative Commons licence, unless indicated
otherwise in a credit line to the material. If material is not included in
the article’s Creative Commons licence and your intended use is not
permitted by statutory regulation or exceeds the permitted use, you will
need to obtain permission directly from the copyright holder. To view a
copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.
References
Adini B, Cohen O, Eide AW etal (2017) Striving to be resilient:
What concepts, approaches and practices should be incor-
porated in resilience management guidelines? Technol Fore-
cast Soc Change 121:39–49. https ://doi.org/10.1016/j.techf
ore.2017.01.020
Akhtar R, Santos JR (2013) Risk-based input-output analysis of hurri-
cane impacts on interdependent regional workforce systems. Nat
Hazards 65:391–405. https ://doi.org/10.1007/s1106 9-012-0369-0
Antunes D, Palma-Oliveira JM, Linkov I (2017) Enhancing organiza-
tional resilience through risk communication: Basic guidelines
for managers. NATO Sci Peace Secur Ser C Environ Secur Part
F1:469–481. https ://doi.org/10.1007/978-94-024-1123-2_18
Badea D, Bârsan G, Virca I, Iancu D (2017) Quantitative and qualita-
tive differences worth considering in approaching critical infra-
structures resilience. MATEC Web Conf. https ://doi.org/10.1051/
matec conf/20171 21110 04
Bandecchi AE, Pazzi V, Morelli S etal (2019) Geo-hydrological and
seismic risk awareness at school: Emergency preparedness and
risk perception evaluation. Int J Disaster Risk Reduct. https ://doi.
org/10.1016/j.ijdrr .2019.10128 0
Banks LH, Davenport LA, Hayes MH etal (2016) Disaster Impact
on Impoverished Area of US: An Inter-Professional Mixed
Method Study. Prehosp Disaster Med 31:583–592. https ://doi.
org/10.1017/S1049 023X1 60009 0X
Beheshtian A, Donaghy KP, Geddes RR, Rouhani OM (2017) Plan-
ning resilient motor-fuel supply chain. Int J Disaster Risk Reduct
24:312–325. https ://doi.org/10.1016/j.ijdrr .2017.06.021
Bellini E, Gaitanidou E, Bekiaris E, Ferreira P (2020) The RESOLUTE
project’s European Resilience Management Guidelines for Criti-
cal Infrastructure: development, operationalisation and testing
for the urban transport system. Environ Syst Decis. https ://doi.
org/10.1007/s1066 9-020-09765 -0
Bellini E, Nesi P, Pantaleo G, Venturi A (2016) Functional resonance
analysis method based-decision support tool for urban transport
system resilience management. In: 2nd IEEE International Smart
Cities Conference, ISC2 2016. Institute of Electrical and Elec-
tronics Engineers Inc., Distributed System and Internet Technol-
ogy Lab, Italy
Berggren P, Van Laere J, Johansson BJ (2018) Using a mixed-methods
assessment approach in a gaming-simulation environment to
increase resilience
Bergström J, Dekker SWA (2014) Bridging the macro and the micro
by considering the meso: Reflections on the fractal nature of
resilience. Ecol Soc 19:36–42. https ://doi.org/10.5751/ES-06956
-19042 2
Bernroider EWN, Margiol S, Taudes A (2016) Towards a general
information security management assessment framework to
compare cyber-security of critical infrastructure organizations
268:127–141
Bloomfield RE, Popov P, Salako K etal (2017) Preliminary interde-
pendency analysis: An approach to support critical-infrastructure
risk-assessment. Reliab Eng Syst Saf 167:198–217. https ://doi.
org/10.1016/j.ress.2017.05.030
Bouchon S, Urquhart J, Gibson R, etal (2014) The role of public-pri-
vate stakeholder collaboration to achieve critical infrastructures
resilience: Main findings from the EU-CIPS MIRACLE project.
pp 105–108
Brown C, Seville E, Vargo J (2017) Measuring the organizational
resilience of critical infrastructure providers: A New Zealand
case study. Int J Crit Infrastruct Prot 18:37–49. https ://doi.
org/10.1016/j.ijcip .2017.05.002
Bruneau M, Chang SE, Eguchi RT etal (2003) A Framework to
quantitatively assess and enhance the seismic resilience
of communities. Earthq Spectra 19:733–752. https ://doi.
org/10.1193/1.16234 97
Bunney SL, Ward S, Butler D (2016) Can UK water service pro-
viders manage risk and resilience as part of a multi-agency
approach? pp 111–114
Burbidge R (2016) Adapting European Airports to a Changing Cli-
mate. In: Transportation Research Procedia. pp 14–23
Carpenter AM (2014) Critical infrastructure resilience: A baseline
study for Georgia. pp 186–197
Cedrini V, Mancini M, Rosi L, etal (2018) Improving resilience
management for critical infrastructures—strategies and prac-
tices across air traffic management and healthcare. In: Safety
and Reliability - Safe Societies in a Changing World - Proceed-
ings of the 28th International European Safety and Reliability
Conference, ESREL 2018. pp 1319–1330
Cegan JC, Filion AM, Keisler JM, Linkov I (2017) Trends and appli-
cations of multi-criteria decision analysis in environmental sci-
ences: literature review. Environ Syst Decis 37:123–133. https
://doi.org/10.1007/s1066 9-017-9642-9
Comes T, Bertsch V, French S (2013) Designing dynamic stress tests
for improved critical infrastructure resilience. pp 307–311
Comes T, Van De Walle B (2014) Measuring disaster resilience: The
impact of hurricane sandy on critical infrastructure systems.
pp 195–204
Cradock-Henry NA, Fountain J, Buelow F (2018) Transformations
for resilient rural futures: The case of Kaikōura. Aotearoa-New
Zealand Sustain. https ://doi.org/10.3390/su100 61952
Cutts M, Wang Y, Yu Q (2017) New Perspectives on Building Resil-
ience into Infrastructure Systems. Nat Hazards Rev. https ://doi.
org/10.1061/(ASCE)NH.1527-6996.00002 03
Dierich A, Tzavella K, Setiadi NJ, etal (2019) Enhanced crisis-
preparation of critical infrastructures through a participatory
qualitative-quantitative interdependency analysis approach. pp
1226–1244
Doyle A, Hynes W, Purcell SM, Rochford M (2018) Exploring the
role of planning in urban resilience enhancement—An irish
perspective. Urban B Ser. https ://doi.org/10.1007/978-3-319-
68606 -6_3
Elo S, Kyngäs H (2008) The qualitative content analysis pro-
cess. J Adv Nurs 62:107–115. https ://doi.org/10.111
1/j.1365-2648.2007.04569 .x
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

373Environment Systems and Decisions (2021) 41:341–376
1 3
Elsevier (2018) Scopus, an eye on global research
Fekete A, Bogardi JJ (2018) Considerations about urban disaster resil-
ience and security—Two concepts in tandem? Ser, Urban B.
https ://doi.org/10.1007/978-3-319-68606 -6_28
Fekete A, Fiedrich F (2018) Synthesis. Urban B. Ser. 503–518
Feldpausch-Parker AM, Peterson TR, Stephens JC, Wilson EJ (2018)
Smart grid electricity system planning and climate disruptions:
A review of climate and energy discourse post-Superstorm
Sandy. Renew Sustain Energy Rev 82:1961–1968. https ://doi.
org/10.1016/j.rser.2017.06.015
Ferreira P, Bellini E (2018) Managing interdependencies in critical
infrastructures—a cornerstone for system resilience. In: Safety
and Reliability - Safe Societies in a Changing World - Proceed-
ings of the 28th International European Safety and Reliability
Conference, ESREL 2018
Field C, Look R (2018) A value-based approach to infrastructure resil-
ience. Environ Syst Decis 38:292–305. https ://doi.org/10.1007/
s1066 9-018-9701-x
Förster P, Schachtebeck PM, Feuerle T etal (2019) An Approach for
Attribute- and Performance-Based Evaluation of Interdependent
Critical Infrastructures 555:35–50
Fox-Lent C, Bates ME, Linkov I (2015) A matrix approach to commu-
nity resilience assessment: an illustrative case at Rockaway Pen-
insula. Environ Syst Decis 35:209–218. https ://doi.org/10.1007/
s1066 9-015-9555-4
Garcia AP, Siaterlis C, Masera M (2010) Testing the fidelity of an
Emulab testbed. pp 307–312
Gasser P, Lustenberger P, Cinelli M etal (2019) A review on resil-
ience assessment of energy systems. Sustain Resilient Infra-
struct. https ://doi.org/10.1080/23789 689.2019.16106 00
Gay LF, Sinha SK (2013) Resilience of civil infrastructure systems:
Literature review for improved asset management. Int J Crit
Infrastructures 9:330–350. https ://doi.org/10.1504/IJCIS
.2013.05817 2
Gheorghe AV, Georgescu A, Bucovețchi O etal (2018) New Dimen-
sions for a Challenging Security Environment: Growing
Exposure to Critical Space Infrastructure Disruption Risk. Int
J Disaster Risk Sci 9:555–560. https ://doi.org/10.1007/s1375
3-018-0197-2
Glavovic BC (2008) Sustainable coastal communities in the age of
coastal storms: Reconceptualising coastal planning as “new”
naval architecture. J Coast Conserv 12:125–134. https ://doi.
org/10.1007/s1185 2-008-0037-4
Golan MS, Jernegan LH, Linkov I (2020) Trends and applications
of resilience analytics in supply chain modeling: systematic
literature review in the context of the COVID-19 pandemic.
Environ Syst Decis 40:222–243. https ://doi.org/10.1007/s1066
9-020-09777 -w
Gonzalez JJ, Bång M, Eden C etal (2017) Stalking resilience: Cities
as vertebrae in society’s resilience backbone 501:31–45
Grosse C (2019) Airports as Critical Infrastructure: The Role of the
Transportation-by-Air System for Regional Development and
Crisis Management. pp 440–444
Große C (2019) Sources of uncertainty in Swedish emergency
response planning. J Risk Res 22:758–772. https ://doi.
org/10.1080/13669 877.2018.14597 96
Haass MJ, Warrender CE, Burnham L etal (2015) Toward an
Objective Measure of Automation for the Electric Grid. Pro-
cedia Manuf 3:5285–5292. https ://doi.org/10.1016/j.promf
g.2015.07.611
Häring I, Sansavini G, Bellini E etal (2017) Towards a generic
resilience management, quantification and development
process: General definitions, requirements, methods, tech-
niques and measures, and case studies. NATO Sci Peace
Secur Ser C Environ Secur Part F1:21–80. https ://doi.
org/10.1007/978-94-024-1123-2_2
Hedel R, Boustras G, Gkotsis I etal (2018) Assessment of the european
programme for critical infrastructure protection in the surface
transport sector. Int J Crit Infrastructures 14:311–335. https ://
doi.org/10.1504/IJCIS .2018.09561 6
Heglund J, Hopkinson KM, Tran HT (2020) Social sensing: towards
social media as a sensor for resilience in power systems and other
critical infrastructures. Sustain Resilient Infrastruct. https ://doi.
org/10.1080/23789 689.2020.17197 28
Heino O, Takala A, Jukarainen P etal (2019) Critical infrastructures:
The operational environment in cases of severe disruption. Sus-
tain. https ://doi.org/10.3390/su110 30838
Herrera I, Grøtan TO, Woltjer R, etal (2017) Applying resilience
concepts in crisis management and critical infrastructures—The
DARWIN project. In: L. W, M. R, T. B (eds) 26th European
Safety and Reliability Conference, ESREL 2016. CRC Press/
Balkema, SINTEF, Trondheim, Norway, p 349
Herrmann Lunecke MG (2015) Urban planning and tsunami impact
mitigation in Chile after February 27, 2010. Nat Hazards
79:1591–1620. https ://doi.org/10.1007/s1106 9-015-1914-4
Hiete M, Merz M, Schultmann F (2011) Scenario-based impact
analysis of a power outage on healthcare facilities in Ger-
many. Int J Disaster Resil Built Environ 2:222–244. https ://doi.
org/10.1108/17595 90111 11671 05
Holling CS (1973) Resilience and Stability of Ecological Systems.
Annu Rev Ecol Syst 4:1–23
Hosseini S, Barker K, Ramirez-Marquez JE (2016) A review of defi-
nitions and measures of system resilience. Reliab Eng Syst Saf
145:47–61. https ://doi.org/10.1016/j.ress.2015.08.006
Hsieh C-H, Tai H-H, Lee Y-N (2014) Port vulnerability assess-
ment from the perspective of critical infrastructure inter-
dependency. Marit Policy Manag 41:589–606. https ://doi.
org/10.1080/03088 839.2013.85652 3
Hughes L, de Jong M, Wang XQ (2016) A generic method for ana-
lyzing the risks to energy systems. Appl Energy 180:895–908.
https ://doi.org/10.1016/j.apene rgy.2016.07.133
Ingrisch J, Bahn M (2018) Towards a Comparable Quantification
of Resilience. Trends Ecol Evol 33:251–259. https ://doi.
org/10.1016/j.tree.2018.01.013
Iturriza M, Abdelgawad AA, Labaka L etal (2017) Smart mature
resilience, system dynamics based interactive learning environ-
ment: A beta version. Int J Saf Secur Eng 7:367–379. https ://
doi.org/10.2495/SAFE-V7-N3-367-379
Johansson BJE, Jaber A, Van Laere J, Berggren P (2018) The lack of
preparedness for payment disruptions in local community core
businesses. pp 893–902
Johnsen S, Skramstad T, Hagen J (2009) Enhancing the safety, secu-
rity and resilience of ICT and SCADA systems using action
research. 311:113–124
Johnsen SO, Øren A (2015) 10 years from risk assessment to regu-
latory action—is complacency creating a reactive and brittle
regulatory regime in Norway? pp 3333–3339
Kachali H, Seville E, Vargo J (2012a) Recovery and resilience of
industry and geographic sectors after the 2010 and 2011 can-
terbury earthquakes. pp 356–360
Kachali H, Stevenson JR, Whitman Z etal (2012) Organisational
Resilience and Recovery for Canterbury Organisations after
the 4 September 2010 Earthquake. Australas J Disaster Trauma
Stud 2012:11–19
Kete N, Punzo G, Linkov I (2018) Enhancing resilience within and
between critical infrastructure systems. Environ Syst Decis
38:275–277. https ://doi.org/10.1007/s1066 9-018-9706-5
Kröger W, Zio E (2011) Vulnerable Systems
Kurth MH, Larkin S, Keisler JM, Linkov I (2017) Trends and appli-
cations of multi-criteria decision analysis: use in govern-
ment agencies. Environ Syst Decis 37:134–143. https ://doi.
org/10.1007/s1066 9-017-9644-7
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

374 Environment Systems and Decisions (2021) 41:341–376
1 3
Labaka L, Comes T, Hernantes J, etal (2014) Implementation meth-
odology of the resilience framework. pp 139–148
LaLone N, Tapia A, Zobel C etal (2017) Embracing human noise
as resilience indicator: twitter as power grid correlate. Sustain
Resilient Infrastruct 2:169–178. https ://doi.org/10.1080/23789
689.2017.13289 20
Landegren F, Höst M, Möller P (2018) Simulation based assessment
of resilience of two large-scale socio-technical IT networks. Int
J Crit Infrastruct Prot 23:112–125. https ://doi.org/10.1016/j.
ijcip .2018.08.003
Lange D, Honfi D (2017) Novel techniques and approaches for risk
based application of resilience concepts to critical infrastruc-
ture: An introduction to the IMPROVER project. In: L. W, M.
R, T. B (eds) 26th European Safety and Reliability Conference,
ESREL 2016. CRC Press/Balkema, SP Technical Research
Institute of Sweden, Sweden, p 353
Leu PO, Peter D (2016) Case study: Information flow resilience of
a retail company with regard to the electricity scenarios of
the sicherheitsverbundsübung schweiz (swiss security network
exercise) SVU 2014. 9578:159–170
Line MB (2013) A study of resilience within information security in
the power industry. In: IEEE AFRICON Conference. Dep. of
Telematics, Norwegian University of Science and Technology
(NTNU), Trondheim, Norway
Linkov I, Eisenberg DA, Plourde K etal (2013) Resilience metrics
for cyber systems. Environ Syst Decis 33:471–476. https ://doi.
org/10.1007/s1066 9-013-9485-y
Linkov I, Fox-Lent C, Read L etal (2018) Tiered approach to
resilience assessment. Risk Anal 38:1772–1780. https ://doi.
org/10.1111/risa.12991
Linkov I, Palma-Oliveira JM (2017) An Introduction to resilience for
critical infrastructures. NATO Sci. Peace Secur. Ser C Environ
Secur Part F1:3–17
Lomba-Fernández C, Hernantes J, Labaka L (2019) Guide for
climate-resilient cities: An urban critical infrastructures
approach. Sustain. https ://doi.org/10.3390/su111 74727
Lykou G, Anagnostopoulou A, Stergiopoulos G, Gritzalis D (2019)
Cybersecurity self-assessment tools: Evaluating the impor-
tance for securing industrial control systems in critical infra-
structures. 11260 LNCS:129–142
Marana P, Eden C, Eriksson H etal (2019) Towards a resilience
management guideline — Cities as a starting point for soci-
etal resilience. Sustain Cities Soc. https ://doi.org/10.1016/j.
scs.2019.10153 1
Matsika E, O’Neill C, Battista U etal (2016) Development of Risk
Assessment Specifications for Analysing Terrorist Attacks Vul-
nerability on Metro and Light Rail Systems. In: Transportation
Research Procedia. 14:1345–1354
Mavhura E (2017) Applying a systems-thinking approach to com-
munity resilience analysis using rural livelihoods: The case
of Muzarabani district, Zimbabwe. Int J Disaster Risk Reduct
25:248–258. https ://doi.org/10.1016/j.ijdrr .2017.09.008
McBurnett LR, Hinrichs MM, Seager TP, Clark SS (2018) Simula-
tion Gaming Can Strengthen Experiential Education in Com-
plex Infrastructure Systems. Simul Gaming 49:620–641. https
://doi.org/10.1177/10468 78118 76772 9
Mendonça D, Wallace WA (2015) Factors underlying organizational
resilience: The case of electric power restoration in New York
City after 11 September 2001. Reliab Eng Syst Saf 141:83–91
Millwood S (2019) What space missions can learn from cyber-secu-
rity breaches and countermeasures in the telecommunications
industry
Mohebbi S, Zhang Q, Christian Wells E etal (2020) Cyber-phys-
ical-social interdependencies and organizational resilience:
A review of water, transportation, and cyber infrastruc-
ture systems and processes. Sustain Cities Soc. https ://doi.
org/10.1016/j.scs.2020.10232 7
Moher D, Liberati A, Tetzlaff J etal (2009) Preferred reporting items
for systematic reviews and meta-analyses: The PRISMA state-
ment. PLoS Med. https ://doi.org/10.1371/journ al.pmed.10000
97
Moher D, Liberati A, Tetzlaff J, Altman DG (2009) The PRISMA
Group (2009) Preferred Reporting Items for Systematic
Reviews and Meta-Analyses: The PRISMA Statement. PLOS
Med 6:1–6. https ://doi.org/10.1371/journ al.pmed.10000 97
Monstadt J, Schmidt M (2019) Urban resilience in the making? The
governance of critical infrastructures in German cities. Urban
Stud 56:2353–2371. https ://doi.org/10.1177/00420 98018 80848
3
Münzberg T, Wiens M, Schultmann F (2015) The effect of coping
capacity depletion on critical infrastructure resilience
Münzberg T, Wiens M, Schultmann F (2017) A spatial-temporal
vulnerability assessment to support the building of commu-
nity resilience against power outage impacts. Technol Fore-
cast Soc Change 121:99–118. https ://doi.org/10.1016/j.techf
ore.2016.11.027
Murphy SA, Brown J, Shankar A, Lichtveld M (2019) A quantitative
assessment of institutions of higher education disaster prepar-
edness and resilience. J Emerg Manag 17:239–250. https ://doi.
org/10.5055/jem.2019.0423
Olausson PM (2019) Planning for resilience in the case of power short-
age: The Swedish STYREL policy. Cent Eur J Public Policy.
https ://doi.org/10.2478/cejpp -2019-0004
Ong JM, Jamero ML, Esteban M etal (2016) Challenges in Build-
Back-Better Housing Reconstruction Programs for Coastal Dis-
aster Management: Case of Tacloban City. Philippines Coast Eng
J. https ://doi.org/10.1142/S0578 56341 64001 06
Onwuegbuzie AJ, Leech NL, Collins KMT (2011) The SAGE Hand-
book of Innovation in Social Research Methods
Pagán-Trinidad I, Lopez RR, Diaz EL (2019) Education and building
capacity for improving resilience of coastal infrastructure
Pathirage C (2010) The role of knowledge management in effective
disaster mitigation strategies: Critical infrastructure
Pathirage C, Al-Khaili K (2016) Disaster vulnerability of Emirati
energy sector and barriers to enhance resilience. Built Environ
Proj Asset Manag 6:403–414. https ://doi.org/10.1108/BEPAM
-08-2015-0035
Patriarca R, Falegnami A, Costantino F, Di Gravio G, De Nicola A,
Villani ML, (2021) WAx: An integrated conceptual framework
for the analysis of cyber-socio-technical systems. Safety Science
136:105142
Patton MQ (2002) Qualitative research and evaluation methods, 3rd
editio. Sage Publications, Thousand Oaks, CA
Pescaroli G (2018) Perceptions of cascading risk and interconnected
failures in emergency planning: Implications for operational
resilience and policy making. Int J Disaster Risk Reduct 30:269–
280. https ://doi.org/10.1016/j.ijdrr .2018.01.019
Petersen L, Fallou L, Carreira E, Utkin A (2018) Public tolerance lev-
els of transportation resilience: A focus on the Oresund region
within the IMPROVER project. 10707 LNCS:13–24
Petersen L, Fallou L, Reilly P, Serafinelli E (2020) Public expecta-
tions of critical infrastructure operators in times of crisis. Sus-
tain Resilient Infrastruct 5:62–77. https ://doi.org/10.1080/23789
689.2018.14693 58
Petersen L, Havarneanu G, Markarian G, McCroneline N (2019a)
CBRNe and vulnerable citizens: Co-creating an app for that.
In: Y. H-A (ed) 6th International Conference on Information
and Communication Technologies for Disaster Management,
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

375Environment Systems and Decisions (2021) 41:341–376
1 3
ICT-DM 2019. Institute of Electrical and Electronics Engineers
Inc.
Petersen L, Lange D, Theocharidou M (2020) Who cares what it
means? Practical reasons for using the word resilience with crit-
ical infrastructure operators. Reliab Eng Syst Saf. https ://doi.
org/10.1016/j.ress.2020.10687 2
Petersen L, Lundin E, Fallou L etal (2020) Resilience for whom? The
general public’s tolerance levels as CI resilience criteria. Int J
Crit Infrastruct Prot. https ://doi.org/10.1016/j.ijcip .2020.10034 0
Petersen L, Sjöström J, Horvath E (2019b) Evaluating critical infra-
structure resilience via tolerance triangles: Hungarian Highway
pilot case study. pp 1210–1225
Petit FD, Eaton LK, Fisher RE etal (2012) Developing an index to
assess the resilience of critical infrastructure. Int J Risk Assess
Manag 16:28–47. https ://doi.org/10.1504/IJRAM .2012.04755 1
Pfeiffer KB, Burdi C, Schlueter S (2017) Local supply chains: The
disaster management perspective. Int J Saf Secur Eng 7:399–405.
https ://doi.org/10.2495/SAFE-V7-N3-399-405
Räikkönen M, Mäki K, Murtonen M etal (2016) A holistic approach
for assessing impact of extreme weather on critical infrastruc-
ture. Int J Saf Secur Eng 6:171–180. https ://doi.org/10.2495/
SAFE-V6-N2-171-180
Räikkönen M, Molarius R, Mäki K etal (2017) Creating stakeholder
value through risk mitigation measures in the context of disaster
management. Infrastructures. https ://doi.org/10.3390/infra struc
tures 20400 14
Rajamaki J, Nevmerzhitskaya J, Virag C (2018) Cybersecurity educa-
tion and training in hospitals: Proactive resilience educational
framework (Prosilience EF). pp 2042–2046
Rand K, Fleming CH (2019) An interdisciplinary review to develop
guidelines for modeling population displacement as a function
of infrastructure reconstruction decisions. Transp Res Interdiscip
Perspect. https ://doi.org/10.1016/j.trip.2019.10007 2
Rand K, Kurth M, Fleming CH, Linkov I (2020) A resilience matrix
approach for measuring and mitigating disaster-induced popu-
lation displacement. Int J Disaster Risk Reduct. https ://doi.
org/10.1016/j.ijdrr .2019.10131 0
Rehana S, Rajulapati CR, Ghosh S etal (2020) Uncertainty quantifi-
cation in water resource systems modeling: Case studies from
India. Water (Switzerland). https ://doi.org/10.3390/w1206 1793
Reilly P, Serafinelli E, Stevenson R, etal (2018) Enhancing critical
infrastructure resilience through information-sharing: Rec-
ommendations for European critical infrastructure operators.
10766 LNCS:120–125
Riegel C (2014) Infrastructure resilience through regional spatial
planning - prospects of a new legal principle in Germany. Int
J Crit Infrastructures 10:17–29. https ://doi.org/10.1504/IJCIS
.2014.05954 1
Rogers L, Borges D, Murray J, etal (2019) NASA’s Mid-Atlantic
Communities and Areas at Intensive Risk Demonstration: :
Translating Compounding Hazards to Societal Risk
Rosenqvist H, Reitan NK, Petersen L, Lange D (2018) ISRA:
IMPROVER societal resilience analysis for critical infrastruc-
ture. In: Haugen S. Barros A. VJE van GCKT (ed) Safety and
Reliability - Safe Societies in a Changing World - Proceed-
ings of the 28th International European Safety and Reliability
Conference, ESREL 2018. CRC Press/Balkema, pp 1211–1220
Saaty TL (1990) How to make a decision: The analytic hierarchy
process. Eur J Oper Res 48:9–26. https ://doi.org/10.1016/0377-
2217(90)90057 -I
Sapeciay Z, Wilkinson S, Costello SB, Adnan H (2019) Building
Organisational Resilience for the Construction Industy: Stra-
tegic Resilience Indicators
Save L, Branlat M, Hynes W, et al (2019) The Development of
Resilience Management Guidelines to Protect Critical Infra-
structures in Europe. 20th Congr. Int. Ergon. Assoc. IEA 2018
819:598–606
Seppänen H, Luokkala P, Zhang Z etal (2018) Critical infrastructure
vulnerability—A method for identifying the infrastructure ser-
vice failure interdependencies. Int J Crit Infrastruct Prot 22:25–
38. https ://doi.org/10.1016/j.ijcip .2018.05.002
Shuang Q, Liu HJ, Porse E (2019) Review of the quantitative resilience
methods in water distribution networks. Water (Switzerland).
https ://doi.org/10.3390/w1106 1189
Simpson DM, Lasley CB, Rockaway TD, Weigel TA (2010) Under-
standing critical infrastructure failure: Examining the experience
of Biloxi and Gulfport, Mississippi after Hurricane Katrina. Int
J Crit Infrastructures 6:246–276. https ://doi.org/10.1504/IJCIS
.2010.03333 9
Sircar I, Sage D, Goodier C etal (2013) Constructing Resilient Futures:
Integrating UK multi-stakeholder transport and energy resilience
for 2050. Futures 49:49–63. https ://doi.org/10.1016/j.futur
es.2013.04.003
Storesund K, Reitan NK, Sjöström J, etal (2018) Novel methodolo-
gies for analysing critical infrastructure resilience. In: C. van G,
S. H, A. B, etal. (eds) 28th International European Safety and
Reliability Conference, ESREL 2018. CRC Press/Balkema, RISE
Fire Research, Trondheim, Norway, pp 1221–1230
Student LR, Goubran R, Kwamena F (2018) Computer vision-assisted
human-in-the-loop measurements: Augmenting qualitative by
increasing quantitative analytics for CI situational awareness
Trucco P, Ward D (2011) A clustering approach to the operational
resilience analysis of Key Resource Supply Chains (KRSC): The
case of Fast Moving Consumer Goods. pp 990–994
Valiquette L’Heureux A, Therrien M-C (2013) Interorganizational
dynamics and characteristics of critical infrastructure networks:
The study of three critical infrastructures in the greater mon-
treal area. J Contingencies Cris Manag 21:211–224. https ://doi.
org/10.1111/1468-5973.12030
Van Laere J, Berggren P, Gustavsson P, etal (2017) Challenges for
critical infrastructure resilience: Cascading effects of payment
system disruptions. pp 281–292
Veettil BK, Ward RD, Lima MDAC etal (2020) Opportunities for
seagrass research derived from remote sensing: A review of
current methods. Ecol Indic. https ://doi.org/10.1016/j.ecoli
nd.2020.10656 0
Verma R, Karimi S, Lee D, etal (2019) Newswire versus Social Media
for Disaster Response and Recovery. pp 132–141
Vugrin ED, Warren DE, Ehlen MA (2010a) A resilience assessment
framework for infrastructure and economic systems: Quantita-
tive and qualitative resilience analysis of petrochemical supply
chains to a hurricane
Vugrin ED, Warren DE, Ehlen MA, Camphouse RC (2010b) A frame-
work for assessing the resilience of infrastructure and economic
systems. In: Sustainable and Resilient Critical Infrastructure
Systems: Simulation, Modeling, and Intelligent Engineering. pp
77–116
Woltjer R, Hermelin J, Nilsson S etal (2018) Using requirements engi-
neering in the development of resilience guidelines for critical
infrastructure. 13th System of Systems Engineering Conference,
SoSE 2018. Institute of Electrical and Electronics Engineers Inc.,
Swedish Defense Research Agency (FOI), Linköping and Stock-
holm, Sweden, pp 615–622
Wood MD, Wells EM, Rice G, Linkov I (2019) Quantifying and map-
ping resilience within large organizations. Omega (United King-
dom) 87:117–126. https ://doi.org/10.1016/j.omega .2018.08.012
Yanow D (2012) Organizational ethnography between toolbox and
world-making. J Organ Ethnogr. https ://doi.org/10.1108/20246
67412 11220 633
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

376 Environment Systems and Decisions (2021) 41:341–376
1 3
Zemba V, Wells EM, Wood MD et al (2019) Defining, measuring,
and enhancing resilience for small groups. Saf Sci 120:603–616.
https ://doi.org/10.1016/j.ssci.2019.07.042
Zio E (2016) Challenges in the vulnerability and risk analysis of critical
infrastructures. Reliab Eng Syst Saf 152:137–150. https ://doi.
org/10.1016/j.ress.2016.02.009
Publisher’s note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

1.
2.
3.
4.
5.
6.
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:
use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
use bots or other automated methods to access the content or redirect messages
override any security feature or exclusionary protocol; or
share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at
onlineservice@springernature.com