Content uploaded by Narayan Kumar Shrestha
Author content
All content in this area was uploaded by Narayan Kumar Shrestha on Nov 02, 2020
Content may be subject to copyright.
Advancing model calibration and uncertainty analysis
of SWAT models using cloud computing infrastructure:
LCC-SWAT
Masood Zamani, Narayan Kumar Shrestha, Taimoor Akhtar,
Trevor Boston and Prasad Daggupati
Q1
ABSTRACT
Calibration and uncertainty analysis of a complex, over-parameterized environmental models such as
the Soil and Water Assessment Tool (SWAT) requires thousands of simulation runs and multiple
calibration iterations. A parallel calibration system is thus desired that can be deployed on cloud-
based architectures for reducing calibration runtime. This paper presents a cloud-based calibration
and uncertainty analysis system called LCC-SWAT that is designed for SWAT models. Two
optimization techniques, sequential uncertainty fitting (SUFI-2) and dynamically dimensioned search
(DDS), have been implemented in LCC-SWAT. Moreover, the cloud-based system has been deployed
on the Southern Ontario Smart Computing Innovation Platform’s (SOSCIP) Cloud Analytics platform
for diagnostic assessment of parallel calibration runtime on both single-node and multi-node CPU
architectures. Unlike other calibrations/uncertainty analysis systems developed on the cloud, this
system is capable of generating a comprehensive set of statistical information automatically, which
facilitates broader analyses of the performance of the SWAT models. Experimental results on SWAT
models of different complexities showed that LCC-SWAT can reduce runtime significantly. The
runtime reduction is more pronounced for more complex and computationally intensive models.
However, the reported runtime efficiency is significantly higher for single node systems. Comparative
experiments with DDS and SUFI-2 show that parallel DDS outperforms parallel SUFI-2 in terms of
both parameter identifiability and reducing uncertainty in model simulations. LCC-SWAT is a flexible
calibration system and other optimization algorithms and asynchronous parallelization strategies can
be added to it in future.
Key words |cloud computing, DDS, optimization, SUFI-2, SWAT
HIGHLIGHTS
•LCC-SWAT: a cloud-based calibration and uncertainty analysis system for SWAT models.
•Two optimization techniques SUFI-2 and DDS have been implemented in the LCC-SWAT.
•LCC-SWAT is capable of generating a comprehensive set of statistical information.
•Experiments showed that LCC-SWAT can reduce runtime significantly.Q2
Masood Zamani
Narayan Kumar Shrestha
Taimoor Akhtar
Prasad Daggupati (corresponding author)
School of Engineering,
University of Guelph,
50 Stone Road West, Guelph, ON,
Canada
N1G 2W1
E-mail: pdaggupa@uoguelph.ca
Masood Zamani
St.Michael’s Hospital,
University of Toronto,
1 King’s College Circle, Toronto, ON,
Canada
M5S 1A8
Trevor Boston
Greenland International Consulting Ltd,
120 Hume Street, Collingwood, ON,
Canada
L9Y 1V5
1© IWA Publishing 2020 Journal of Hydroinformatics |in press |2020
doi: 10.2166/hydro.2020.066
Uncorrected Proof
GRAPHICAL ABSTRACT
INTRODUCTION
Watershed models are widely used by water resources plan-
ners and managers in the decision-making process (Devia
et al. ;Leta et al. ). With easier access to parallel
computing power and the ready availability of higher-resol-
ution observed datasets (including climate data, water
quality data, etc.), the computational complexity of water-
shed models, and especially physically based distributed
and semi-distributed watershed models is increasing signifi-
cantly (Yang et al. ). Moreover, complex physically
based watershed models are typically characterized by a
large number of parameters, and complex calibration objec-
tives that are highly non-linear and multi-modal (Beven &
Binley ;Abbaspour et al. ;Tolson & Shoemaker
). Hence, automatic parameter estimation of a complex
watershed model is often hindered by high-dimensionality
and multi-modality of the underlying calibration optimiz-
ation problem (Nossent et al. ), and thus, the
calibration process is computationally intensive (Ercan
et al. ). The calibration challenge is exacerbated further
by the need for understanding model uncertainty,
especially for models that simulate water quality (e.g., maxi-
mum daily pollutants loads) (Shirmohammadi et al. ;
Borah et al. ). The computationally intensive nature
of complex watershed models, thus, necessitates
optimization algorithms and frameworks that are computa-
tionally efficient and can use parallel computing resources
(Ahmadisharaf et al. ).
The Soil and Water Assessment Tool (SWAT) is a highly
popular watershed modeling tool (Arnold et al. ) that is
widely used for the development of complex, highly parame-
terized and computationally expensive watershed models
(Nossent et al. ;Borah et al. ). Many optimization
algorithms have been developed in the past literature for
addressing the computational challenge of calibrating
SWAT and other complex watershed models (Tayfur ).
For instance, Abbaspour et al. ()proposed Sequential
Uncertainty Fitting (SUFI-2) to efficiently calibrate (within
a few thousand simulations) complex SWAT models.
Tolson & Shoemaker ()proposed the Dynamically
Dimensioned Search (DDS) method to calibrate complex
and high-dimensional (i.e., with many parameters) hydrolo-
gic and watershed models. Efficient Markov Chain Monte
Carlo (MCMC) methods, e.g., the Shuffled Complex Evol-
ution Metropolis (SCEM-UA) method (Vrugt et al. )
and the DiffeRential Evolution Adaptive Metropolis
(DREAM) algorithm (Vrugt et al. ;Vrugt ), and
the Multiple-response Bayesian Calibration (MRBC) frame-
work (Han & Zheng ) that quantify input and
2M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
parameter uncertainty during calibration have also been
extensively applied to watershed problems. Given the
inherent multi-objective nature of watershed model cali-
bration (Gupta et al. ,;Yapo et al. ),
numerous multi-objective algorithms have also been pro-
posed and used for watershed model calibration and
uncertainty quantification. Some notable mentions are the
ParaSol (van Griensven & Meixner ), the Borg Multiob-
jective Evolutionary Algorithm (Borg-MOEA) (Hadka &
Reed ), the Pareto archived dynamically dimensioned
search (PA-DDS) (Asadzadeh & Tolson ) and the
Non-dominated Sorting Genetic Algorithm II (NSGA-II)
(Deb et al. ;Ercan & Goodall ). ParaSol (van
Griensven & Meixner ) also applies thresholds on
different objectives to filter/identify behavioral solutions
(Beven & Binley ) and, subsequently, quantify model
uncertainty.
The use of desktop/stand-alone computational
resources is less effective and, in some cases, not feasible
for automatic calibration and analysis of large-scale water-
shed models (especially distributed and semi-distributed
models) with complex physical domains and multiple
water resource issues such as water quality, droughts and
floods (Abbaspour et al. ;Arnold et al. ;Gupta
et al. ). Hence, parallel implementations of many cali-
bration frameworks and algorithms have been introduced
in the recent literature (Kan et al. ). Rouholahnejad
et al. ()implemented a parallel version of the SUFI-2
algorithm that is widely used for calibration of SWAT
models. SUFI-2 is part of SWAT-CUP (Abbaspour ),
which is a popular stand-alone program for calibration of
SWAT models. Ercan et al. ()implemented a parallel
version of DDS for SWAT calibration with deployment on
a windows-based cloud infrastructure. Joseph & Guillaume
()presented a parallel implementation of the DREAM
algorithm that is specifically designed for parameter esti-
mation and uncertainty quantification of SWAT watershed
models. Bacu et al. ()developed grid-based architectural
components for SWAT (gSWAT) with up to 60 worker
nodes. The gSWAT with its inherent SUFI-2 algorithm was
used in a fine resolution SWAT model of the Black Sea
catchment (Rodila et al. ) in the scope of an EU/FP7
enviroGRIDS project (enviroGRIDS ) and in a large-
scale Danube River Basin project (Gorgan et al. ;
Rodila et al. ). The gSWAT computing infrastructure
was found to optimize the SWAT model when running in
parallel (Bacu et al. ). Zhang et al. ()also paralle-
lized the SWAT model itself (rather than the calibration
framework) by simultaneously simulating output for each
distributed model land unit (also called Hydrological
Response Unit, HRU).
The above-mentioned parallel calibration frameworks
clearly illustrate the value of parallel processing and high-
performance computing in addressing the challenge of cali-
brating large and complex watershed models (Humphrey
et al. ;Ercan et al. ;Astsatryan et al. ;Zhang
et al. ). However, many prior studies do not adequately
discuss two key aspects of parallel watershed model cali-
bration, i.e., (i) compatibility of parallel algorithm
implementations with different cloud platforms and (ii) the
impact of cloud infrastructure on computing efficiency of
parallel algorithms and frameworks.
In general, commercialized cloud computing platforms
currently support either Windows or Linux operating sys-
tems, and in some cases, both operating systems are
supported. Thus operating system compatibility is an impor-
tant criterion in developing a cloud-based calibration
platform. In Humphrey et al. (), a cloud-based cali-
bration system for SWAT models was developed, based on
Microsoft Windows Azure (Chappell ). In addition, a
multi-component enterprise cloud service was developed
and studied for watershed calibration, and the virtual
machines (VMs) for the cloud platform were created using
Hadoop and Openstack which are open-source software
(Astsatryan et al. ;Zhang et al. ).
Hardware infrastructure of cloud platforms is also an
important factor that can significantly impact speed-up
and efficiency of frameworks where simulations are exe-
cuted in parallel. O’Donncha et al. ()showed that
parallel performance/efficiency of a fluid dynamics model
varies significantly on single versus multi-node architec-
tures. In their review of the parallel watershed calibration
literature, Kan et al. ()note that prior work on under-
standing the effectiveness of parallel watershed model
frameworks on different cloud platforms is very limited
and needs to be explored more in future. Motivated by this
need, we implemented a cloud-based watershed calibration
and uncertainty analysis system under Linux Operating
3M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
system using SOSCIP’s cloud analytic platform (SOSCIP
).
The watershed calibration system proposed in this study
is called Linux-based Cloud Calibration system for SWAT
(LCC-SWAT) and is specifically designed for complex water-
shed models developed using SWAT. LCC-SWAT includes
two parallel optimization methods SUFI-2 (Abbaspour
et al. ,) and DDS (Tolson & Shoemaker ).
The design of our cloud-based calibration system is compati-
ble with the commonly used stand-alone SWAT-CUP system
(Abbaspour ). We believe that this will encourage exist-
ing SWAT (and SWAT-CUP) users and modelers to test their
models using LCC-SWAT. LCC-SWAT has been added to
the Canadian Watershed Evaluation Tool (CANWET™)
platform to provide efficient, automatic calibration and visu-
alization of SWAT models.
This paper also includes a comprehensive comparison of
parallel calibration results obtained from SUFI-2 and DDS
on the SWAT model of the Grand River Basin (6,542 km
2
)
in Ontario, Canada. To the best of our knowledge, parallel
implementations of DDS and SUFI-2 have not been compared
in the past. Moreover, the detailed runtime performance of
LCC-SWAT is evaluated on different cloud architectures
(i.e., single versus multi-node CPU systems), and by using
three SWAT models of increasing complexities and sizes
(19–215,918 km
2
). We believe that the availability of such a
cloud-based system is an important contribution to watershed
modeling software and to the future implementation of
improved cloud-based calibration frameworks. In addition,
unlike other cloud-based systems, LCC-SWAT automatically
generates comprehensive statistical reports pertaining to the
SWAT model calibration and uncertainty analysis which facili-
tates more comprehensive analyses of the calibration
parameters and the overall model performance.
BACKGROUND
Soil and Water Assessment Tool
SWAT (Arnold et al. ) is a long-term continuous hydro-
logic and water quality model. It is one of the most widely
used models in the hydro-environmental domain (Arnold
et al. ). Being a semi-distributed and physically based
model, SWAT has a high number of parameters related to
hydrology, erosion and sediment transport, nutrients, pesti-
cides, fecal bacteria, among others (Leta et al. ),
making it one of the more complex and over-parameterized
hydro-environmental models (Nossent et al. ). For mod-
eling purposes, SWAT divides a watershed into several sub-
watersheds. A sub-watershed is further divided into Hydro-
logical Response Units (HRUs) which are unique
combination of land-use, soil and slope. An HRU is the com-
putation unit of the SWAT model (Arnold et al. ).
Cloud computing infrastructure
The cloud computing infrastructure on which the new water-
shed calibration system was deployed and tested and was
created by the Southern Ontario Smart Computing Inno-
vation Platform (SOSCIP) consortium. The calibration
system was designed and developed on the SOSCIP’s cloud
analytic platform (SOSCIP ). The allocated cloud
resource contains two VMs, or nodes, using the Linux Oper-
ating system, each with 24 computational cores, or CPUs.
The calibration system has 196 GB of RAM and 2TB of a net-
work’s storage which is managed as centralized data storage
to retrieve and store data by VMs using Network File System
(NFS) in a faster and more efficient manner as compared to
that of distributed data storages. The calibration process
can be performed on a single or multiple VMs. In addition,
the proposed cloud-based calibration system does not have
a limitation for employing the maximum number of available
computational cores if the computational resource on the
cloud platform is extended.
Optimization algorithms
Sequential uncertainty fitting (SUFI-2)
The SUFI-2 (Abbaspour et al. ,) method was devel-
oped to address the degree of all uncertainties quantified by
the p-stat measure which is the percentage of measured data
grouped by the 95% prediction uncertainty (95PPU). The r-
stat is another measure that quantifies the strength of the
uncertainty analysis of a calibration from the average of
the 95PPU band divided by the standard deviation of the
measured data. The SUFI-2 method aims to detect the
4M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
majority of the measured data with the smallest uncertainty
band. The 95PPU is calculated at the 2.5 and 97.5% levels of
the cumulative distribution of an output variable obtained
by using Latin Hypercube Sampling (LHS) (McKay et al.
). Therefore, it eliminates the lowest performing 5% of
simulations. The value of the p-stat ranges between 0 and
100, and the value of r-stat ranges between 0 and infinity.
A calibration that exactly corresponds to the measured
data has a p-stat of 1 and r-stat of 0. LHS is a statistical
method to sample evenly over the sample space from the
random parameter values of a multidimensional distri-
bution. An LH is considered as a predefined number of
dimensions where each sample is the only one in each
axis-aligned hyperplane that contains the sample. LHS is
applied usually to reduce the computational time of running
Monte Carlo simulations which can decrease the processing
time by up to 50%.
Dynamically dimensioned search
DDS (Tolson & Shoemaker ) is a heuristic optimization
algorithm for calibration of watershed simulation models.
The DDS optimization method was introduced for cali-
bration problems that require a large set of decision
variables whose lower and upper values are predefined.
DDS aims to find an optimal solution within a user-defined
number of simulations or, in general, function evaluations.
Initially, the DDS algorithm explores globally the solution
space and changes the search domain gradually to local
searches when the number of function evaluations or simu-
lations is reached to a predefined maximum number of
iterations. In each iteration, the changes in the search
domain are achieved dynamically and probabilistically by
reducing dimensions (or the number of decision values) in
search neighborhoods of a solution space. The decision
values are the parameters of a watershed model which are
adjusted. The probability that decision variable xis selected
in iteration iis computed as follows:
Px(i)¼1ln (i)
ln (m)(1)
where mis the maximum number of function evaluations
for a calibration. In Equation (1), the probability of adjusting
the value of a decision variable is reduced gradually by the
increases in the number of iteration i.
THE LINUX-BASED CLOUD CALIBRATION SYSTEM
FOR SWAT (LCC-SWAT) DEVELOPMENT
System design
Figure 1 provides an overview of the LCC-SWAT calibration
system workflow/design, runs in the Ubuntu 18.04 operating
system. It has three core components. The first component is
user-defined where a user creates a SWAT model, sets the
calibration parameters using SWAT-CUP protocols and
uploads the model and calibration setup files to the LCC-
SWAT system on the cloud using Secure File Transfer Proto-
col (SFTP). The second component is the parallel
optimization strategy. Two parallel optimization implemen-
tations are currently included in LCC-SWAT, which are
discussed further in the ‘System implementation’and ‘Paral-
lelization of optimization algorithms’sections. The final
component of the system is the SWAT parallelization rou-
tines, which allow multiple SWAT simulations in the batch-
parallel mode. If the optimization algorithm is set up to run
nmodel runs in an iteration (i.e., nparameter sets are pro-
vided by the optimization algorithm), these are equally
distributed among available computing resources/cores.
System implementation
The entire LCC-SWAT framework, including the two
optimization methods (discussed in the ‘Parallelization
of optimization algorithms’section), their batch-parallel
components and input/output communications with the
SWAT model were implemented in C þþ. The actual
model files are created and configured by the SWAT-
CUP program under the Windows operating system on a
personal computer. The entire model files are uploaded
by the user to LCC-SWAT and copied to the internal net-
work storage. VMs retrieve and store the original and
updated model files from the network storage. In order
to make the inputs and outputs of the parallel optimiz-
ation methods compatible with SWAT-CUP, we
5M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
implemented (and replicated) the SWAT-CUP modules
that edit model files and represent inputs and extract
outputs to/from the SWAT program according to the
desired time series of simulations, and other modules
that describe the calibration workflow as defined in
SWAT-CUP’smanual(Abbaspour ). The Linux ver-
sion of SWAT program and SWAT-CUP’s‘swat_edit’
module were embedded and called as external routines
at each step of simulation during the optimization. A
number of SWAT-CUP’s modules were not available for
the Linux operating system. The functionality of these
modules has to be replicated to work on Linux, e.g.,
‘SUFI_extract_rch’. Hence, LCC-SWAT is compatible
with SWAT-CUP and users of LCC-SWAT can formulate
their SWAT model calibration using SWAT-CUP. The
optimization methods proposed in LCC-SWAT do not
employ a multi-objective function; however, multiple per-
formance metric values (e.g., R
2
, NSE) are stored for each
simulation during calibration for post-optimization analy-
sis. LCC-SWAT can calibrate concurrently multiple
watershed models by multiple users. However, the cali-
bration times are increased differently based on how the
cloud platform allocates the computational resources,
thesizeofmodelfiles and the number of users.
Figure 1 |Overview of the Linux-based Cloud Calibration system for Soil and Water Assessment Tool (LCC-SWAT) system design.
6M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
Parallelization of optimization algorithms
The design of the cloud-based calibration system that
employs the paralleled SUFI-2 algorithm is illustrated as fol-
lows. If nCPUs are employed in total to perform m
simulations, the input decision values generated by LHS
are grouped in kparameter sets where kis equal to m
divided by n. In other words, the calibration process is per-
formed in kiterations where nsimulations are performed in
each iteration. The results of simulations are collected and
aggregated in each iteration. Next, after a predefined
number of simulations is reached, the optimal calibrated
values of the decision variables (or parameter set) are
assigned to the simulation with the best fitness value as
shown in Figure 2. Lastly, a comprehensive statistical
report is computed for all simulations on the cloud platform
for more robust statistical analyses.
The design of the cloud-based calibration system that
uses iteratively the paralleled DDS optimization method is
shown in Figure 3. Initially, nrandom samples or parameter
sets are created with regards to the defined ranges of the
SWAT model’s decision variables, and constant nis also equal to the number of computational cores. The samples
are distributed to the cloud’s VMs, and each core performs
one simulation by using one of the parameter sets. The par-
ameter set xwhose simulation generates the best fitness
value is identified. Then, the DDS optimization is performed
ntimes on parameter set xto create nnew parameter sets.
The next iteration is started by repeating the distribution of
the new parameter sets to the VMs. The calibration process
is terminated when the maximum number of simulations is
reached. Lastly, the parameter set xthat is identified in the
last iteration usually corresponds to the simulation with
the best fitness evaluation in all iterations. By the paralleled
DDS optimization, a calibration performed with a high
number of simulations usually generates a more optimal par-
ameter set compared to that of a calibration with a lower
number of simulations.
Testing the cloud calibration system for different SWAT
models
In order to test the effectiveness of the developed cloud com-
puting system (i.e., LCC-SWAT) as a function of increasing
size and complexity of watersheds, three SWAT models
Figure 3 |An overview of the cloud calibration system using the Dynamically Dimen-
sioned Search (DDS) optimization algorithm.
Figure 2 |An overview of the cloud calibration system using the Sequential Uncertainty
Fitting (SUFI-2) optimization algorithm.
7M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
were set up, i.e., (i) a small agricultural Wigal Creek water-
shed (19 km
2
)(Zhang et al. ), (ii) a medium-sized
Grand River Basin (6,542 km
2
)(Kaur et al. ) and (iii) a
large-sized Canadian Great Lakes Basin (215,918 km
2
).
High spatial resolution spatial dataset (digital elevation
model, land-use and soil; Supplementary Table S1), daily
meteorological dataset (precipitation and temperature; Sup-
plementary Table S1) and crop-management data were
sourced from different agencies and used during the process.
The model setup resulted in 452, 2679 and 29449 HRUs,
respectively, for Wigal, Grand River and Canadian Great
Lakes SWAT models. Models were run for 1,000 simulations
in monthly timescale for a period of 12 years in the cloud plat-
form using the SUFI-2 optimization algorithm.
We tested LCC-SWAT on two criteria, i.e., (i) via analyz-
ing calibration runtime of the framework with varying
number of allocated processors (4–48) and nodes/VMs (1–
2) and (ii) via analyzing comparative performance of the
two algorithms currently implemented in the framework
(DDS and SUFI-2).
The primary purpose of the calibration runtime compari-
son was to provide some insights and guidelines on
identifying optimal computing resource allocation for LCC-
SWAT. The proposed calibration system can be deployed
on multiple VMs or nodes with a predefined number of pro-
cessors. The number of cores allocated for any calibration
job should be within the bounds specified in the MPICH2
package, an open-source implementation of the message pas-
sing interface (MPI) method for parallel programming in
Cþþ (used in LCC-SWAT). Moreover, core allocation for a
calibration job should also be within available computational
resource limitations. For instance, we deployed and tested
LCC-SWAT on the SOSCIP cloud platform, where two com-
puting nodes were available with 24 cores each (48 cores in
total). Hence, the maximum core allocation for calibration
was 48 in our experiments. Ideally, a calibration process
with msimulations (assuming all simulations can be executed
simultaneously, i.e., in a single batch) and ncore allocations
where mis greater than ncan be performed faster by specify-
ing n(on a single or multi-node system) to be equal to the
number of available physical cores (which were 48 for our
experiments). However, parallel runtime performance, typi-
cally, does not scale proportionally with the number of
allocated cores and can vary significantly. Variations in
runtime performance can be attributed to multiple reasons,
e.g., physical cloud infrastructure and coding structure of
the underlying simulation, etc. (Hadka & Reed ).
Hence, we analyzed runtime performance of LCC-SWAT
with a different number of core allocations to (i) deduce opti-
mal number of core allocations for LCC-SWAT’s deployment
on SOSCIP and (ii) provide insights on deducing optimal
core allocation for deployment on other cloud infrastructure.
For this analysis, we tested the cloud-based calibration system
using nodes in regular increments of four cores at both single
and double node (VMs) configurations.
As different optimization algorithms have their own advan-
tages in terms of converging to the global optima of multi-
dimensioned parameter search, it is important that more than
one optimization algorithm is tested. Hence, the effectiveness
of the above-mentioned optimization algorithms (SUFI-2 and
DDS) was tested for the medium-sized Grand River Basin. Fol-
lowing a global sensitivity analysis using the SWAT-CUP, the 18
most sensitive SWAT parameters (Supplementary Table S2)
were considered to optimize monthly streamflow measured
at Grand River near Marsville, one of an upstream streamflow
gauging station of the Grand river for an 8-year time period
(2008–2015). The global sensitivity analysis regresses par-
ameters generated using the LHS methodology (McKay et al.
) against a chosen objective function. We conducted the
sensitivity analysis at monthly timescale using streamflow at
the same Grand River near Marsville for the same 8-year
time period (2008–2015) using the Nash–Sutcliffe Efficiency
(NSE; Nash & Sutcliffe ) as the objective function. It
should be noted that SWAT inherently runs in a daily timescale,
as such, in our case, the daily simulations are aggregated in the
monthly timescale. The range (maximum and minimum) of
sensitive parameters values (Supplementary Table S2) were
chosen based on similar reported works in cold-climate
region basins (Faramarzi et al. ;Shrestha et al. ;
Zhang et al. ;Kaur et al. ).
RESULTS AND DISCUSSION
Optimal number of cores
Figure 4 shows the evolution of model runtime when utilizing
an increasing number of cores in single and double node
8M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
configuration for three separate SWAT models. As expected,
the relative runtimes were higher for more complex SWAT
models (e.g., Great Lakes Basin). Figure 4 indicates that utiliz-
ation of a larger number of cores reduced, in our case, the LCC-
SWAT calibration runtime. However, for both single and
double node configurations, improvement (i.e., reduction) in
calibration runtime continued until the utilization of 20
cores only. The runtime then increased when utilizing the
maximum (24) cores on a single node. The same trend was
also observed for calibration runs utilizing two nodes.
Hence, we found 20 coresin both single anddouble node allo-
cation scenarios to be the optimal core allocation.
As mentioned in the ‘Testing the cloud calibration
system for different SWAT models’section, runtime per-
formance of parallel frameworks may not be proportional
to the number of allocated cores, and this trend is also
observed for LCC-SWAT. For instance, with single node
deployment, we observed that LCC-SWAT’s runtime
performance deteriorated when allocated processors
increased from 20 to 24. One plausible reason for this
deterioration is that when all 24 processors are allocated,
the processors’resources are shared for the execution of
simulations and the operating system’s internal tasks and
job scheduling. Thus, without idle processors, the concur-
rent execution of different tasks assigned to a node’s
processors causes overloads and results in time latencies
and increasing the overall runtime of the parallel cali-
bration. The computational runtime improvement for a
single node system may also be limited by hardware con-
figuration. For instance, LCC-SWAT is deployed on a
cloud infrastructure where the physical cores on a machine
are doubled by hyper-threading technology to form logical
cores which share execution, memory and I/O resources.
Hence, given SWAT’s high I/O requirements (Zhang et al.
), runtime performance deterioration is expected for
the cloud infrastructure employed in this study.
Figure 4 |Runtime as the function of number of cores in single- and double-node configuration for the three Soil and Water Assessment Tool (SWAT) models. Also shown is the com-
putation time overhead factor.
9M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
We also observed deterioration in runtime performance
with the utilization of multiple nodes, i.e., the computational
time of a calibration run on two nodes was more than that of
the calibration on a single node with the same number of
total processors. This deterioration was due to expected net-
work latency and I/O operations required in
communication between multi-node systems. Moreover,
SWAT is an I/O intensive (Zhang et al. ) simulation
model, and thus, parallel communication bottlenecks arising
in parallel application of SWAT simulations in multi-node
systems can easily supersede the potential advantage of
the availability of additional cores.
Issues pertaining to linear scaling of runtime perform-
ance of SWAT parallelization frameworks, with the added
number of cores, are also observed in other studies (Ercan
et al. ;Zhang et al. ;Bacu et al. ). Furthermore,
use of an increased number of cores has been debated from
a cost-effective point of view. For instance, Ercan et al. ()
showed, with an experiment involving 256 cores, that use of
64 cores was the most desirable from the economical point
of view.
Figure 4 also reports calculation time overhead, calcu-
lated as the ratio of the runtime in any configuration to
the runtime for the optimal cores (20). This metric indicates
that a cloud-based computing system (regardless of cloud
infrastructure) could reduce calibration runtime of SWAT
significantly (compared to desktop systems up to 8 cores).
From the perspective of computational time overhead, the
added value of such a cloud-based computing system is high-
lighted especially for the complex SWAT model, the Great
Lakes Basin. For this SWAT model, the computation time
overhead for all the configurations is lower than that
observed for the less complex model (e.g., Wigle Water-
shed). For example, for two nodes configuration and using
48 cores, the computation time overhead for the Wigle
Watershed was 3.64 which reduced to 2.00 and 1.87 for
the Grand River and Great Lakes Basin, respectively.
Comparison of SUFI-2 and DDS algorithms
As a test of the cloud calibration system, both SUFI-2 and
DDS algorithms were run for 1,000 simulations with NSE
(Nash & Sutcliffe ) as the objective function to optimize
monthly streamflow at Grand River near Marysville. While
the NSE was the objective function, we also calculated
PBIAS and R
2
to assign a qualitative rating to model simu-
lation (Moriasi et al. ). Moreover, two performance
aspects were considered during the comparative analysis
of DDS and SUFI-2, i.e., (i) posterior parameter distributions
(using behavioral solutions) obtained from both algorithms
(see Section Analyzing posterior parameter distributions)
and (ii) calibration statistics (of best calibrations found)
and predictive uncertainty bounds obtained from both algor-
ithms (see Section ‘Calibration statistics and predictive
uncertainty’).
Analyzing posterior parameter distributions
Figure 5 shows the posterior distributions of the three most
sensitive parameters (identified during global sensitivity
analysis; see Section ‘Testing the cloud calibration system
for different SWAT models’), obtained from DDS and
SUFI-2. Following Moriasi et al. (), all solutions with
an NSE value more than 0.8 were considered as ‘behavioral
solutions’, and the posterior distributions (represented by
histograms in Figure 5) were estimated using behavioral sol-
utions only. The results in Figure 5 show that DDS was
successful in obtaining a narrower and more well-defined
parameter distribution for all three parameters. The DDS-
based posterior distribution of all parameters showed a
clear and high relative frequency (∼0.7), while the SUFI-2-
based posterior distribution (Supplementary Table S2)
seems to be spread in a wider range. In stochastic modeling
paradigm, the ability of an optimization algorithm to clearly
identify an optimal range parameter is important given the
issues related to parameter identifiability (Chavent ),
especially for the SWAT model that is often regarded as
an over-parameterized model (Nossent et al. ). Thus,
the posterior distribution obtained from DDS is clearly
better.
The relative superiority of DDS can be attributed to the
algorithm’s iterative search dynamics, where, in each paral-
lel simulation batch, new candidate calibration solutions are
obtained by perturbing the best calibration found so far (see
Figure 3 and Section ‘Dynamically dimensioned search’).
Moreover, DDS only perturbs a subset of parameters in
each algorithm iteration, and the number of parameters to
be perturbed reduces as the algorithm progresses (see
10 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
Section ‘Dynamically dimensioned search’and Equation (1)).
This strategy is especially effective for calibration problems
with many parameters (Asadzadeh & Tolson ). SUFI-
2, on the other hand, uses LHS (McKay et al. ), which
is a uniform stochastic space-filling design, and thus,
posterior distributions obtained from SUFI-2 are more
uniform. However, if the budget of functions evaluations
for SUFI-2 is increased (from 1,000; which is not desirable
for computationally expensive SWAT models), posterior dis-
tributions of parameters will become more well-defined and
narrower.
Calibration statistics and predictive uncertainty
Figure 6 shows the 95% predictive parameter uncertainty
bands on the monthly streamflow for the 8-year period
(2008–2015), for both DDS and SUFI-2. Owing to the
better identifiability of parameters by the DDS algorithm
over the SUFI-2 algorithm, the DDS 95% predictive par-
ameter uncertainty band on monthly streamflow also
Figure 6 |The 95% predictive parameter uncertainty (PPU) band on monthly streamflow
of Grand River near Marsville. PBIAS: Percentage of bias, NSE: Nash-Sutcliffe
Efficiency, and R2: Coefficient of determination.
Figure 5 |Posterior distribution of top three most sensitive parameters for all behavioral solutions from Sequential Uncertainty Fitting (SUFI-2) and Dynamically Dimensioned Search (DDS)
optimization algorithms. SMTMP: Snow melt temperature, CN2: Curve number for moisture condition II, and SFTMP: Snow fall temperature.
11 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
consistently out-performed of the SUFI-2 algorithm
(Figure 6). While the p-stat (percentage of the observations
encapsulated in the 95% PPU band) values for both optimiz-
ation algorithms are fairly comparable, there is a significant
difference in the r-stat (the thickness of the 95% PPU band).
The SUFI-2-based optimization resulted in a wider 95% PPU
band (r-stat ¼1.14), inferring to higher uncertainty in simu-
lated monthly streamflow at Grand River near Marysville.
Furthermore, all the goodness-of-fit statistics (pertaining to
the deterministic modeling paradigm) for the best-fit simu-
lation also showed consistent underperformance of the
SUFI-2 algorithm (compared to DDS). Following Moriasi
et al. ()model performance criterion, the qualitative
rating for the SUFI-2 based simulated monthly streamflow
is ‘good’, while the same for the DDS-based simulation is
‘very good’.
As stated in the ‘Sequential uncertainty fitting (SUFI-2)’
and ‘Analyzing posterior parameter distributions’sections,
SUFI-2 uses LHS (McKay et al. ), with uniform a priori
distribution of model parameter from the defined range, to
search for optimal solutions in a high-dimensional (i.e., with
18 parameters) parameter space. It is therefore evident that
SUFI-2 optimization may not always reach the neighborhood
of the global optima. Whereas the DDS algorithm initially
explores globally in the solution space and changes the
search domain gradually to local searches (by reducing the
number of parameters to be perturbed; see Section ‘Dynami-
cally dimensioned search’). Hence, there is a higher chance
that the DDS algorithm may find solutions close to the
global optima. In our example case, the higher performance
of the DDS algorithm may be related to the above-stated
reasons. It should, however, be noted that further studies
are needed to explicitly conclude the added advantage of
one optimization algorithm over another. This cloud-based
calibration (and uncertainty analysis) system indeed offers a
platform to conduct such a computationally demanding task.
Recommendations, limitations and future perspective
of the work
It is well known that SWAT is a highly parameterized model
(Nossent et al. ), as such is highly I/O intensive. The
LCC-SWAT system is flexible and can be deployed on any
cloud platform (and with the different number of available
computational cores). However, increasing the number of
cores may require the addition of more computational
nodes or VMs. Since multiple nodes may be linked together
by networks with I/O traffic affecting, overall computational
time, it is imperative that multi-node computational over-
head is considered before deploying LCC-SWAT (and
similar frameworks) on multi-node cloud infrastructure (as
indicated in the results discussed in the ‘Optimal number
of cores’section). When frameworks similar to LCC-
SWAT are deployed on multi-node cloud platforms, the
architecture of the network’s storage determines signifi-
cantly the data storage, data retrieval and computational
costs. Therefore, a very fast and dedicated network storage
is a great advantage (and is highly recommended for LCC-
SWAT deployment on multi-node systems) to boost the
high demands of data accesses by VMs during parallel
multi-node simulation runs in LCC-SWAT.
LCC-SWAT’s implementation is modular and, thus, can
incorporate extensions and modifications in future, to
improve the system’s ease-of-use and calibration perform-
ance. In this regard, three key extension/improvement
avenues are (i) inclusion of more optimization algorithms,
especially multi-objective algorithms for exploring calibration
trade-offs, (ii) asynchronous parallel implementation of exist-
ing and new algorithms, for enhancing runtime efficiency and
(iii) implementation of an interactive user interface (the user
interface of LCC-SWAT is currently, console-based). Given
the computational time overhead induced by the cloud infra-
structure and high I/O intensive nature of SWAT simulations,
parallel efficiency of the LCC-SWAT framework may benefit
significantly from the inclusion of asynchronous parallel
optimization algorithms (Zhabitskaya & Zhabitsky ).
The user interface of LCC-SWAT can be enhanced from a
console-based interaction to a more user-friendly visual inter-
face, and the interface enhancement can be used as a
blueprint when the LCC-SWAT system is planned to be distrib-
uted for client-based accesses through the internet.
Development of a friendly user-interface may be especially
important to enable broad uptake and use of the LCC-SWAT
system. Many research and public organizations (e.g., U.S.
Environmental Protection Agency (USEPA), Ontario Ministry
of Natural Resources and Forestry (OMAFRA), etc.), use
SWAT for watershed modeling (Francesconi et al. )and
watershed nutrient management. However, efficient and
12 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
effective SWAT calibration remains a challenge for such
organizations. It is envisioned that LCC-SWAT could be
used as a cloud calibration service tool by such organizations
(especially organizations in Canada) in future.
SUMMARY AND CONCLUSIONS
In this study, a cloud calibration and uncertainty analysissystem
offering two paralleled optimization algorithms (SUFI-2
and DDS) is developed for SWAT models, and deployed on
the SOSCIP Cloud Analytic platform. The proposed cloud-
based system, called LCC-SWAT, is a key contribution to
the watershed calibration practice that allows parallel bench-
marking of alternate parallel optimization algorithms on
different cloud computing architectures. An illustration of
the potential application of LCC-SWAT in parallel bench-
marking of calibration algorithms is provided in this study
via a comparison of parallel SUFI-2 and parallel DDS with
a budget of 1,000 evaluations and with 20 cores each. Results
show that the performance of DDS is better than SUFI-2.
Results of performance benchmarking of the LCC-
SWAT system on both single (i.e., with one VM) and dual
node (i.e., with two VMs) architectures are also provided
with the application to three SWAT models of increasing
complexities. Although a maximum of 48 cores in two
VMs were available, results indicate that 20 cores in a
single virtual machine is an optimal configuration (as per
runtime perspective) for the cloud architecture tested in
this study. However, for more complex watershed models,
the runtime efficiency of multi-node systems improves
since the computation time overhead reduces and core util-
ization improves. These results also indicated that an
asynchronous parallel implementation may further improve effi-
ciency and scalability of LCC-SWAT for multi-node systems.
Moreover, the design of LCC-SWAT is modular and flexible;
thus, other single and multi-objective parallel algorithms can
be added to enrich the system for efficiently solving future
large-scale watershed model calibration problems. Finally,
LCC-SWATwasalsosuccessfullyintegratedintothe
CANWET™platform. The platform is designed to facilitate the
use of modeling as a means of watershed management and
policy testing. Further information is available at https://www.
grnland.com/Greenland-Technologies-Group/CANWET.html.
ACKNOWLEDGEMENTS
The authors thank SOSCIP Smart Computing for Innovation
for providing advanced computing platform.
FUNDING
The corresponding author was supported with Natural
Sciences and Engineering Research (NSERC) discovery grant
(number: 2017-04400). Funding and access to parallel comput-
ing resources were provided to the project by SOSCIP –Smart
Computing for Innovation. Greenland International Consult-
ing Ltd was an in-kind contributor to the research.
CONFLICTS OF INTEREST
The authors declare that there is no conflict of interest.
AUTHOR CONTRIBUTION
P.D. and T.B. conceptualized the study. M.Z. developed the
LCC-SWAT platform. N.K.S. tested the platform for various
SWAT models for runtime, parameter identification, cali-
bration and uncertainty analysis. M.Z., N.K.S. and T.A.
drafted and revised the manuscript. P.D. and T.B. provided
their comments and revision.
DATA AVAILABILITY STATEMENT
Data cannot be made publicly available; readers should con-
tact the corresponding author for details Q3.
REFERENCES
Abbaspour, K. C. SWAT Calibration and Uncertainty
Programs –A User Manual. Swiss Federal Institute of
Aquatic Science and Technology (Eawag).
Abbaspour, K. C., Johnson, C. A. & van Genuchten, M. T.
Estimating uncertain flow and transport parameters using a
sequential uncertainty fitting procedure.Vadose Zone
Journal 3(4), 1340–1352.
13 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
Abbaspour, K. C., Yang, J., Maximov, I., Siber, R., Bogner, K.,
Mieleitner, J., Zobrist, J. & Srinivasan, R. Modelling
hydrology and water quality in the pre-alpine/alpine Thur
watershed using SWAT.Journal of Hydrology 333 (2–4),
413–430.
Ahmadisharaf, E., Camacho René, A., Zhang Harry, X.,
Hantush Mohamed, M. & Mohamoud Yusuf, M.
Calibration and validation of watershed models and
advances in uncertainty analysis in TMDL studies.Journal
of Hydrologic Engineering 24 (7), 03119001.
Arnold, J. G., Srinivasan, R., Muttiah, R. S. & Williams, J. R.
Large area hydrologic modeling and assessment part I:
model development.Journal of the American Water
Resources Association 34 (1), 73–89.
Arnold, J. G., Kiniry, J. R., Srinivasan, R., Williams, J. R., Haney,
E. B. & Neitsch, S. L. Soil and Water Assessment Tool
Input/Output File Documentation, Version 2009. Agrilife
Blackland Research Center, Temple, Texas.
Arnold, J. G., Moriasi, D. N., Gassman, P. W., Abbaspour, K. C.,
White, M. J., Srinivasan, R., Santhi, C., Harmel, R. D., van
Griensven, A., Van Liew, M. W., Kannan, N. & Jha, M. K.
SWAT: model use, calibration, and validation. American
Society of Agricultural and Biological Engineers 55 (4), 18.
Asadzadeh, M. & Tolson, B. Pareto archived dynamically
dimensioned search with hypervolume-based selection for
multi-objective optimization.Engineering Optimization
45 (12), 1489–1509.
Astsatryan, H., Narsisian, W. & Asmaryan, S. SWAT
hydrological model as a DaaS cloud service.Earth Science
Informatics 9(3), 401–407.
Bacu, V., Mihon, D., Rodila, D., Stefanut, T. & Gorgan, D.
Grid based architectural components for SWAT model
calibration. In HPCS 2011 - International Conference on
High Performance Computing and Simulation,4–8 July,
Istanbul, Turkey, pp. 193–198.
Bacu, V., Nandra, C., Stefanut, T. & Gorgan, D. SWAT model
calibration over Cloud infrastructures using the BigEarth
platform. In 2017 13th IEEE International Conference on
Intelligent Computer Communication and Processing (ICCP),
pp. 453–460.
Beven, K. & Binley, A. The future of distributed models:
model calibration and uncertainty prediction.Hydrological
Processes 6(3), 279–298.
Borah, D. K., Ahmadisharaf, E., Padmanabhan, G., Imen, S. &
Mohamoud, Y. M. Watershed models for development
and implementation of total maximum daily loads.Journal of
Hydrologic Engineering 24 (1), 03118001.
Chappell, D. Introducing Windows Azure. David Chappell
and Associates.
Chavent, G. On parameter identifiability.IFAC Proceedings
Volumes 18 (5), 531–536.
Deb, K., Pratap, A., Agarwal, S. & Meyarivan, T. A fast and
elitist multiobjective genetic algorithm: NSGA-II.IEEE
Transactions on Evolutionary Computation 6(2), 182–197.
Devia, G. K., Ganasri, B. P. & Dwarakish, G. S. A review on
hydrological models.Aquatic Procedia 4, 1001–1007.
enviroGRIDS The enviroGRIDS Project. Available from:
http://www.envirogrids.net/
Ercan, M. B. & Goodall, J. L. Design and implementation of a
general software library for using NSGA-II with SWAT for
multi-objective model calibration.Environmental Modelling
& Software 84, 112–120.
Ercan, M. B., Goodall, J. L., Castronova, A. M., Humphrey, M. &
Beekwilder, N. Calibration of SWAT models using the
cloud.Environmental Modelling & Software 62, 188–196.
Faramarzi, M., Srinivasan, R., Iravani, M., Bladon, K. D.,
Abbaspour, K. C., Zehnder, A. J. B. & Goss, G. G.
Setting up a hydrological model of Alberta: data
discrimination analyses prior to calibration.Environmental
Modelling & Software 74,48–65.
Francesconi, W., Srinivasan, R., Pérez-Miñana, E., Willcock, S. P.
& Quintero, M. Using the Soil and Water Assessment
Tool (SWAT) to model ecosystem services: a systematic
review.Journal of Hydrology 535, 625–636.
Gorgan, D., Bacu, V., Mihon, D., Rodila, D., Abbaspour, K. &
Rouholahnejad, E. Grid based calibration of SWAT
hydrological models.Natural Hazards and Earth System
Sciences 12 (7), 2411–2423.
Gupta, H. V., Sorooshian, S. & Yapo, P. O. Toward improved
calibration of hydrologic models: multiple and
noncommensurable measures of information.Water
Resources Research 34 (4), 751–763.
Gupta, H. V., Kling, H., Yilmaz, K. K. & Martinez, G. F.
Decomposition of the mean squared error and NSE
performance criteria: implications for improving
hydrological modelling.Journal of Hydrology 377 (1), 80–91.
Gupta, H. V., Clark, M. P., Vrugt, J. A., Abramowitz, G. & Ye, M.
Towards a comprehensive assessment of model
structural adequacy.Water Resources Research 48 (8). Q4
Hadka, D. & Reed, P. Borg: an auto-adaptive many-objective
evolutionary computing framework.Evolutionary
Computation 21 (2), 231–259.
Hadka, D. & Reed, P. Large-scale parallelization of the Borg
multiobjective evolutionary algorithm to enhance the
management of complex environmental systems.
Environmental Modelling & Software 69, 353–369.
Han, F. & Zheng, Y. Multiple-response Bayesian calibration
of watershed water quality models with significant input and
model structure errors.Advances in Water Resources 88,
109–123.
Humphrey, M., Beekwilder, N., Goodall, J. L. & Ercan, M. B.
Calibration of watershed models using cloud computing. In
2012 IEEE 8th International Conference on E-Science, pp.
1–8.
Joseph, J. F. & Guillaume, J. H. A. Using a parallelized
MCMC algorithm in R to identify appropriate likelihood
functions for SWAT.Environmental Modelling & Software
46, 292–298.
14 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
Kan, G., He, X., Li, J., Ding, L., Hong, Y., Zhang, H., Liang, K. &
Zhang, M. Computer aided numerical methods for
hydrological model calibration: an overview and recent
development.Archives of Computational Methods in
Engineering 26 (1), 35–59.
Kaur, B., Shrestha, N. K., Daggupati, P., Rudra, R. P., Goel, P. K.,
Shukla, R. & Allataifeh, N. Water security assessment of
the grand river watershed in Southwestern Ontario, Canada.
Sustainability 11 (7), 1883.
Leta, O. T., Nossent, J., Velez, C., Shrestha, N. K., van Griensven,
A. & Bauwens, W. Assessment of the different sources of
uncertainty in a SWAT model of the River Senne (Belgium).
Environmental Modelling and Software 68, 129–146.
McKay, M. D., Beckman, R. J. & Conover, W. J. A
comparison of three methods for selecting values of input
variables in the analysis of output from a computer code.
Technometrics 21 (2), 239–245.
Moriasi, D. N., Gitau, M. W., Pai, N. & Daggupati, P.
Hydrologic and water quality models: performance measures
and evaluation criteria. 58 (6).
Q5
Nash, J. E. & Sutcliffe, J. V. River flow forecasting through
conceptual models part I –a discussion of principles.Journal
of Hydrology 10 (3), 282–290.
Nossent, J., Elsen, P. & Bauwens, W. Sobol’sensitivity
analysis of a complex environmental model.Environmental
Modelling & Software 26 (12), 1515–1525.
O’Donncha, F., Ragnoli, E., Venugopal, S., James, S. C. & Katrinis,
K. On the efficiency of executing hydro-environmental
models on cloud.Procedia Engineering 154, 199–206.
Rodila, D., Bacu, V. & Gorgan, D. Comparative parallel
execution of SWAT hydrological model on multicore and
grid architectures.International Journal of Web and Grid
Services 8(3), 304–320.
Rodila, D., Ray, N. & Gorgan, D. Conceptual model for
environmental science applications on parallel and
distributed infrastructures.Environmental Systems Research
4(1), 23.
Rouholahnejad, E., Abbaspour, K. C., Vejdani, M., Srinivasan, R.,
Schulin, R. & Lehmann, A. A parallelization framework
for calibration of hydrological models.Environmental
Modelling & Software 31,28–36.
Shirmohammadi, A., Chaubey, I., Harmel, R. D., Bosch, D. D.,
Muñoz-Carpena, R., Dharmasri, C., Sexton, A., Arabi, M.,
Wolfe, M. L., Frankenberger, J., Graff, C. & Sohrabi, T. M.
Uncertainty in TMDL models.Transactions of the
ASABE 49 (4), 1033–1049.
Shrestha, N. K., Du, X. & Wang, J. Assessing climate change
impacts on fresh water resources of the Athabasca River
Basin, Canada.Science of The Total Environment 601–602,
425–440.
SOSCIP Cloud Analytics. Smart Computing for Innovation
(SOSCIP). Available from: https://www.soscip.org/
2017impactreport/platforms/.
Tayfur, G. Modern optimization methods in water resources
planning, engineering and management.Water Resources
Management 31 (10), 3205–3233.
Tolson, B. A. & Shoemaker, C. A. Dynamically dimensioned
search algorithm for computationally efficient watershed
model calibration.Water Resources Research 43 (1). Q6
van Griensven, A. & Meixner, T. A global and efficient multi-
objective auto-calibration and uncertainty estimation method
for water quality catchment models.Journal of
Hydroinformatics 9(4), 277–291.
Vrugt, J. A. Markov chain Monte Carlo simulation using the
DREAM software package: theory, concepts, and MATLAB
implementation.Environmental Modelling & Software 75,
273–316.
Vrugt, J. A., Gupta, H. V., Bouten, W. & Sorooshian, S. A
Shuffled Complex Evolution Metropolis algorithm for
optimization and uncertainty assessment of hydrologic
model parameters. Water Resources Research 39 (8). Q7
Vrugt, J. A., ter Braak, C. J. F., Clark, M. P., Hyman, J. M. &
Robinson, B. A. Treatment of input uncertainty in
hydrologic modeling: doing hydrology backward with
Markov Chain Monte Carlo simulation.Water Resources
Research 44 (12). Q8
Yang, J., Jakeman, A., Fang, G. & Chen, X. Uncertainty
analysis of a semi-distributed hydrologic model based on
a Gaussian Process emulator.Environmental Modelling &
Software 101, 289–300.
Yapo, P. O., Gupta, H. V. & Sorooshian, S. Multi-objective
global optimization for hydrologic models.Journal of
Hydrology 204 (1), 83–97.
Zhabitskaya, E. & Zhabitsky, M. Asynchronous differential
evolution. In: Mathematical Modeling and Computational
Science. MMCP 2011. Lecture Notes in Computer Science
(G. Adam, J. Buša & M. Hnatič, eds). Springer, Berlin.
Zhang, D., Chen, X. & Yao, H. Development of a prototype
web-based decision support system for watershed
management. Water 7(2). Q9
Zhang, D., Chen, X., Yao, H. & James, A. Moving SWAT
model calibration and uncertainty analysis to an enterprise
Hadoop-based cloud.Environmental Modelling & Software
84, 140–148.
Zhang, Y., Hou, J., Cao, Y., Gu, J. & Huang, C. OpenMP
parallelization of a gridded SWAT (SWATG).Computers &
Geosciences 109, 228–237.
Zhang, B., Shrestha, N., Daggupati, P., Rudra, R., Shukla, R., Kaur,
B. & Hou, J. Quantifying the impacts of climate change
on streamflow dynamics of two major rivers of the Northern
Lake Erie Basin in Canada. Sustainability 10 (8). Q10
Zhang, B., Shrestha, N. K., Rudra, R., Shukla, R., Daggupati, P.,
Goel, P. K., Dickinson, W. T. & Allataifeh, N. Threshold
storm approach for locating phosphorus problem areas: an
application in three agricultural watersheds in the Canadian
Lake Erie basin. Journal of Great Lakes Research.Q11
First received 23 April 2020; accepted in revised form 22 September 2020. Available online 20 October 2020
15 M. Zamani et al. |A Linux-based cloud calibration system for SWAT (LCC-SWAT) Journal of Hydroinformatics |in press |2020
Uncorrected Proof
Author Queries
Journal: Journal of Hydroinformatics
Manuscript: HYDRO-D-20-00066
Q1 Please indicate which authors, if any, are IWA members.
Q2 Please confirm whether the Highlights points used in the article are correct.
Q3 Please confirm whether the Data availability statement used in the article are correct.
Q4 Please provide missing page range for reference ‘Gupta et al. 2012’.
Q5 Please provide missing Journal title and page number for reference ‘Moriasi et al. 2015’.
Q6 Please provide missing page range for reference ‘Tolson and Shoemaker 2007’.
Q7 Please provide missing page range for reference ‘Vrugt et al. 2003’.
Q8 Please provide missing page range for reference ‘Vrugt et al. 2008’.
Q9 Please provide missing page range for reference ‘Zhang et al. 2015’.
Q10 Please provide missing page range for reference ‘Zhang et al. 2018’.
Q11 Please provide missing volume and page number for reference ‘Zhang et al. 2020’.
Disclaimer
This is the uncorrected version of your paper sent to you with the DOI that will be used for the published paper (Version of
Record). The uncorrected version will show online while the following services are applied to your manuscript; copyediting,
proofreading and typesetting. To see the most current version of your paper, please use the DOI provided.
