Content uploaded by Romain Lardy
Author content
All content in this area was uploaded by Romain Lardy on Aug 18, 2014
Content may be subject to copyright.
International Environmental Modelling and Software Society (iEMSs)
7th Intl. Congress on Env. Modelling and Software, San Diego, CA, USA,
Daniel P. Ames, Nigel W.T. Quinn and Andrea E. Rizzoli (Eds.)
http://www.iemss.org/society/index.php/iemss-2014-proceedings
Calibration of simulation platforms including highly
interweaved processes: the MAELIA multi-agent
platform
Romain Lardy1,2, Pierre Mazzega3,4, Christophe Sibertin-Blanc1, Yves Auda3, José-Miguel
Sanchez-Perez5,6, Sabine Sauvage5,6, Olivier Therond2
1 UMR 5505 IRIT, CNRS, University of Toulouse, France
2 UMR 1248 AGIR, INRA-INPT, Castanet-Tolosan, France
3 UMR5563 GET, IRD-UPS-CNRS-CNES, OMP, Toulouse, France
4 Joint Mixt Laboratory OCE, UnB / IRD, LAGEQ, Universidade de Brasília, Brazil
5"Laboratoire Ecologie fonctionnelle et Environnement, UMR CNRS 5245, 31326 Castanet-Tolosan, France
6"CNRS, UMR EcoLab, 31326 Castanet Tolosan Cedex, France
Email addresses: romain.lardy@toulouse.inra.fr; pmazzega@gmail.com; sibertin@ut-capitole.fr;
yves.auda@get.obs-mip.fr; jose-miguel.sanchez-perez@univ-tlse3.fr; sabine.sauvage@univ-tlse3.fr;
olivier.therond@toulouse.inra.fr
Abstract: The MAELIA project develops an agent-based modeling and simulation platform to study
the environmental, economic and social impacts of various regulations regarding water use and water
management in combination with climate change. An integrated modelling approach has been used to
model the investigated social-ecological system. MAELIA combines spatiotemporal models of
ecologic (e.g. water flow and plant growth) and human decision-making processes (e.g. cropping
plan), socio-economic dynamics (e.g. land cover changes). Due to the diversity and the interweaving
of the processes considered, the calibration and evaluation of such a multi-agent platform is a
scientific challenge. Indeed, many parameters can reveal to be influential on the model outputs, with a
high level of interactions between parameters impacts. In order to get an overview of the model
behaviour and to screen influential parameters, multiple sensitivity analyses were performed, while
considering some sub-sets of processes or not. This step-by-step sensitivity analyses enabled to
disentangle the different influences and interactions, and was a preliminary step to the calibration
process. In our case, the calibration, which is a multi-objective (e.g. reproducing water flows and
anthropic dynamics, traduced by different numerical criteria such as joint use of L2-norm with
variance-covariance matrix and indices of squared errors on water crisis temporality) optimization
problem, was achieved thanks to metamodels built on an appropriated design of experiments.
Keywords: multi-objective calibration; sensitivity analysis; multi-agent platform, integrated modelling
1. INTRODUCTION
Water is a critical resource for a number of social and human activities and for the sustainability of
ecosystems. To study such question, integrated assessment and modelling has been playing an
increasing role since last decades (Jakeman et al., 2006). The MAELIA project developed a high-
resolution agent-based modeling and simulation platform to study the environmental, economic and
social impacts of various regulations regarding water use and water management in combination with
climate change. An integrated modelling approach has been used to model the investigated social-
ecological system (Gaudou et al., 2013). MAELIA combines spatiotemporal models of ecologic (e.g.
water flow and plant growth), human decision-making processes (cropping plan and crop
management, water releases from dams, water use restriction) and socio-economic dynamics (e.g.
demography and land cover changes).
R. Lardy et al. Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
Sensitivity analysis aims at increasing our knowledge of models through the study of the influence of
parameters on outputs. It allows to distinguish influential parameters and to quantify their influences,
in order to point which require a calibration step. Some authors (e.g. Sorooshian and Gupta, 1995)
consider that the parameter specification steps (i.e. the choice of parameters) is one of the most
important step of the calibration process. For hydrological models, calibration is a difficult problem,
due to the complex structure of models and the high number of parameters to consider. Moreover,
running those models is usually time consuming, which induces high constrains on the calibration
process (Tang et al., 2006). In addition to the hand-calibration (also called “trial-and-error”) which
needs a lot of expertise and is time-consuming, a large set of algorithms and criteria exists (e.g.
Bekele and Nicklow, 2007; Zhang et al., 2009). In the hydrological model domains, local search
algorithms (such as gradient descent) have been replaced by genetic algorithm to avoid convergence
problem (Bekele and Nicklow, 2007). Calibration of platforms simulating interactions between agents
decision-making and ecological processes, of which hydrological ones, like MAELIA, remains a great
scientific challenge.
This paper presents methods developed to calibrate the MAELIA multi-agent simulation platform
(Figure 1). The second section gives a brief description of the MAELIA model which reveals to be time
consuming (about 6 hours for 10 years simulations) and memory demanding (up to 5 Go of RAM
needed), so that the number of simulations that can be run is strongly reduced. The third and the
fourth section detail respectively the sensitivity analysis method (step 2 of Figure 1) and the
calibration process (step 3 and 4). The need for an efficient sensitivity analysis method leads to chose
the Morris method (Morris, 1991), but to be able to distinguish all parameter’s effects, we applied a
multiple sensitivity analysis. For the same reason, the optimisation method is based on the Design
and Analysis of Simulation Experiment (DACE) (Kleijnen, 2008). For calibration itself, we used the
Multi-point Approximation Method (MAM) (e.g. Polynkin and Toropov, 2012). The principle of this
method is to replace the original optimization problem by a succession of simpler and time-negligible
problems. This approximation is achieved thanks to metamodels on a limited part of the parameter
space. Once a local optimum is found, we start from this point for the next step. In section five, we
analysis and discuss the results. The concluding section identifies key results and explores future
research needs.
2. THE MAELIA MODEL
The Maelia model is implemented with GAMA (Taillandier et al., 2012), an open-source generic
agent-based modeling and simulation platform. The model represents interactions between ecological
and socio-economic processes and human activities. The water flows representation is based on the
SWAT model (Soil and Water Assessment Tool; Arnold et al., 1998). A description of the agricultural
processes can be found in (Murgue et al., 2014, this conference) and a more complete description of
the whole Maelia model in (Therond et al., this conference). The Maelia model uses a lot of data
coming from different sources. Most of them require pre-processing to solve heterogeneity,
compatibility and consistency issues and to be put at the required temporal and spatial resolutions
(i.e. form days to yearly resolution, and plot to region scale). Input data relate to environmental
(climate, soil properties, topography, …) and socio-economical (e.g. land use and land cover
changes, domestic and industrial water use, water norms, …) characteristics of a water basin. The
model is applied in the Adour-Garonne Basin (AGB, South-West France), were water scarcity is a
serious problem with an annual deficit between demands and resources of 250 million m3. In this river
basin, irrigated agriculture is the main consumer of water (about 80%) during the low-water period.
The model is modular in the way that the user can activate or not some module (e.g. farmers modules
with prescribed land-use or farmers with decision-making of land-use). This aspect is important, as it
helps reproducing different type of situations.
3. SENSITIVITY ANALYSIS
Sensitivity analysis aims to provide significant information on the model behaviour. This is the key first
step of model validation process. It allows:
- to check model stability (failure frequency or model divergence) ;
R. Lardy et al. Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
- to verify whether parameters are influential at an expected level on the right processes (e.g.
whether some parameters are influential whereas they should not, or at the opposite if they
are not whereas they should be) ;
- to check the interactions between processes by quantifying the influence of parameters
characteristic of one process on other processes.
3.1. The Morris method
The Elementary Effects screening method initially developed by Morris (1991) and extended by
Campolongo et al. (2007) allows identifying the important parameters of a model, including those
involved in interactions. It is based on a “One-factor-At-a-Time” (OAT) design of experiments, and is
generally used when the number of model parameters is large enough to require computationally
expensive simulations. For each parameter, two sensitivity estimates are obtained, both based on the
calculation of incremental ratios at various points in the input space of parameters. We improved the
exploration of the parameter space by the use of a Latin Hypercube Sampling (LHS, McKay et al.,
1979), as already applied, for example, in Van Griensven et al. (2002). Around each point of the LHS
of dimension t, an OAT is achieved, so the total number of model evaluations needed is t(n+1), where
n is the number of parameter. In order to improve the quality of the design we used an LHS
maximised by the ‘maximin’ criterion (Johnson et al., 1990), which maximizes the minimum distance
between two points of the design.
3.2. Implementation
As our aim is at the same time to screen influential parameters for calibration and to check the model
consistency, we choose an approach based on multiple sensitivity analysis. This step-by-step
sensitivity analyses enabled to disentangle the different influences and interactions.
In our case, the sensitivity analyses were performed respectively over 30, 7 or 37 parameters (all
belonging respectively to the hydrologic sub-model, to the farmers sub-model with forced cropping
plan or farmer choice, or to the full model). The chosen size of the LHS is twelve, and the elementary
increment of the OAT corresponds to a shift of 1/12 probability over his Uniform distribution. The
resulting number of simulations (372, for 12 local OAT of 31 simulations; 96 for 12 local OAT of 8
simulations; 456 for 12 local OAT of 38 simulations) is in accordance with literature which suggest at
least five OAT for robustness (Confalonieri et al., 2010), and the same number of levels and
trajectories (e.g. Saltelli et al., 2004). As we do not have enough information to chose a distribution,
we used the Uniform one as in the SWAT literature (e.g. Moreau et al., 2013) where authors also
recognize that they do not have enough information to determine a distribution curve. The use of
Uniform distribution is highly common when the main objective is to get knowledge about the model
behaviour (Monod et al., 2006). Simulations were performed over ten years (2000-2009), but the two
first years were considered as spin-up period and ignored when looking at the hydrologic sensitivity.
Table 1. List of output considered in the sensitivity analyses
Description
Unit
Real Evapotranspiration
mm
Soil water content
mm
Percolation
mm
Water input in aquifers
mm
Deep water aquifers input
mm
Capillary rise water amount
mm
Shallow water aquifers content
mm
Runoff
mm
Lateral flow
mm
Deep water flow
mm
Total irrigation volume
m3
Water flow over 22 locations
(hydrographic points)
m3 s-1
Daily Farmers work
h
Number of working dates
-
Working dates
Doy
First Irrigation date of each farmer
Doy
Last Irrigation date of each farmer
Doy
Crop yields
t ha-1
Surfaces per type of crops
ha
Number of parcels per type of crops
-
R. Lardy et al. / Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
Simulations were partly distributed on a local computer (Quad-Core Intel Xeon: 8 threads with 32 Go
RAM) and on the VO biomed European grid. Then sensitivity indices were calculated thanks to the
‘sensitivity’ R package (http://rss.acs.unt.edu/Rdoc/library/sensitivity/html/sensitivity-package.html).
Sensitivity of 14 types of outputs (Table 1) was calculated, by considering scaled parameters (i.e. a
[0; 1] parameter range values). The sensitivity was considered either over the full year or during the
low water period (1st of May-30th September). For each output, sensitivity of the average or the
standard deviation, were studied, excepted for dates were the coefficient of variation was used as a
proxy of uniformity measure. The hydrologic sensitivity was also check over spatial pattern (e.g.
upstream and downstream area).
4. CALIBRATION
Once influential parameters are ranked (step 2), we can start the calibration step (step 3 and 4, on
Figure 1) on parameters that have not been discard as non (or not enough) influential parameters at
the sensitivity analysis step. However, as our model is time consuming, sequential approaches (e.g.
classical Bayesian calibration) would not be possible. We need a distributed optimisation method and
the use of high performance computing to achieve it. But still with the use of the European grid, the
number of simulations must remain low, as the number of available computed nodes is limited by the
length of the simulation (about 6 hours) and the RAM (about 6 Go) used by each simulation. This is
why we used the Design and Analysis of Simulation Experiment (DACE) (Kleijnen, 2008) domain.
Indeed, the use of a proper design of experiments allows to reduce the number of simulations and by
the way the needed computing time. On top of that, we think that the use of metamodels is necessary
to achieve our calibration. A response surface, also called metamodel, is a model or approximation of
this implicit Input/Output (I/O) function that characterizes the relationships between inputs and outputs
in much simpler terms than the full simulation (Kleijnen et al., 2005).
Figure 1 Description of the calibration process
Another aspect of our calibration problem is that we want to perform a multi-objective calibration. To
solve such inverse problem, different types of optimisation methods exist. Two main strategies can be
used. In the first one, we rank criteria and optimize them in this order or combine criteria into a single
objective function. While the second strategy consists in searching for a set of parameters values on
the Pareto front. Keeping a set of parameters allows building the parameter distribution and so easily
performing an uncertainty analysis at the end of our process. In order to reduce the size of the
calibration problem, we will first calibrate the hydrological parameters (step 3) on the first data set.
Then, a second step (step 4, on Figure 1) will consist in calibrating other influential parameters (i.e.
R. Lardy et al. / Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
non hydrologic parameters, e.g. the number of working hours per day per labor unit) and adjusting
previously calibrated parameters (e.g. no change of value higher than 15 %) if needed.
To perform our calibration, we used the Multi-point Approximation Method (MAM) (e.g. Polynkin and
Toropov, 2012), which is a multi-objective calibration method, based on the DACE domain. The
principle of this method is to replace the original optimization problem by a succession of simpler and
negligible in time problems. This approximation is achieved thanks to metamodels on a limited part of
the parameter space. Once a local optimum is found, we start from this point for the next step. A new
parameter space is defined and optimization is repeated until convergence. Metamodels are
regressed on simulation from a local experimental plan. The MAM is known to be able to deal with a
high number of parameters (Polynkin and Toropov, 2012).
4.1. Comparison data
To perform our calibration, we have access to two sets of (semi) observed data. The first one consists
of water flow data on which irrigation effects have been removed (data provided by the SMEAG, a
public institute in charge of water managment), named hereafter unaffected data. They cover the low
water period (1st June to 31st October) from 1979 to 2008 on three locations (hydrographic points).
This data set was used in the first calibration step (step 3, in Figure 1). Indeed, by using the model
with forced land use, we get a model mainly sensitive to hydrologic parameters and that reproduce
the comparison data set. The second data set contains observed data at 22 locations over the whole
case study area from 2000 to 2010 (data provided by the regional State service for environment). It
will be used in the second calibration step (step 4, in Figure 1).
4.2. Comparison criteria
As our aim is to calibrate the model in order to well reproduce water flow (values and dynamics)
and more precisely during the low water period, we used four different criteria. The first one aims at
reproducing values and dynamic of flow during the low water period while the three others focus on
dates and length of this period. Our first criterion (C1) of comparison is based on the L2 norm with the
variance-covariance matrix estimated on unaffected data. Some weights are added to give more
influence to low water period than to the rest of the year, and to reduce influence of hydrological flow
peaks. The three other criteria correspond to the squared error of the length, the starting date and the
ending date of the low water periods. Each criterion was calculated on the three hydrographic points
of the unaffected data set. Then, in order to reduce our number of criteria (twelve, i.e. four per
location), we summed the sites, leading to only four numerical criteria to optimize.
4.3. Implementation
We only present preliminary results of calibration to illustrate this approach. After selection of most
influential parameters (step 2), we started the calibration process with an LHS (step 3.1 and 3.2),
optimized by the maximin criterion, which contains 160 points. We regressed three different kriging
metamodels (depending on covariance assumption) (step 3.3). For each metamodel, we used 600
chains of the first descent heuristic (French, 1982) (a local search metaheuristic) to search for the
optimal parameter sets (step 3.4). Based on theses optimal points, we searched for those located on
the Pareto front. To define the new trusted area (step 3.5) for next step of the calibration, we merged
the intervals [5 – 95 %] of each metamodels, and then run again the whole procedure.
5. RESULTS AND DISCUSSION
Thanks to the sensitivity analysis, we got a list of influential parameters (Table 2). But it was also
useful to check (and to correct) the model consistency and to disentangle the different effects (for
example, to understand how some hydrologic parameter may appear as the most influential on the
irrigation process). In addition of bug detection, the sensitivity analysis led us to question the precision
R. Lardy et al. / Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
of some forcing data. Indeed, getting some process (runoff and snow processes) more influential than
expected has incited us to refine the number of slope and altitude classes in input (data not shown).
The list of influential parameters is consistent with the SWAT literature. However some difference may
be noticed. For example, the initial water content of shallow aquifers is influential in our model
whereas most papers ignore it. This implies that a spin-up run step should be added to the model.
Moreover, one can see a spatial pattern on influential parameter, which may imply that it will be
necessary to calibrate more than one set of parameters. For example, we could calibrate one set for
the lowland and one for the upland part, or one set per sub-watershed.
Our sensitivity analysis is robust relatively to the size of the design of experiment. Indeed, the list of
influential parameters and their ranking has already converged whether we use eight instead of
twelve chains (data not shown). Moreover previous studies have shown that the Morris method is
relevant for screening parameters of environmental models (e.g. Drouet et al., 2011) and that the
ranking of parameters is consistent with more accurate (and more expensive) methods.
Each step of the calibration procedure allows for a reduction of the Pareto-optimal distribution
(Figure 2) and a more accurate parameter set. In order to improve the efficiency of the calibration, we
could make new iterations. This will allow us to get a more optimal parameter set, and above all a
more precise parameter set distribution and therefore a more accurate uncertainty evaluation (step 6,
in Figure 1). It is interesting to note that if some default parameter value (i.e. the one from literature)
are well situated in the Pareto-optimal set, some other shows high difference, which might be due to
the specificity of the AGB basin or to the SWAT integration into MAELIA.
Table 2. List of parameter considered as influential on water flow in the MAELIA platform. For each
parameter, we consider whether the mean or standard deviation of the elementary effect of the
parameter is higher than 10% of the maximum value.
Parameter
Frequency of occurrence of the
parameter above the threshold
Surface runoff lag time
100%
SCS curve number for forest
100%
Threshold water level in shallow aquifer for revap process
83%
Grassland leaf Area
83%
Deep groundwater revap coefficient
75%
SCS curve number for grassland
75%
Shallow aquifer initialisation
75%
Groundwater revap coefficient
67%
Manning coefficient for tributary channel
58%
Precipitations change with altitude
58%
Groundwater delay
50%
Threshold water level in shallow aquifer for baseflow process
50%
Temperature change with altitude
50%
Minimum snowmelt rate
50%
Snow fall min temperature
42%
SCS curve number for water
25%
Maximum snowmelt rate
17%
R. Lardy et al. / Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
Figure 2 Boxplots of Pareto-optimal distribution obtained after the second step of the multi-point
approximation method, for 6 parameters that have already converged. The values are scaled by the
parameter range from the literature. The black bold line correspond to the median, the box represent
the 0.25 to 0.75 quantile range. The whiskers extend to the"1.5 of the interquartile range from the box,
and the red line corresponds to the default parameter value.
6. CONCLUSION
This paper presents a still going calibration process, that illustrates well the issues of calibrating a
complex multi-agent model including an important number of interweaved heterogeneous processes.
Indeed, the most common calibration methods would be inefficient due especially to the interactions
between formalisms, which have different forms (from classical differential equations to agent
behaviour algorithms) and spatiotemporal resolutions. In order to get an overview of the model
behaviour and to screen influential parameters, multiple sensitivity analyses were performed, while
considering some sub-sets of processes or not. This step-by-step sensitivity analyses enabled to
disentangle the different influences and interactions, and was a preliminary step to the calibration
process. In our case, the calibration, which is a multi-objective (e.g. reproducing water flows and
anthropic dynamics, traduced by different numerical criteria such as joint use of L2-norm with
variance-covariance matrix and indices of squared errors on water crisis temporality) optimization
problem, was achieved thanks to the Multi-Point Approximation method. This latter consists in a
succession of optimisation on metamodels thanks to a metaheuristic. The approximation models were
regressed on an appropriate succession of models. This calibration approach gives us an ensemble
of parameters set (all of them placed on the Pareto front). Based on that, the evaluation of the model
will be completed by an uncertainty analysis.
ACKNOWLEDGMENTS
The MAELIA Project “Multi-Agents for EnvironmentaL norm Impact Assessment”
(http://maelia1.wordpress.com/) is funded by the French “Sciences & Technologies for Aeronautics
and Space” Foundation (http://www.fondationstae.net/). Results obtained in this paper were computed
on the biomed virtual organization of the European Grid Infrastructure (http://www.egi.eu). We thank
the European Grid Infrastructure and supporting National Grid Initiatives (listed here:
http://lsgc.org/en/Biomed:home#Supporting_National_Grid_Initiatives) for providing the technical
support, computing and storage facilities.
REFERENCES
Arnold, J.G., Srinivasan, R., Muttiah, R.S., Williams, J.R., 1998. Large Area Hydrologic Modeling and
Assessment Part I: Model Development1. JAWRA J. Am. Water Resour. Assoc. 34, 73–89.
Bekele, E.G., Nicklow, J.W., 2007. Multi-objective automatic calibration of SWAT using NSGA-II. J.
Hydrol. 341, 165–176.
Cibin, R., Sudheer, K.P., Chaubey, I., 2010. Sensitivity and identifiability of stream flow generation
parameters of the SWAT model. Hydrol. Process. 24, 1133–1148.
R. Lardy et al. / Calibration of simulation platforms including highly interweaved processes: the MAELIA multi-agent platform"
Confalonieri, R., Bellocchi, G., Bregaglio, S., Donatelli, M., Acutis, M., 2010. Comparison of sensitivity
analysis techniques: a case study with the rice model WARM. Ecol. Model. 221, 1897–1906.
Drouet, J.-L., Capian, N., Fiorelli, J.-L., Blanfort, V., Capitaine, M., Duretz, S., Gabrielle, B., Martin, R.,
Lardy, R., Cellier, P., Soussana, J.-F., 2011. Sensitivity analysis for models of greenhouse gas
emissions at farm level. Case study of N2O emissions simulated by the CERES-EGC model.
Environ. Pollut. 159, 3156–3161.
French, S., 1982. Sequencing and scheduling: an introduction to the mathematics of the job-shop.
Ellis Horwood Chichester.
Gaudou, B., Sibertin-Blanc, C., Therond, O., Bai, V., Vavasseur, M., Mazzega, P., 2013. The maelia
multi-agent platform for integrated assessment of low-water management issues, in: MABS, Multi-
Agent-Based Simulation XIV-International Workshop (to Appear, 2013).
Jakeman, A.J., Letcher, R.A., Norton, J.P., 2006. Ten iterative steps in development and evaluation of
environmental models. Environ. Model. Softw. 21, 602–614.
Kleijnen, J.P., 2008. Design of experiments: overview, in: Simulation Conference, 2008. WSC 2008.
Winter. IEEE, pp. 479–488.
Kleijnen, J.P.C., Sanchez, S.M., Lucas, T.W., Cioppa, T.M., 2005. State-of-the-Art Review: A User’s
Guide to the Brave New World of Designing Simulation Experiments. Inf. J. Comput. 17, 263–289.
Monod, H., Naud, C., Makowski, D., 2006. Uncertainty and sensitivity analysis for crop models, in:
Wallack, D., Makowski, D., Jones, J.W. (Eds.), Working with Dynamic Crop Models. Elsevier,
Amsterdam; Boston, pp. 55–100.
Moreau, P., Viaud, V., Parnaudeau, V., Salmon-Monviola, J., Durand, P., 2013. An approach for
global sensitivity analysis of a complex environmental model to spatial inputs and parameters: A
case study of an agro-hydrological model. Environ. Model. Softw. 47, 74–87.
Morris, M.D., 1991. Factorial Sampling Plans for Preliminary Computational Experiments.
Technometrics 33, 161–174.
Muleta, M.K., Nicklow, J.W., 2005. Sensitivity and uncertainty analysis coupled with automatic
calibration for a distributed watershed model. J. Hydrol. 306, 127–145.
Murgue, C., Lardy, R., Vavasseur, M., Leenhardt, Therond, O., 2014. Fine spatio-temporal simulation
of cropping and farming systems effects on irrigation withdrawal dynamics within a river basin, in:
Ames, D.P., Quinn, N., Rizzoli, A.E. (Eds.), Presented at the iEMSs, 7th International Congress on
Environmental Modelling and Software, San Diego, California, USA.
Polynkin, A., Toropov, V.V., 2012. Mid-range metamodel assembly building based on linear
regression for large scale optimization problems. Struct. Multidiscip. Optim. 45, 515–527.
Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity analysis in practice: a guide to
assessing scientific models. John Wiley & Sons, Hoboken, NJ.
Sorooshian, S., Gupta, V.K., 1995. Model calibration, in: Singh, V.P. (Ed.), Computer Models of
Watershed Hydrology. Water Ressources Publication, Highlands Ranch, CO.
Taillandier, P., Vo, D.-A., Amouroux, E., Drogoul, A., 2012. GAMA: a simulation platform that
integrates geographical information data, agent-based modeling and multi-scale control, in:
Principles and Practice of Multi-Agent Systems. Springer, pp. 242–258.
Tang, Y., Reed, P., Wagener, T., 2006. How effective and efficient are multiobjective evolutionary
algorithms at hydrologic model calibration? Hydrol. Earth Syst. Sci. Discuss. 10, 289–307.
Therond, O., Sibertin-Blanc, C., Lardy, R., Gaudou, B., Balestat, M., Hong, Y., Louail, T., Nguyen,
V.B., Panzoli, D., Sanchez-Perez, J.-M., Sauvage, S., Taillandier, P., Vavasseur, M., Mazzega, P.,
2014. Integrated modelling of social-ecological systems: The MAELIA high-resolution multi-agent
platform to deal with water scarcity problems, in: Ames, D.P., Quinn, N., Rizzoli, A.E. (Eds.),
Presented at the iEMSs, 7th International Congress on Environmental Modelling and Software, San
Diego, California, USA.
Zhang, X., Srinivasan, R., Bosch, D., 2009. Calibration and uncertainty analysis of the SWAT model
using Genetic Algorithms and Bayesian Model Averaging. J. Hydrol. 374, 307–317.
