Content uploaded by Qingyun Duan
Author content
All content in this area was uploaded by Qingyun Duan on Dec 02, 2017
Content may be subject to copyright.
A preview of the PDF is not available
Optimal Use of the SCE-UA Global Optimization Method for Calibrating Watershed Models
Abstract and Figures
The difficulties involved in calibrating conceptual watershed models have, in the past, been partly attributable to the lack of robust optimization tools. Recently, a global optimization method known as the SCE-UA (shuffled complex evolution method developed at The University of Arizona) has shown promise as an effective and efficient optimization technique for calibrating watershed models. Experience with the method has indicated that the effectiveness and efficiency of the algorithm are influenced by the choice of the algorithmic parameters. This paper first reviews the essential concepts of the SCE-UA method and then presents the results of several experimental studies in which the National Weather Service river forecast system-soil moisture accounting (NWSRFS-SMA) model, used by the National Weather Service for river and flood forecasting, was calibrated using different algorithmic parameter setups. On the basis of these results, the recommended values for the algorithmic parameters are given. These values should also help to provide guidelines for other users of the SCE-UA method.
Figures - uploaded by Qingyun Duan
Author content
All figure content in this area was uploaded by Qingyun Duan
Content may be subject to copyright.
... Recognizing that multiple soil parameters influence various state variables and flux outputs in hydrological model simulations, a robust optimization of the identified sensitive soil parameters was essential to prevent overfitting in the model's outputs. To address this, the Shuffled Complex Evolution algorithm (SCE-UA) proposed by (Duan et al., 1994) was employed to optimize these sensitive model soil parameters, bridging the gap between the model-simulated and SMAP-derived soil moisture. The SMAP product was used as the actual observations of soil moisture in this study. ...
... The optimal values for the sensitive soil parameters were determined using the modified SCE-UA optimization algorithm (Duan et al., 1994). Numerous studies (Cooper et al., 1997;Franchini et al., 1998;Kuczera, 1997;Thyer et al., 1999) have demonstrated that the modified SCE-UA algorithm outperforms other local and global search optimization algorithms in the automatic calibration of hydrological models. ...
Estimation of accurate soil physical and hydraulic properties are of prime importance for the management of water resources in agriculture-dominant regions. This study introduces a simplified framework for estimating soil physical and hydraulic properties crucial for managing agricultural water resources. The developed framework optimizes soil properties for the Regional Hydrological Extremes Assessment System (RHEAS) to enhance the performance of its core hydrological model, Variable Infiltration Capacity (VIC). These soil properties were optimized using six years (2015–2021) of satellite soil moisture observations from NASA’s Soil Moisture Active Passive (SMAP) mission with a modified Shuffled Complex Evolution (SCE-UA) optimization algorithm. A total of three most sensitive soil properties that control model soil moisture simulations, such as Ksat (Saturated hydraulic conductivity), expt (exponent parameter in Campbell’s equation for hydraulic conductivity), and bd (Bulk density) were optimized for the Lower Mekong River (LMR) basin. To better assess the impact of optimized soil properties, streamflow simulation as well as agricultural drought severity assessment, were estimated using the RHEAS framework’s VIC Routing module and Soil Moisture Deficit Index (SMDI) module, respectively. The streamflow simulation involved four approaches: an initial open-loop setup, one optimized with SMAP soil moisture data (SMAP), another optimized with actual streamflow data (Runoff), and a final one combining the previous two datasets (SMAP_Runoff). Switching from the initial setup to the SMAP-optimized model increased the Nash-Sutcliffe Efficiency (NSE) by 56.4 % and upgrading from the streamflow-optimized to the combined data model raised the NSE by 21.9 %. This showcases the benefits of optimizing soil properties for more accurate simulations. Furthermore, the optimized model accurately represented the severity and extent of historical agricultural droughts, aligning with regional drought reports of LMR basin. This framework offers a valuable tool for hydrological modeling and drought management, particularly in data-scarce and agriculture-intensive regions, informing agricultural water resource management, irrigation decision-making, and food security initiatives within the LMR basin and beyond.
... With the establishment of the ML-based relationship between the remediation input and output variables, an optimization algorithm developed at the University of Arizona (SCE-UA [52]) was used to search for optimal strategies and associated parameters by minimizing the total remediation cost (e.g., EX treatment volume) under different regulatory constraints and site requirements. The SCE-UA algorithm was chosen due to its excellent performance and efficiency in global search [53][54][55] with the detailed optimization procedure described elsewhere [56]. . ...
... With the establishment of the ML-based relationship between the remediation input and output variables, an optimization algorithm developed at the University of Arizona (SCE-UA [52]) was used to search for optimal strategies and associated parameters by minimizing the total remediation cost (e.g., EX treatment volume) under different regulatory constraints and site requirements. The SCE-UA algorithm was chosen due to its excellent performance and efficiency in global search [53][54][55] with the detailed optimization procedure described elsewhere [56]. ...
A numerical approach assisted by machine learning was developed for screening and optimizing soil remediation strategies. The approach includes a reactive transport model for simulating the remediation cost and effect of applicable remediation technologies and their combinations for a target site. The simulated results were used to establish a relationship between the cost and effect using a machine learning method. The relationship was then used by an optimization method to provide optimal remediation strategies under various constraints and requirements for the target site. The approach was evaluated for a site contaminated with both arsenic and polycyclic aromatic hydrocarbons at a former shipbuilding factory in Guangzhou City, China. An optimal strategy was obtained and successfully implemented at the site, which included the partial excavation of the contaminated soils and natural attenuation of the residual contaminated soils. The advantage of the approach is that it can fully consider the natural attenuation capacity in designing remediation strategies to reduce remediation costs and can provide cost-effective remediation strategies under variable constraints for policymakers. The approach is general and can be applied for screening and optimizing remediation strategies at other remediation sites.
... The solver of "mnewt" is used to solve equations by iteratively calculating the values of model function "modelx" and its Jacobian matrix "modeljacx" (see the codes for details in the "Code and data availability" section). The Shuffled Complex Evolution Algorithm (SCE-UA) has been proven to be a robust method for parameter optimization (Duan et al., 1994;Muttil and Jayawardena, 2008), and the SCE-UA method from the "spotpy" package in Python (Houska et al., 2015; https://pypi.org/project/spotpy/, last access: 7 June 2024) was applied here. ...
Biochar (BC) application to croplands aims to sequester carbon and improve soil quality, but its impact on soil organic carbon (SOC) dynamics is not represented in most land models used for assessing land-based climate change mitigation; therefore, we are unable to quantify the effects of biochar application under different climate or land management conditions. Here, to fill this gap, we implement a submodel to represent biochar in a microbial decomposition model named MIMICS (MIcrobial-MIneral Carbon Stabilization). We first calibrate and validate MIMICS with new representations of the density-dependent microbial turnover rate, adsorption of available organic carbon on mineral soil particles, and soil moisture effects on decomposition using global field-measured cropland SOC at 285 sites. We further integrate biochar in MIMICS by accounting for its effect on microbial decomposition and SOC sorption/desorption and optimize two biochar-related parameters in these processes using 134 paired SOC measurements with and without biochar addition. The MIMICS-biochar version can generally reproduce the short-term (≤ 6 years) and long-term (8 years) SOC changes after adding (mean addition rate of 25.6 t ha−1) biochar (R2= 0.79 and 0.97, respectively) with a low root-mean-square error (RMSE = 3.73 and 6.08 g kg−1, respectively). Our study incorporates sorption and soil moisture processes into MIMICS and extends its capacity to simulate biochar decomposition, providing a useful tool to couple with dynamic land models to evaluate the effectiveness of biochar application with respect to removing CO2 from the atmosphere.
... Due to the limited duration of the annual precipitation, potential evapotranspiration, and runoff series in the reference period (1956)(1957)(1958)(1959)(1960)(1961)(1962)(1963)(1964)(1965)(1966)(1967)(1968), the entire hydro-meteorological datasets for that period were employed for parameter calibration in the GR1A model. Utilizing the SCE-UA optimization algorithm [48], the parameter of the GR1A model was calibrated, resulting in an optimized parameter X of 0.79. Figure 5 shows the simulated and observed annual runoff processes for the reference period (1956-1968), demonstrating a general agreement. The NSE and R 2 values of the GR1A model were both 0.83, and the Bias was 1.76%. ...
The natural hydrological cycle of basins has been significantly altered by climate change and human activities, leading to considerable uncertainties in attributing runoff. In this study, the impact of climate change and human activities on runoff of the Ganjiang River Basin was analyzed, and a variety of models with different spatio-temporal scales and complexities were used to evaluate the influence of model choice on runoff attribution and to reduce the uncertainties. The results show the following: (1) The potential evapotranspiration in the Ganjiang River Basin showed a significant downward trend, precipitation showed a significant upward trend, runoff showed a nonsignificant upward trend, and an abrupt change was detected in 1968; (2) The three hydrological models used with different temporal scales and complexity, GR1A, ABCD, DTVGM, can simulate the natural distribution of water resources in the Ganjiang River Basin; and (3) The impact of climate change on runoff change ranges from 60.07% to 82.88%, while human activities account for approximately 17.12% to 39.93%. The results show that climate change is the main driving factor leading to runoff variation in the Ganjiang River Basin.
... However, theoretically the truth could never be known for any variable or system even with abundant observation records. To avoid such a problem, we here apply another method, that is, the SCE-UA (Shuffled Complex Evolution-University of Arizona (Duan et al., 1992(Duan et al., , 1993(Duan et al., , 1994) algorithm to obtain soil texture from the satellite-estimated θ w and θ c without explicitly solving the PTF equations. The basic working flow of this algorithm is depicted in Figure 2. ...
Soil moisture (SM) plays an important role in regulating regional weather and climate. However, the simulations of SM in current land surface models (LSMs) contain large biases and model spreads. One primary reason contributing to such model biases could be the misrepresentation of soil texture in LSMs, since current available large‐scale soil texture data are often generated from extrapolation algorithm based on a scarce number of in‐situ geological measurements. Fortunately, recent advancements in satellite technology provide a unique opportunity to constrain the soil texture data sets by introducing observed information at large spatial scales. Here, two major soil texture baseline data sets (Global Soil Data sets for Earth system science, GSDE and Harmonized World Soil Data from Food and Agriculture Organization, HWSD) are optimized with satellite‐estimated soil hydraulic parameters. The optimized soil maps show increased (decreased) sand (clay) content over arid regions. The soil organic carbon (SOC) content increases globally especially over regions with dense vegetation cover. The optimized soil texture data sets are then used to run simulations in one example LSM, that is, Noah LSM with Multiple Parameters. Results show that the simulated SM with satellite‐optimized soil texture maps is improved at both grid and in‐situ scales. Intercase comparison analyses show the SM improvement differs between simulations using different soil maps and soil hydraulic schemes. Our results highlight the importance of incorporating observation‐oriented calibration on soil texture in current LSMs. This study also joins the call for a better soil profile representation in the next generation of Earth System Models (ESMs).
... The shuffled complex evolution-University of Arizona (SCE-UA) optimization algorithm has been used to calibrate the model parameters. SCE-UA is a popular evolutionary algorithm (Duan et al., 1994) for global optimization, which has been extensively used to calibrate hydrological models (Liu et al., 2019;Xue et al., 2016). While KGE is chosen as the objective function for the calibration, other performance indicators have been used to test the model performance. ...
Satellite-based precipitation estimates are a critical source of information for understanding and predicting hydrological processes at regional or global scales. Given the potential variability in the accuracy and reliability of these estimates, comprehensive performance assessments are essential before their application in specific hydrological contexts. In this study, six satellite-based precipitation products (SPPs), namely, CHIRPS, CMORPH, GSMaP, IMERG, MSWEP, and PERSIANN, were evaluated for their utility in hydrological modeling, specifically in simulating streamflow using the Variable Infiltration Capacity (VIC) model. The performance of the VIC model under varying flow conditions and timescales was assessed using statistical indicators, viz., R², KGE, PBias, RMSE, and RSR. The findings of the study demonstrate the effectiveness of VIC model in simulating hydrological components and its applicability in evaluating the accuracy and reliability of SPPs. The SPPs were shown to be valuable for streamflow simulation at monthly and daily timescales, as confirmed by various performance measures. Moreover, the performance of SPPs for simulating extreme flow events (streamflow above 75%, 90%, and 95%) using the VIC model was assessed and a significant decrease in the performance was observed for high-flow events. Comparative analysis revealed the superiority of IMERG and CMORPH for streamflow simulation at daily timescale and high-flow conditions. In contrast, the performances of CHIRPS and PERSIANN were found to be poor. This study highlights the importance of thoroughly assessing the SPPs in modeling diverse flow conditions.
... It was calibrated using the Kling-Gupta Efficiency (KGE) 175 objective function (Kling et al., 2012;Kling & Gupta, 2009) with streamflow observation data and ERA5 data spanning 1981-2018. The calibration process's duration varied depending on the availability of streamflow data for each catchment, requiring at least 10 years of observation data, including a 2-year warm-up period, entailed and 10,000 model evaluations using the SCE-UA (Shuffled Complex Evolution -University of Arizona; Duan et al., 1994) algorithm. 180 ...
Climate change impact studies rely on ensembles of General Circulation Model (GCM) simulations. Combining ensemble members is challenging due to uncertainties in how well each model performs. The concept of model democracy where equal weight is given to each model, is common but criticized for ignoring regional variations and dependencies between models. Various weighting schemes address these concerns, but their effectiveness in impact studies, which integrate GCM outputs with separate impact models, remains unclear. This study evaluated the impact of six weighting strategies on future streamflow projections using a pseudo-reality approach, where each GCM is treated as “the true” climate. The analysis involved an ensemble of 22 CMIP6 climate simulations and used a hydrological model across 3,107 North American catchments. Since climate model outputs often undergo bias correction before being used in hydrological models, this study implemented two approaches: one with bias correction applied to precipitation and temperature inputs, and one without. Weighting schemes were evaluated based on biases relative to the pseudo-reality GCM for annual mean temperature, precipitation and streamflow. Results show that unequal weighting schemes produce significantly better precipitation and temperature projections than equal weighting. For streamflow projections, unequal weighting offered minor improvement only when bias correction was not applied. However, with bias correction, both equal and unequal weighting delivered similar results. While bias correction has limitations, it remains essential for realistic streamflow projections in impact studies. A pragmatic strategy may be to combine model democracy with selective model exclusion based on robust performance metrics. This study provides insights on how weighting affects hydrological assessments. It emphasizes the need for careful approaches and further research to manage uncertainties in climate change impact studies. These findings will help improve the accuracy of climate projections and improve the reliability of hydrological impact assessments in a changing climate.
A stochastic parameter estimation procedure to be applied to large conceptual rainfall-runoff models is proposed. The procedure is based on a maximum likelihood approach, which is enhanced to allow for the use of prior information about some of the parameters. A stochastic model of base flow is proposed, and an algorithm for automatic identification of the time interval of base flow activity is developed. This algorithm, which permits the estimation of prior values for the base flow discharge coefficients, is successfully applied to two basins in the United States. The application of the maximum likelihood estimation procedure to large conceptual rainfall-runoff models is divided into two stages. In the first stage the procedure is applied to a simplified version of the National Weather Service River Forecasting System model, the parameters of which are estimated by using synthetically generated data. This stage produces valuable information about the performance of stochastic parameter estimation procedures in large conceptual rainfall-runoff models. In the second stage the maximum likelihood procedure is applied to estimating the parameters of the National Weather Service River Forecasting System model for the Bird Creek and for the Cohocton River basins, respectively. In the first case a severe structural error arising from a deficient formulation of the channel causes some of the parameters to converge to unrealistic values. In the second case the stochastic model performs in a mathematically correct but hydrologically unappealing fashion. The reasons for this problems are attributed to the non-identifiability of the threshold parameter of the upper zone tension water element and to interaction between several of the percolation function parameters.
The results of using different criteria for automatically fitting a nonlinear catchment model to real data are examined. One criterion used is ad hoc while the remainder, which are drawn from recent literature, are based on the maximum likelihood principle subject to different simplifying assumptions about the nature of the calculated flow residuals (the differences between measured and computed daily flows). For each criterion the model is fitted using seven different sets of initial parameter values. Final parameter values are examined for consistency. For each criterion the seven sets of final parameter values are averaged and used to predict river flows for data not used in the fitting. The reliabilities of predicted flows are then compared. The maximum likelihood criterion that assumes uncorrelated but heteroscedastic residuals is found to be the best criterion for the model and data used. Examination of 2-dimensional sections through the model's response surface shows that it is better conditioned for the maximum likelihood criteria than for the ad hoc criterion.
The National Weather Service River Forecast System consists of a set of interrelated computer programs developed to provide continuous hydrologic forecasting. The catchment model is conceptual in design and generally must be calibrated for each watershed. Current calibration procedures consist of a combination of trial-and-error and automatic-parameter optimization tech-niques. Initial parameter values are determined from analyses of the hydro-meteorological data base of a catchment. Soil moisture storage components, depletion coefficients, and other appropriate parameters are adjusted based on the results of trial-and-error simulations. An automatic-parameter optimization program utilizing a direct-search technique is used to make final modifications to the parameter values. The calibration procedures and results of a case study are presented in the paper to show how the techniques are applied. Current and important future research areas, including initial parameter estimation, physically based automatic parameter optimization, interactive computer graphics applications, and estimation theory techniques such as maximum likelihood and stochastic approximations, are discussed.
Firstly, initial parameter estimates were made from recession analysis of observed runoff. Secondly, the parameters were calibrated individually in an iteration loop starting with the snow routine, over the soil routine and finally the runoff-response function. This was done by minimizing different objective functions for different parameters and only over subperiods where the parameters were active. Approximately 350 objective function evaluations were needed to find the optimal parameter set, which resulted in a computer time of about 17 hours on a 386 processor PC for a 10-yr calibration period. -from Author
EPA’S Storm Water Management Model (SWMM) simulates all aspects of the hydrologic and quality cycles. Using expert system technology, an interactive user-support framework has been developed to automate the calibration of the runoff block. It acts as a front end to assist the user in the initial estimation of the parameter values and in building the SWMM input files. It interprets the simulation results and suggests some useful adjustments in the value of significant parameters thus reducing the user’s time and effort. For the interpretation of simulation results, production rules are employed to help the user decide what parameters need to be adjusted. Some heuristics have been developed to evaluate the new parameter values. The combination of simulation techniques and expert system methodologies facilitates the use of sophisticated models such as SWMM.
A case study is presented which illustrates some of the error analysis,
sensitivity analysis, and parameter estimation procedures reviewed in
the first part of this paper. It is shown that those procedures, most of
which come from statistical nonlinear regression theory, are invaluable
in interpreting errors in precipitation-runoff modeling and in
identifying appropriate calibration strategies.