Fig 1 - available via license: CC BY-NC-ND
Content may be subject to copyright.
Example of ensemble prediction graph provided to participants throughout the challenge; here for prediction time point 5. The grey area represents the cone of incidence predictions 1-4 weeks ahead (min and max across all teams) while the red line is the mean. The black dotted line represents the synthetic epidemic curve.
Source publication
Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014-2015 and involving 16 international academic teams and US government agencies, and c...
Contexts in source publication
Context 1
... to poor reporting in scenario 4, in which accurate in- formation on containment policies was lacking. Participants were asked to provide disease forecasts at 5 different time points of each of the 4 scenarios, typically comprising two time points in the ascending phase, a time point near the peak, and two time points in the descending phase ( Fig. ...
Context 2
... a new batch of predictions was submitted by challenge partici- pants, graphs displaying ensemble predictions for 1-4 week-ahead in- cidence were generated and shared with participants ( Fig. 1). A second workshop was held at the NIH in February 2016 after the conclusion of the challenge to review the structure of the different participating models, discuss performance results, and disseminate findings among policy and government experts. The challenge was coordinated by a team of modelers and epide- miologists from ...
Context 3
... the challenge, the coordination team computed en- semble prediction envelopes, based on the mean, minimum and maximum of the incidence forecasts submitted by the 8 participating teams (Fig. 1). Further, after the conclusion of the challenge, a Bayesian averaging approach was introduced to calculate an alternative en- semble estimate based on the point estimates of the 8 model forecasts, in which each model forecast was weighted by prediction accuracy in the previous time points (Raftery et al., 2005;Vrugt et al., 2007) ...
Context 4
... were asked to provide inter-quartile ranges (IQR) for their weekly incidence predictions; by definition of an IQR, 50% of ob- servations falling in IQRs indicates a well calibrated forecast (Supplementary Fig. 1). The mean percentage of IQRs which contained the ground-truth incidence value was 38.6% (minimum and maximum of 0% and 82.5% respectively). ...
Similar publications
Background
Mobility restrictions—trade and travel bans, border closures and, in extreme cases, area quarantines or cordons sanitaires—are among the most widely used measures to control infectious diseases. Restrictions of this kind were important in the response to epidemics of SARS (2003), H1N1 influenza (2009), Ebola (2014) and, currently in the...
Citations
... Because of the clear performance benefits, multimodel ensembles are commonplace across fields, including weather (Alley, Emanuel, and Zhang 2019), climate (Tebaldi and Knutti 2007), and economics (Aastveit et al. 2018). More recently, multi-model ensembles have been used to improve predictions of infectious disease outbreaks (Viboud et al. 2018;Johansson et al. 2019;McGowan et al. 2019;Reich et al. 2019;Cramer et al. 2022). ...
Combining predictions from multiple models into an ensemble is a widely used practice across many fields with demonstrated performance benefits. The R package hubEnsembles provides a flexible framework for ensembling various types of predictions, including point estimates and probabilistic predictions. A range of common methods for generating ensembles are supported, including weighted averages, quantile averages, and linear pools. The [hubEnsembles]{.pkg} package fits within a broader framework of open-source software and data tools called the "hubverse", which facilitates the development and management of collaborative modelling exercises.
... Understanding and predicting epidemic dynamics are crucial for public health planning and response [9,14,16]. Accurate models facilitate the timely implementation of control measures, resource allocation, and strategy development, which are essential for minimizing the impact of infectious diseases. For instance, during the COVID-19 pandemic, models by Ferguson et al. [7] guided decisions on lockdowns and social distancing, significantly influencing policy and saving lives. ...
Classical compartmental models in epidemiology often struggle to accurately capture real-world dynamics due to their inability to address the inherent heterogeneity of populations. In this paper, we introduce a novel approach that incorporates heterogeneity through a mobility variable, transforming the traditional ODE system into a system of integro-differential equations that describe the dynamics of population densities across different compartments. Our results show that, for the same basic reproduction number, our mobility-based model predicts a smaller final pandemic size compared to classic compartmental models, whose population densities are represented as Dirac delta functions in our density-based framework. This addresses the overestimation issue common in many classical models. Additionally, we demonstrate that the time series of the infected population is sufficient to uniquely identify the mobility distribution. We reconstruct this distribution using a machine-learning-based framework, providing both theoretical and algorithmic support to effectively constrain the mobility-based model with real-world data.
... However, forecasts based on these types of models can help characterize the mechanisms driving the process [7]. They may offer higher forecasting performance than purely statistical approaches based on statistical evaluation criteria like mean absolute and squared errors [8][9][10][11]. Here we focus on dynamical models that can characterize growth processes that give rise to waves of variable shapes and sizes [12][13][14]. ...
... Ensemble models that combine the strength of multiple models may exhibit significantly enhanced predictive performance (e.g [11,17,38,39]). An ensemble model derived from the top-ranking I models is denoted by the Ensemble(1), illustrated in Fig. 4. Thus, Ensemble (2) and Ensemble(3) refer to the ensemble models generated from the combination of the top-ranking 2 and 3 models, respectively. ...
Background
Dynamical mathematical models defined by a system of differential equations are typically not easily accessible to non-experts. However, forecasts based on these types of models can help gain insights into the mechanisms driving the process and may outcompete simpler phenomenological growth models. Here we introduce a friendly toolbox, SpatialWavePredict, to characterize and forecast the spatial wave sub-epidemic model, which captures diverse wave dynamics by aggregating multiple asynchronous growth processes and has outperformed simpler phenomenological growth models in short-term forecasts of various infectious diseases outbreaks including SARS, Ebola, and the early waves of the COVID-19 pandemic in the US.
Results
This tutorial-based primer introduces and illustrates a user-friendly MATLAB toolbox for fitting and forecasting time-series trajectories using an ensemble spatial wave sub-epidemic model based on ordinary differential equations. Scientists, policymakers, and students can use the toolbox to conduct real-time short-term forecasts. The five-parameter epidemic wave model in the toolbox aggregates linked overlapping sub-epidemics and captures a rich spectrum of epidemic wave dynamics, including oscillatory wave behavior and plateaus. An ensemble strategy aims to improve forecasting performance by combining the resulting top-ranked models. The toolbox provides a tutorial for forecasting time-series trajectories, including the full uncertainty distribution derived through parametric bootstrapping, which is needed to construct prediction intervals and evaluate their accuracy. Functions are available to assess forecasting performance, estimation methods, error structures in the data, and forecasting horizons. The toolbox also includes functions to quantify forecasting performance using metrics that evaluate point and distributional forecasts, including the weighted interval score.
Conclusions
We have developed the first comprehensive toolbox to characterize and forecast time-series data using an ensemble spatial wave sub-epidemic wave model. As an epidemic situation or contagion occurs, the tools presented in this tutorial can facilitate policymakers to guide the implementation of containment strategies and assess the impact of control interventions. We demonstrate the functionality of the toolbox with examples, including a tutorial video, and is illustrated using daily data on the COVID-19 pandemic in the USA.
... Predicting the trajectory of an epidemic to support control and mitigation planning is the primary objective of infectious disease forecasting. To this end, large-scale, collaborative forecasting efforts across multiple disease systems, such as influenza [1][2][3], dengue [4], West Nile [5], and Ebola viruses [6], have been integrated into routine public health workflows and emergency response [7]. Researchers in academia, private institutions, and the United States (US) government built upon these frameworks to incorporate forecasts into the COVID-19 information systems used to inform pandemic response and created the US COVID-19 Forecast Hub. ...
During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1–4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.
... There is a growing interest in comparing models and combining insights across different modelling approaches. Several projects have been undertaken in which public health questions were addressed by combining model-based outcomes in an ensemble forecast [121][122][123]. For WNV, a recent ensemble effort is the study of Holcomb et al. [124], which, based on the 2020 Centers for Disease Control and Prevention (CDC) Challenge data, found that simpler models (i.e. with fewer variables) based on historical cases often outperformed more complex ones in predicting cases of WNV neuroinvasive disease, suggesting minimal gains from additional factors. ...
Mathematical models within the Ross–Macdonald framework increasingly play a role in our understanding of vector-borne disease dynamics and as tools for assessing scenarios to respond to emerging threats. These threats are typically characterized by a high degree of heterogeneity, introducing a range of possible complexities in models and challenges to maintain the link with empirical evidence. We systematically identified and analysed a total of 77 published papers presenting compartmental West Nile virus (WNV) models that use parameter values derived from empirical studies. Using a set of 15 criteria, we measured the dissimilarity compared with the Ross–Macdonald framework. We also retrieved the purpose and type of models and traced the empirical sources of their parameters. Our review highlights the increasing refinements in WNV models. Models for prediction included the highest number of refinements. We found uneven distributions of refinements and of evidence for parameter values. We identified several challenges in parametrizing such increasingly complex models. For parameters common to most models, we also synthesize the empirical evidence for their values and ranges. The study highlights the potential to improve the quality of WNV models and their applicability for policy by establishing closer collaboration between mathematical modelling and empirical work.
... Reservoir water temperature forecasts generated using a MME consisting of process and baseline models performed better overall than using individual models or a PM MME. Our results support previous research that shows that MME methods often outperform individual models (Atiya, 2020;Johansson et al., 2019;Viboud et al., 2018). For example, in a large diverse forecasting competition of multiple finance and demography variables, 70% of the most accurate forecasts were MMEs (Atiya, 2020). ...
... While many forecasting studies use baseline models as null models to evaluate forecasts, here we show their value as a component of the forecast themselves. These baseline models, despite their simplicity, provide additional forecast information that the complex PMs do not, and highlight that model complexity does not necessarily translate to forecast skill (Viboud et al., 2018;Ward et al., 2014). Even simple models, lacking any domain expertise, can provide useful information to an MME . ...
Water temperature forecasting in lakes and reservoirs is a valuable tool to manage crucial freshwater resources in a changing and more variable climate, but previous efforts have yet to identify an optimal modeling approach. Here, we demonstrate the first multi‐model ensemble (MME) reservoir water temperature forecast, a forecasting method that combines individual model strengths in a single forecasting framework. We developed two MMEs: a three‐model process‐based MME and a five‐model MME that includes process‐based and empirical models to forecast water temperature profiles at a temperate drinking water reservoir. We found that the five‐model MME improved forecast performance by 8%–30% relative to individual models and the process‐based MME, as quantified using an aggregated probabilistic skill score. This increase in performance was due to large improvements in forecast bias in the five‐model MME, despite increases in forecast uncertainty. High correlation among the process‐based models resulted in little improvement in forecast performance in the process‐based MME relative to the individual process‐based models. The utility of MMEs is highlighted by two results: (a) no individual model performed best at every depth and horizon (days in the future), and (b) MMEs avoided poor performances by rarely producing the worst forecast for any single forecasted period (<6% of the worst ranked forecasts over time). This work presents an example of how existing models can be combined to improve water temperature forecasting in lakes and reservoirs and discusses the value of utilizing MMEs, rather than individual models, in operational forecasts.
... Ensemble models that combine the strength of multiple models may exhibit significantly enhanced predictive performance (e.g. (Bleichrodt et al., 2023;Chowell et al., 2020;Chowell & Luo, 2021;Ray & Reich, 2018;Viboud et al., 2018)). An ensemble model derived from the top-ranking K models are denoted by Ensemble(K) and illustrated in Fig. 3. Thus, Ensemble(2) and Ensemble(3) refer to the ensemble models generated from the combination of the top-ranking 2 and 3 models, respectively. ...
An ensemble n-sub-epidemic modeling framework that integrates sub-epidemics to capture complex temporal dynamics has demonstrated powerful forecasting capability in previous works. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. In this tutorial paper, we introduce and illustrate SubEpiPredict, a user-friendly MATLAB toolbox for fitting and forecasting time series data using an ensemble n-sub-epidemic modeling framework. The toolbox can be used for model fitting, forecasting, and evaluation of model performance of the calibration and forecasting periods using metrics such as the weighted interval score (WIS). We also provide a detailed description of these methods including the concept of the n-sub-epidemic model, constructing ensemble forecasts from the top-ranking models, etc. For the illustration of the toolbox, we utilize publicly available daily COVID-19 death data at the national level for the United States. The MATLAB toolbox introduced in this paper can be very useful for a wider group of audiences, including policymakers, and can be easily utilized by those without extensive coding and modeling backgrounds.
... By considering past glucose readings, lifestyle factors, and other relevant information, the LSTM model provided accurate and timely predictions, assisting [7] Ensemble Learning in Infectious Diseases: Ensemble learning, a technique that combines the predictions of multiple models, has shown significant success in predicting and managing infectious diseases. The ability of ensemble models to integrate diverse perspectives and mitigate individual model biases is crucial in the dynamic landscape of infectious disease outbreaks.An influential application is observed in the work of Viboud et al. (2016), where researchers employed ensemble learning to predict the spread of infectious diseases. By combining the outputs of various models, the ensemble approach improved the accuracy of predictions and enhanced the understanding of the complex dynamics of infectious disease transmission. ...
... The adaptability and robustness of ensemble models make them valuable tools for anticipating and responding to infectious disease challenges. [8] Graph Neural Networks (GNNs) in Genetic Disorders: Graph Neural Networks (GNNs) have gained prominence in genetic research, particularly in the analysis of genetic interactions and identifying potential markers for genetic disorders. GNNs excel in capturing complex relationships in biological networks, providing insights into the intricate mechanisms underlying genetic conditions.An exemplar study by Gilvary et al. (2021) showcases the application of GNNs in predicting genetic markers associated with a specific disorder. ...
The intersection of artificial intelligence (AI) technologies with the realm of healthcare has initiated groundbreaking
progress, marking the onset of an unprecedented era in the pursuit of enhanced well-being. This section serves as the
preamble to the overarching theme of the research paper, underscoring the paramount significance of AI in the
transformative overhaul of disease-related mechanisms. In the ensuing discourse, we delve into the intricate symbiosis
between AI methodologies and healthcare, illuminating the novel perspectives, methodologies, and outcomes that define
this paradigm shift. This inquiry positions AI as the linchpin in an intricate web of scientific inquiry and technological
innovation, propelling the trajectory toward a redefined landscape in the relentless pursuit of a healthier future.
... In contrast, dynamical models based on rates of change equations (i.e., differential equations) are less frequently applied by non-specialists in specific scientific fields. However, forecasts based on dynamic models can provide more information about the process of interest by characterizing specific mechanisms and parameters involved in their dynamics [6][7][8][9] . For instance, the simple phenomenological growth models discussed in this tutorial can help characterize growth rates, scaling of growth, doubling times, reproduction numbers, and turning points, as well as predicting the size of a growing population (i.e., carrying capacity) or an epidemic outbreak (i.e., epidemic size) at different time horizons with quantified uncertainty 10-17 . ...
Simple dynamic modeling tools can help generate real-time short-term forecasts with quantified uncertainty of the trajectory of diverse growth processes unfolding in nature and society, including disease outbreaks. An easy-to-use and flexible toolbox for this purpose is lacking. This tutorial-based primer introduces and illustrates GrowthPredict, a user-friendly MATLAB toolbox for fitting and forecasting time-series trajectories using phenomenological dynamic growth models based on ordinary differential equations. This toolbox is accessible to a broad audience, including students training in mathematical biology, applied statistics, and infectious disease modeling, as well as researchers and policymakers who need to conduct short-term forecasts in real-time. The models included in the toolbox capture exponential and sub-exponential growth patterns that typically follow a rising pattern followed by a decline phase, a common feature of contagion processes. Models include the 1-parameter exponential growth model and the 2-parameter generalized-growth model, which have proven useful in characterizing and forecasting the ascending phase of epidemic outbreaks. It also includes the 2-parameter Gompertz model, the 3-parameter generalized logistic-growth model, and the 3-parameter Richards model, which have demonstrated competitive performance in forecasting single peak outbreaks. We provide detailed guidance on forecasting time-series trajectories and available software (https://github.com/gchowell/forecasting_growthmodels), including the full uncertainty distribution derived through parametric bootstrapping, which is needed to construct prediction intervals and evaluate their accuracy. Functions are available to assess forecasting performance across different models, estimation methods, error structures in the data, and forecasting horizons. The toolbox also includes functions to quantify forecasting performance using metrics that evaluate point and distributional forecasts, including the weighted interval score. This tutorial and toolbox can be broadly applied to characterizing and forecasting time-series data using simple phenomenological growth models. As a contagion process takes off, the tools presented in this tutorial can help create forecasts to guide policy regarding implementing control strategies and assess the impact of interventions. The toolbox functionality is demonstrated through various examples, including a tutorial video, and the examples use publicly available data on the monkeypox (mpox) epidemic in the USA.
... 10 : "Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist". As mathematical models have become more central to policy and decision-making in the public health community, there has also been a growing debate on their validity and usefulness 16,17 , their general lack of transparency 18 and the scarce availability of model performance comparisons and tools to do so systematically 10 . ...
... : "Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist". As mathematical models have become more central to policy and decision-making in the public health community, there has also been a growing debate on their validity and usefulness 16,17 , their general lack of transparency 18 and the scarce availability of model performance comparisons and tools to do so systematically 10 . This could be further complicated by the growing scientific evidence and interest pointing at model ensembles as best performing strategies for predictive analytics 19,20 . ...
Many epidemiological models and algorithms are used to fit the parameters of a given epidemic curve. On many occasions, fitting algorithms are interleaved with the actual epidemic models, which yields combinations of model-parameters that are hard to compare among themselves. Here, we provide a model-agnostic framework for epidemic parameter fitting that can (fairly) compare different epidemic models without jeopardizing the quality of the fitted parameters. Briefly, we have developed a Python framework that expects a Python function (epidemic model) and epidemic data and performs parameter fitting using automatic configuration. Our framework is capable of fitting parameters for any type of epidemic model, as long as it is provided as a Python function (or even in a different programming language). Moreover, we provide the code for different types of models, as well as the implementation of 4 concrete models with data to fit them. Documentation, code and examples can be found at https://ulog.udl.cat/static/doc/epidemic-gga/html/index.html.
