Figure - available from: Journal of Clinical Monitoring and Computing
This content is subject to copyright. Terms and conditions apply.
(Top) Visual representation of the CoMET display as it appeared on a large screen monitor in the surgical ICU. Units on the Respiratory and Cardiovascular instability axes represent the fold-increase in risk compared to average for emergency intubation and hemorrhage, respectively. Each bed is represented by a comet with the bed number in the head. The head of the comet represents the risks at the current time, and the tail is 3 h long. (Bottom) a patient in bed 86 has been selected for individual inspection. This patient’s respiratory (green) and cardiovascular (orange) instability over the preceding 24 h is displayed in the bottom right. The respiratory instability rose from average risk (onefold) to fivefold average risk over the 10 h prior to emergency intubation at 19:35. Similarly, cardiovascular instability fluctuated between average and twofold average; a blood culture for suspicion of infection was ordered at 18:22. These data were taken from recorded real-time monitoring performed during this patient’s hospital stay, prior to the period that CoMET was displayed in the SICU
Source publication
Predictive analytics monitoring, the use of patient data to provide continuous risk estimation of deterioration, is a promising new application of big data analytical techniques to the care of individual patients. We tested the hypothesis that continuous display of novel electronic risk visualization of respiratory and cardiovascular events would i...
Similar publications
Purpose of Review
The aim of this review is to analyze the cardiovascular pathophysiology of septic shock. Using visual representations of a left ventricular cycle in the pressure/volume plane, we describe hemodynamic derangement occurring in septic shock and subsequent changes at each step of treatment allowing a rapid understanding of complex alt...
Vasoactive and inotropic medications are essential for sepsis management; however, the association between the maximum vasoactive–inotropic score (VISmax) and clinical outcomes is unknown in adult patients with sepsis. We investigated the VISmax as a predictor for mortality among such patients in the emergency department (ED) and compared its progn...
Citations
... The models were trained separately on cardiorespiratory and cardiovascular events of clinical deterioration leading to escalation in care delivery. (Moss et al 2017, Ruminski et al 2019, Keim-Malpass et al 2022, Blackwell et al 2020. The predictors of the CoMET score include: (1) cardiorespiratory dynamics measured from continuous electrocardiogram (ECG) (including heart rate variability, and pairwise cross-correlations between heart rate (HR) and ECG-derived respiratory rate (RR) local dynamics score, coefficient of sample entropy (COSEn), detrended fluctuation analysis (DFA) of heart inter-beat intervals) -all sampled every 2 seconds; (2) electronic medical record derived parameters (including: vital signs (temperature, HR, blood pressure, RR, SpO2,), oxygen flow rate, laboratory results (complete blood count, basic metabolic panel) -all sampled every 15 minutes. ...
Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool developed before COVID-19 and demonstrate model performance during the COVID-19 pandemic. The analytic system (CoMETⓇ, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10,422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns. Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit (ICU), primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.725 to 0.737. The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic.
... 11 In order to achieve this goal, the "Five Rights" of CDSSs is one framework that can be used to develop and assess CDSSs for use. 12 This framework describes five concepts that effective CDSSs should provide: (1) right information, (2) right people, (3) right format, (4) right channels, and (5) right time. 10 Clinical decision support systems for early identification of sepsis 13 and acute physiological decline 14 have successfully been reported in the critical care literature. Early mobility CDSSs may mitigate workflow barriers to implementing early mobility in the ICU. ...
Early mobility is a form of critical care rehabilitation ranging from in-bed exercise to ambulation and is known to limit post–intensive-care functional decline. Multiple barriers prevent widespread early mobility implementation. Clinical decision support systems can optimize the decision-making process and overcome barriers to care. However, critical care early mobility clinical decision support does not exist. Our objective was to develop and establish the content validation of the operational definitions for a novel critical care early mobility assessment instrument. Content validation was conducted in two steps: (1) development of operational definitions by the research team and (2) content validation with 10 nurse experts. Quality assessment and suitability of the operational definitions for 30 items, across four safety domains, were assessed using a 7 point-Likert scale. Item content validity index scores ranged from 0.6 to 0.1 after the first validation round. Domains containing items with item content validity index scores less than the lower limit of 0.78 were modified for clarity of language. Item content validity index scores ranged from 0.8 to 1.0, and the scale-level content validity index was 0.93 after the second content validation round. Excellent content validity of the operational definitions was achieved. This is the first methodological step in a larger project to develop clinical decision support for critical care early mobility.
... according to the study. Recent research has confirmed that, among the variables recorded on day one, independent correlations exist between SOFA scores and the 28-day mortality and the severity of sepsis [24,26] . Within the current study, CRP levels increasing were associated with increasing risk for mortality (p> 0.05, 0.326). ...
... While predictive analytic models have been advanced to the bedside, not many have been externally validated in a new data set, and far fewer have been implemented and used in clinical practice (Moorman 2021, Ruminski et al 2019, de Hond et al 2022 . External validation in contemporaneous data is a particularly important aspect of the vision of predictive analytics monitoring at the bedside, as we know now that data can drift (Loftus et al 2022). ...
... The models were trained separately on cardiorespiratory and cardiovascular events of clinical deterioration leading to escalation in care delivery. (Moss et al 2017, Ruminski et al 2019, Blackwell et al 2020 Randomization began on January 4, 2021 and ended October 4, 2022. We enrolled 10,422 patient visits into the randomized controlled trial. ...
Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool that was developed prior to COVID-19 and demonstrates model performance during the COVID-19 pandemic. The analytic system (CoMET®, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10,422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns. Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit (ICU), primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.732 to 0.745. The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic. We speculate that some of the model performance’s stability is due to continuous cardiorespiratory monitoring, which should not drift as practices, policies, and patient populations change.
Clinical Trial registration
ClinicalTrials.gov NCT04359641 ;
https://clinicaltrials.gov/ct2/show/NCT04359641 .
... Of the 28 single-role displays, 7 were for physicians, 34,37,41,55,64,76,87 and 15 were exclusively for bedside nurses (Tables 1 and 2). 33,35,39,47,50,56,62,63,65,69,75,89,90,93,94 Aside from the primary care teams, 13 displays also supported remote patient telemetry teams who act as the first line of deterioration surveillance. Among them, 6 displays were exclusively designed for remote support teams, 38,43,51,52,67,78,79 and 7 were designed for remote and bedside care team members. ...
... 46,53,57,60,82,85,88 Among the 61 displays, nurses were target users in 47 displays. 33,35,36,39,40,42,[44][45][46][47][48][49][50]53,54,[56][57][58][59][60][61][62][63]65,66,[68][69][70][71][72][73][74][75]77,80,81,[83][84][85][86][88][89][90][91][92][93][94][95] Information display types Of the 64 included studies, 24 studies included information display screenshots or references to accessible publications that contain screenshots. [39][40][41][42]44,46,48,51,54,56,57,59,66,68,75,[79][80][81]83,86,88,91,92,96 Thirty-six interventions had a single alert modality. ...
... 33,35,36,39,40,42,[44][45][46][47][48][49][50]53,54,[56][57][58][59][60][61][62][63]65,66,[68][69][70][71][72][73][74][75]77,80,81,[83][84][85][86][88][89][90][91][92][93][94][95] Information display types Of the 64 included studies, 24 studies included information display screenshots or references to accessible publications that contain screenshots. [39][40][41][42]44,46,48,51,54,56,57,59,66,68,75,[79][80][81]83,86,88,91,92,96 Thirty-six interventions had a single alert modality. Among them, 24 were simple alerts on the EHR screen, pager, or short message service (SMS) messages to mobile devices, 33,35,38,43,45,47,49,56,58,[64][65][66][69][70][71]76,[80][81][82]84,87,88,90,93 7 were single-patient displays, 41,44,48,68,83,92,95 and 5 were multiple-patient views. ...
Objective:
Surveillance algorithms that predict patient decompensation are increasingly integrated with clinical workflows to help identify patients at risk of in-hospital deterioration. This scoping review aimed to identify the design features of the information displays, the types of algorithm that drive the display, and the effect of these displays on process and patient outcomes.
Materials and methods:
The scoping review followed Arksey and O'Malley's framework. Five databases were searched with dates between January 1, 2009 and January 26, 2022. Inclusion criteria were: participants-clinicians in inpatient settings; concepts-intervention as deterioration information displays that leveraged automated AI algorithms; comparison as usual care or alternative displays; outcomes as clinical, workflow process, and usability outcomes; and context as simulated or real-world in-hospital settings in any country. Screening, full-text review, and data extraction were reviewed independently by 2 researchers in each step. Display categories were identified inductively through consensus.
Results:
Of 14 575 articles, 64 were included in the review, describing 61 unique displays. Forty-one displays were designed for specific deteriorations (eg, sepsis), 24 provided simple alerts (ie, text-based prompts without relevant patient data), 48 leveraged well-accepted score-based algorithms, and 47 included nurses as the target users. Only 1 out of the 10 randomized controlled trials reported a significant effect on the primary outcome.
Conclusions:
Despite significant advancements in surveillance algorithms, most information displays continue to leverage well-understood, well-accepted score-based algorithms. Users' trust, algorithmic transparency, and workflow integration are significant hurdles to adopting new algorithms into effective decision support tools.
... These data include continuous ECG, bedside monitors, vital signs, and clinical evaluation parameters. Such predictive models can detect physiological changes with illness by analyzing time series data [3][4][5][6]. A score called heart rate characteristics index (HRC index) score have been calculated to produce early warning of illness using logistic regression algorithm [7]. ...
In intensive care medicine, making fast judgments based on a large amount of information is normal. ICU doctors frequently base their judgments on their own experiences while making medical decisions. Early detection of risky levels of parameters can improve care, have long-term impacts, and minimize morbidity in neonatal intensive care. To continuously track the variables linked to patient admission and health, it is essential to provide each intensive care unit with its estimating system. To monitor and forecast patient status in the ICU, this research proposed a Markov chain model and estimated transition probabilities. The patient’s condition is divided into stable, improvement, and deterioration. It is determined by patient age, BMI, weight, central venous pressure (CVP), hourly urine output, and blood pressure. This research output will help in the early diagnosis of the patient’s condition. Significantly raising the standard of care in the ICU will be made possible by this analysis.
... Phase 2-When the sepsis alert fires-the bedside provider activates a workflow that allows them to perform a secondary clinical evaluation (SCE) to evaluate the alert in the context of the patient's clinical status. Frequently the decentralized active approach is criticized for failing because bedside nurses and providers fail to respond to alerts due to alert fatigue [26][27][28]. However, this approach only fails when the institution is relying solely on the EHR to mobilize the alert. ...
... UCD involves the development of an interface that is tailored to clinical workflows thereby maximizing efficiency. Ruminski et al. found that displaying a visual monitor significantly reduced the rate of sepsis [27]. Furthermore, studies have shown that color coding and screen positioning in the user's visual field can improve provider satisfaction and reduce sepsis rates by over 50%. ...
A sepsis deterioration index is a numerical value predicting the chance of a patient become septic by a predictive model. This model usually has pre-specified input variables that have a high likelihood of predicting the output variable of sepsis. For the purposes of predicting sepsis deterioration, we will primarily be using regression to determine the association between variables (also known as features) to eventually predict an outcome variable which in this case is sepsis. Among the cohort examined in our model at Cedars Sinai, we found patients who met or exceeded the set threshold of 68.8 had an 87% probability of deterioration to sepsis during their hospitalization with sensitivity of 39% and a median lead time of 24 hours from when the threshold was first exceeded. There is no easy way to determine an intervention point of the deterioration predictive model. The author’s recommendation is to continually modify this inflection point guided by data from near-misses and mis-categorized patients. Collecting real-time feedback from end-users on alert accuracy is also crucial for a model to survive. An ML deterioration model to predict sepsis produces ample value in a healthcare organization if deployed in conjunction with human intervention and continuous prospective re-assessment.
... Examples of multipatient integrated displays include several that have applied visualization techniques to make it easy to rapidly identify and respond to: a) which patients have the most urgent needs [70,73,79,80], b) potential problem areas with regard to hospital resource management [74][75][76], and c) gaps in compliance with patient safety or other important evidence-base practices [71,81,82]. ...
... Kohden's 'CoMET display which presents cardiovascular and respiratory deterioration risk on a two-axis plot (see Figure 14-10). The head of a 'comet' (displaying patient identity as a bed number) is plotted at the scale position of the current score and risk is coded using both size of the dot and color [79]. A fading tail conveys the prior three hours of scores making it easy to visualize both rate and direction of change across two related deterioration risk scores. ...
... Percentile data represented in gray scale provides an additional interpretation of level of risk in comparison to other patients. In a clinical trial of the CoMET display, the rate of septic shock remained constant on a control unit (medical ICU) at the same hospital, while a reduction in septic shock was demonstrated for an intervention unit using the novel display (surgical ICU) [79]. [79]). ...
A well-designed visualization can be rapidly interpreted and understood. The advantages of visual representations of quantitative data and meaningful integration of related patient information comes at the cost of much effort in identifying the right information to group together, how to present it, to whom, and in what context. Even small gains from these efforts become great when multiplied by many users making critical health care decisions. In this chapter, we draw on theoretical models of human cognition and visual perception, understanding of workflow and information needs in the context of clinical applications, and user-centered design methods to promote the design of effective information integration and visualization. We provide a grounding in how humans use vision to think which has led to the development of principles to guide the design of information integration and visualization. We present design methods, including contextual inquiry and participatory design activities, relevant to understanding clinical information integration and visualization needs. Finally, in a world of increasing information overload, we review examples of research and design efforts to identify and artfully present the most relevant patient information to support a variety of clinical decisions such as prioritizing resources to patients with the most urgent needs, detection of missed actions, and making the right treatment choices.
... The outputs that generate CoMET, which are the products of logistic regression, are instantaneous metrics 'f 'cardiorespirat'ry' a'd 'cardiovascular instabil'ty', and the output display charts the trajectory of this instability over the prior 3 hours to allow identification of patients with a deteriorating trajectory (see Fig. 1). [11][12][13][14] A potential shortcoming of this approach is that, while excellent at reflecting patient trajectory now, i.e. based on the prior several hours of data, the CoMET score is unable to reflect the cumulative burden of illness sustained throughout the days and sometimes weeks of their hospital stay. As such, the score may fail to reflect all that we have learned about a given patient and their illness to date. ...
... The models that have informed CoMET development have been described previously. 8,9,11,13,15 Comprehensive clinical data that is incorporated into the CoMET score are shown in Table 1. The CoMET score is the fold-increase in probability of an event occurring in the next 8 hours, with a score of 1 meaning that the risk of the event occurring is the average risk for that patient in that particular unit. ...
... 8 However, the CoMET score is also an instantaneous score, based on vital signs, lab values and continuous cardiorespiratory monitoring parameters that are happening now. 13 The novelty of the cCoMET score explored here is that it includes the 'now', but adds this to all of the predicted risk that has occurred from admission to now, to add potentially important information to current risk prediction. None of the digital tools compared in the paper by Mann et al. 38 included the ability to reflect all of the physiological insults sustained by the patient since admission like the cCoMET can. ...
Predictive analytics tools variably take into account data from the electronic medical record, lab tests, nursing charted vital signs and continuous cardiorespiratory monitoring data to deliver an instantaneous score that indicates patient risk or instability. Few, if any, of these tools reflect the risk to a patient accumulated over the course of an entire hospital stay. Current approaches fail to best utilize all of the cumulatively collated data regarding the risk or instability sustained by the patient. We have expanded on our instantaneous CoMET predictive analytics score to generate the cumulative CoMET score (cCoMET), which sums all of the instantaneous CoMET scores throughout a hospital admission relative to a baseline expected risk unique to that patient. We have shown that higher cCoMET scores predict mortality, but not length of stay, and that higher baseline CoMET scores predict higher cCoMET scores at discharge/death. cCoMET scores were higher in males in our cohort, and added information to the final CoMET when it came to the prediction of death. In summary, we have shown that the inclusion of all repeated measures of risk estimation performed throughout a patients hospital stay adds information to instantaneous predictive analytics, and could improve the ability of clinicians to predict deterioration, and improve patient outcomes in so doing.
... This shared version of a new characterization of patients and the trajectories of their illnesses is made possible by novel and advanced analyses of the electronic health record (EHR) and continuous cardiorespiratory monitoring. Indeed, reports of improvement in patient outcomes brought about by predictive analytics are increasing [1][2][3][4][5][6] . ...
The idea that we can detect subacute potentially catastrophic illness earlier by using statistical models trained on clinical data is now well-established. We review evidence that supports the role of continuous cardiorespiratory monitoring in these predictive analytics monitoring tools. In particular, we review how continuous ECG monitoring reflects the patient and not the clinician, is less likely to be biased, is unaffected by changes in practice patterns, captures signatures of illnesses that are interpretable by clinicians, and is an underappreciated and underutilized source of detailed information for new mathematical methods to reveal.
