Figure - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Population information for data included for risk stratification using machine learning.
Source publication
Machine learning can help clinicians to make individualized patient predictions only if researchers demonstrate models that contribute novel insights, rather than learning the most likely next step in a set of actions a clinician will take. We trained deep learning models using only clinician-initiated, administrative data for 42.9 million admissio...
Context in source publication
Context 1
... Premier Healthcare Database (PHD) 15 is a large-scale, provider-based, all-payer database containing data on more than 215 million total patients and 115 million inpatient admissions. It includes more than six million inpatient admissions each year between 2013 and 2018 and a total of over 35 million admissions more than one day between 2013 and 2017 (training) and 6.7 million admissions lasting longer than one day in 2018 (test) ( Table 2). The PHD has been certified as de-identified via expert determination in compliance with HIPAA. ...Similar publications
Gastric endoscopic resection (ER) is widely performed in Korea. This study aimed to investigate the overall status of gastric ER in Korea. We enrolled ESD or EMR cases performed for gastric cancer and adenoma from 2012 to 2017 by searching the NHIS database. The annual trend of gastric ER and the clinical characteristics were investigated. Institut...
Citations
... Previous studies have shown that artificial intelligence tools using ML algorithms can improve treatment, enhance quality of care and patient safety, reduce burden on providers, and generally increase the efficiency with which resources are used, resulting in potential cost savings or health gains [7,32,[35][36][37][38]. In addition, our findings align with those of previous studies that highlight the potential of ML applications to predict individual patients' risk of specific medical conditions and associated complications to offer specialized care programs to high-risk patients [39,40]. Our study also confirms and extends the findings of a few studies that examined other ML applications and highlighted the potential to identify patients at risk for substance misuse and abuse, including OUD and opioid overdose [31,38,41]. ...
Background
Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment.
Objective
This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians.
Methods
The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient’s OUD risk level, which was then compared to the ML application’s risk assignments.
Results
A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified.
Conclusions
A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients’ different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.
... These actions, in turn, influence the patient's resulting state, and the cycle repeats iteratively. 36 To summarize, the concept of integrating 'clinician-initiated' and 'non-clinician-initiated' data is essential for effective implementation of AI tools for Impella tMCs. 36 To date, AI models potentially able to manage the complexity typical of the real clinical world have not yet been validated in the healthcare scenario, and further relevant technological advancements (generative AI) are needed before such a model will be made available at bedside. ...
... 36 To summarize, the concept of integrating 'clinician-initiated' and 'non-clinician-initiated' data is essential for effective implementation of AI tools for Impella tMCs. 36 To date, AI models potentially able to manage the complexity typical of the real clinical world have not yet been validated in the healthcare scenario, and further relevant technological advancements (generative AI) are needed before such a model will be made available at bedside. ...
Aims
Artificial intelligence (AI) has emerged as a potential useful tool to support clinical treatment of heart failure, including the setting of mechanical circulatory support (MCS). Modern Impella pumps are equipped with advanced technology (SmartAssist), enabling real‐time acquisition and display of data related to both pump performance and the patient's haemodynamic status. These data emerge as an ‘ideal’ source for data‐driven AI applications to predict the clinical course of an ongoing therapeutic protocol. Yet, no evidence of effective application of AI tools in the setting of Impella support is available. On this background, we aimed at identifying possible future applications of AI‐based tools in the setting of temporary MCS with an Impella device.
Methods
We explored the state of research and development at the intersection of AI and Impella support and derived future potential applications of AI in routine Impella clinical management.
Results
We identified different areas where the future implementation of AI tools may contribute to addressing important clinical challenges in the setting of Impella support, including (i) early identification of the best suited pathway of care according to patients' conditions at presentation and intention to treat, (ii) prediction of therapy outcomes according to different possible therapeutic actions, (iii) optimization of device implantation procedures and evaluation of proper pump position over the whole course of support and (iv) prevention and/or rationale management of haemocompatibility‐related adverse events. For each of those areas, we discuss the potential advantages, challenges and implications of harnessing AI‐driven insights in the setting of MCS with an Impella device.
Conclusions
Temporary MCS with an Impella device has great potential to benefit from the integration of AI‐based tools. Such tools may indeed translate into groundbreaking innovation supporting clinical decision‐making and therapy regulation, in particular in complex scenarios such as the multidevice MCS strategy.
... This approach means that all variables of interest might not be included, and the degree to which decision thresholds are based on physician behaviour rather than faithful representations of patient physiology is not clear. 43 Additionally, no dataset is perfect. Therefore, to maximise algorithm performance, initial applications should involve well-curated data, followed by validation on realworld data to assess whether the variables used are likely to improve or negatively affect decision-making. ...
Objectives
Assess understanding of impactibility modelling definitions, benefits, challenges and approaches.
Design
Qualitative assessment.
Setting
Two workshops were developed. Workshop 1 was to consider impactibility definitions and terminology through moderated open discussion, what the potential pros and cons might be, and what factors would be best to assess. In workshop 2, participants appraised five approaches to impactibility modelling identified in the literature.
Participants
National Health Service (NHS) analysts, policy-makers, academics and members of non-governmental think tank organisations identified through existing networks and via a general announcement on social media. Interested participants could enrol after signing informed consent.
Outcome measures
Descriptive assessment of responses to gain understanding of the concept of impactibility (defining impactibility analysis), the benefits and challenges of using this type of modelling and most relevant approach to building an impactibility model for the NHS.
Results
37 people attended 1 or 2 workshops in small groups (maximum 10 participants): 21 attended both workshops, 6 only workshop 1 and 10 only workshop 2. Discussions in workshop 1 illustrated that impactibility modelling is not clearly understood, with it generally being viewed as a cross-sectional way to identify patients rather than considering patients by iterative follow-up. Recurrent factors arising from workshop 2 were the shortage of benchmarks; incomplete access to/recording of primary care data and social factors (which were seen as important to understanding amenability to treatment); the need for outcome/action suggestions as well as providing the data and the risk of increasing healthcare inequality.
Conclusions
Understanding of impactibility modelling was poor among our workshop attendees, but it is an emerging concept for which few studies have been published. Implementation would require formal planning and training and should be performed by groups with expertise in the procurement and handling of the most relevant health-related real-world data.
... Notable examples include its pivotal role in the analysis of medical images, 27 its proficiency in detecting drug interactions, 28 and its ability to identify high-risk patients to enable early intervention. 29 However, the current limitations of ChatGPT, including the possibility for hallucinations, its lack of citations, and outstanding legal considerations, 30 may preclude its widespread adoption into clinical practice as a clinician-recommended education tool. Although some of these limitations are unanimous among generative AI chatbots today, given the speed at which the technology is evolving, it is highly plausible that chatbots may achieve the consistency and reliability necessary to be deemed clinicianrecommended education tools in the near future. ...
Background
Although demonstrating remarkable promise in other fields, the impact of artificial intelligence (including ChatGPT in hand surgery and medical practice) remains largely undetermined. In this study, we asked ChatGPT frequently asked patient-focused questions surgeons may receive in clinic from patients who have carpel tunnel syndrome (CTS) and evaluated the quality of its output.
Methods
Using ChatGPT, we asked 10 frequently asked questions that hand surgeons may receive in the clinic before carpel tunnel release (CTR) surgery. Included questions were generated from the authors’ own experiences regarding conservative and operative treatment of CTS.
Results
Responses from the following 10 questions were included: (1) What is CTS and what are its signs and symptoms? (2) What are the nonsurgical options for CTS? (3) Should I get surgery for CTS? (4) What is a CTR and how is it preformed? (5) What are the differences between open and endoscopic CTR? (6) What are the risks associated with CTR and how frequently do they occur? (7) Does CTR cure CTS? (8) How much improvement in my symptoms can I expect after CTR? (9) How long is the recovery after CTR? (10) Can CTS recur after surgery?
Conclusions
Overall, the chatbot provided accurate and comprehensive information in response to most common and nuanced questions regarding CTS and CTR surgery, all in a way that would be easily understood by many patients. Importantly, the chatbot did not provide patient-specific advice and consistently advocated for consultation with a healthcare provider.
... Clinicians usually follow thought processes such as "the probability of AKI onset increases when additional risk factors such as infections and diuretics (triggers) are added to the background of cancer cachexia (underlying risks)." When interpreting the combination of underlying clinical backgrounds and additional stratified risks that lead to AKI development, analyzing the individual AI models' predictive reasoning can be a valuable approach to explore the most critical AKI risks, which are challenging to understand using routine medical data [35]. In the future, this approach will help effectively determine the appropriate assessment and intervention for patients with complicated AKI risks (S5 Fig in S1 File) [36]. ...
Background
Acute kidney injury (AKI) is a critical complication of immune checkpoint inhibitor therapy. Since the etiology of AKI in patients undergoing cancer therapy varies, clarifying underlying causes in individual cases is critical for optimal cancer treatment. Although it is essential to individually analyze immune checkpoint inhibitor-treated patients for underlying pathologies for each AKI episode, these analyses have not been realized. Herein, we aimed to individually clarify the underlying causes of AKI in immune checkpoint inhibitor-treated patients using a new clustering approach with Shapley Additive exPlanations (SHAP).
Methods
We developed a gradient-boosting decision tree-based machine learning model continuously predicting AKI within 7 days, using the medical records of 616 immune checkpoint inhibitor-treated patients. The temporal changes in individual predictive reasoning in AKI prediction models represented the key features contributing to each AKI prediction and clustered AKI patients based on the features with high predictive contribution quantified in time series by SHAP. We searched for common clinical backgrounds of AKI patients in each cluster, compared with annotation by three nephrologists.
Results
One hundred and twelve patients (18.2%) had at least one AKI episode. They were clustered per the key feature, and their SHAP value patterns, and the nephrologists assessed the clusters’ clinical relevance. Receiver operating characteristic analysis revealed that the area under the curve was 0.880. Patients with AKI were categorized into four clusters with significant prognostic differences (p = 0.010). The leading causes of AKI for each cluster, such as hypovolemia, drug-related, and cancer cachexia, were all clinically interpretable, which conventional approaches cannot obtain.
Conclusion
Our results suggest that the clustering method of individual predictive reasoning in machine learning models can be applied to infer clinically critical factors for developing each episode of AKI among patients with multiple AKI risk factors, such as immune checkpoint inhibitor-treated patients.
... Artificial intelligence (AI) has been introduced to healthcare with the promise of assisting or automating tasks to reduce human workload. In publications, medical AI models have been reported to produce promising results in a variety of data-driven scenarios, including clinical decision support, medical image interpretation and risk prediction [1][2][3] . However, real-world implementation of medical AI interventions has so far been limited and the potential benefits not yet realised. ...
The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.
... For instance, AI can now reliably identify diabetic retinopathy without requiring an ophthalmologist to verify the algorithm's diagnosis [1c]. The uses of artificial intelligence (AI) in medicine have been growing in many areas, including in the analysis of medical images [2], the detection of drug interactions [3], the identification of high risk patients [4], and the coding of medical notes [5]. ...
Artificial intelligence (AI) is an established branch of computer sciences that utilizes computer algorithms to solve problems that otherwise is done by humans. It is there in our daily routines be it in form of search engines like Google or to home assistants Alexa and, nowadays, OpenAI with its chatbot. AI has the potential to revolutionize the diagnosis, prediction, and treatment of glomerular disease by leveraging its ability to analyze large datasets and identify patterns. AI utility in the field of Nephrology is immense, particularly in the areas of diagnosis, treatment, and prediction of various kidney diseases as well as its ability to improve diagnostic accuracy. The reason of early and precise identification of acute kidney injury (AKI) is of paramount importance for the ascertaining the overall morbidity and mortality of a patient. Instead of the human brain the machine learning algorithms can help to identify early signs of kidney disease by recognizing patterns in patient demographic data, lab results, imaging, and medical history, and hence allow timely diagnosis and prompt initiation of treatment plans that ultimately improve patient outcome. AI holds the promise of advancing personalized medicine to new levels. AI will augment in decision making and should best be labelled as "Augmented Intelligence" once it comes to its role in medicine. It is mandatory to train nephrologists in the fundamentals of AI because time has come to shift from the traditional practice of decision-making of kidney diseases to AI based tools to quickly analyse patients' information and come to a quick decision.
... In health machine learning (ML) models, clinician-initiated and non-clinician-initiated data play distinct roles [5]. Clinician-initiated data, although helpful, might lead models to mimic existing clinical decisions and reduce the efficacy of risk stratification models, which aim to identify patients at risk before clinical recognition. ...
Acute Myeloid Leukemia (AML) is a complex disease requiring accurate risk stratification for effective treatment planning. This study introduces an innovative ensemble machine learning model integrated with the European LeukemiaNet (ELN) 2022 recommendations to enhance AML risk stratification. The model demonstrated superior performance by utilizing a comprehensive dataset of 1,213 patients from National Taiwan University Hospital (NTUH) and an external cohort of 2,113 patients from UK-NCRI trials. On the external cohort, it improved a concordance index (c-index) from 0.61 to 0.64 and effectively distinguished three different risk levels with median hazard ratios ranging from 18% to 50% improved. Key insights were gained from the discovered significant features influencing risk prediction, including age, genetic mutations, and hematological parameters. Notably, the model identified specific cytogenetic and molecular alterations like TP53 , IDH2 , SRSF2 , STAG2 , KIT , TET2 , and karyotype (-5, -7, -15, inv(16)), alongside age and platelet counts. Additionally, the study explored variations in the effectiveness of hematopoietic stem cell transplantation (HSCT) across different risk levels, offering new perspectives on treatment effects. In summary, this study develops an ensemble model based on the NTUH cohort to deliver improved performance in AML risk stratification, showcasing the potential of integrating machine learning techniques with medical guidelines to enhance patient care and personalized medicine.
... The specific items and their correlations are shown in SDC, Table 3 The secondary outcome included prolonged length of stay (PLOS), which was defined as a length of stay greater than that of the 90th percentile [22] of patients in the same surgical specialties at the same hospital. Patients whose hospitalization ended in mortality with a length of stay less than the 90th percentile were excluded from cohorts for which PLOS was predicted [23] . ...
Background
When they encounter various highly related postoperative complications, existing risk evaluation tools that focus on single or any complications are inadequate in clinical practice. This seriously hinders complication management because of the lack of a quantitative basis. An interpretable multi-label model framework that predicts multiple complications simultaneously is urgently needed.
Materials and Methods
We included 50,325 inpatients from a large multicenter cohort (2014–2017). We separated patients from one hospital for external validation and randomly split the remaining patients into training and internal validation sets. A MARKov-EmbeDded (MARKED) multi-label model was proposed, and three models were trained for comparison: binary relevance (BR), a fully connected network (FULLNET), and a deep neural network (DNN). Performance was mainly evaluated using the area under the receiver operating characteristic curve (AUC). We interpreted the model using Shapley Additive Explanations. Complication-specific risk and risk source inference were provided at the individual level.
Results
There were 26,292, 6574, and 17,459 inpatients in the training, internal validation, and external validation sets, respectively. For the external validation set, MARKED achieved the highest average AUC (0.818, 95% confidence interval: 0.771–0.864) across eight outcomes (compared with BR, 0.799 [0.748–0.849], FULLNET, 0.806 [0.756–0.856], and DNN, 0.815 [0.765–0.866]). Specifically, the AUCs of MARKED were above 0.9 for cardiac complications (0.927 [0.894–0.960]), neurological complications (0.905 [0.870–0.941]), and mortality (0.902 [0.867–0.937]). Serum albumin, surgical specialties, emergency case, American Society of Anesthesiologists score, age, and sex were the six most important preoperative variables. The interaction between complications contributed more than the preoperative variables, and formed a hierarchical chain of risk factors, mild complications, and severe complications.
Conclusion
We demonstrated the advantage of MARKED in terms of performance and interpretability. We expect that the identification of high-risk patients and inference of the risk source for specific complications will be valuable for clinical decision-making.
... ML can process high-complexity clinical information and use the acquired knowledge to diagnose, manage, and predict disease outcomes [19]. Patient risk stratification is one of the most wide potential applications of ML techniques [20]. Several studies have already developed ML-based risk assessment models for patients with sepsis in the ICU settings [21][22][23], such as Extreme Gradient Boosting (XGBoost), stepwise logistic regression, stochastic gradient boosting, and random forest (RF). ...
Purpose
This study aimed to construct a mortality model for the risk stratification of intensive care unit (ICU) patients with sepsis by applying a machine learning algorithm.
Methods
Adult patients who were diagnosed with sepsis during admission to ICU were extracted from MIMIC-III, MIMIC-IV, eICU, and Zigong databases. MIMIC-III was used for model development and internal validation. The other three databases were used for external validation. Our proposed model was developed based on the Extreme Gradient Boosting (XGBoost) algorithm. The generalizability, discrimination, and validation of our model were evaluated. The Shapley Additive Explanation values were used to interpret our model and analyze the contribution of individual features.
Results
A total of 16,741, 15,532, 22,617, and 1,198 sepsis patients were extracted from the MIMIC-III, MIMIC-IV, eICU, and Zigong databases, respectively. The proposed model had an area under the receiver operating characteristic curve (AUROC) of 0.84 in the internal validation, which outperformed all the traditional scoring systems. In the external validations, the AUROC was 0.87 in the MIMIC-IV database, better than all the traditional scoring systems; the AUROC was 0.83 in the eICU database, higher than the Simplified Acute Physiology Score III and Sequential Organ Failure Assessment (SOFA),equal to 0.83 of the Acute Physiology and Chronic Health Evaluation IV (APACHE-IV), and the AUROC was 0.68 in the Zigong database, higher than those from the systemic inflammatory response syndrome and SOFA. Furthermore, the proposed model showed the best discriminatory and calibrated capabilities and had the best net benefit in each validation.
Conclusions
The proposed algorithm based on XGBoost and SHAP-value feature selection had high performance in predicting the mortality of sepsis patients within 24 h of ICU admission.
