Establishment and validation of predictive model of ARDS in critically ill patients

Abstract

Background

Acute respiratory distress syndrome (ARDS) is a prevalent complication among critically ill patients, constituting around 10% of intensive care unit (ICU) admissions and mortality rates ranging from 35 to 46%. Hence, early recognition and prediction of ARDS are crucial for the timely administration of targeted treatment. However, ARDS is frequently underdiagnosed or delayed, and its heterogeneity diminishes the clinical utility of ARDS biomarkers. This study aimed to observe the incidence of ARDS among high-risk patients and develop and validate an ARDS prediction model using machine learning (ML) techniques based on clinical parameters.

Methods

This prospective cohort study in China was conducted on critically ill patients to derivate and validate the prediction model. The derivation cohort, consisting of 400 patients admitted to the ICU of the Peking University Third Hospital(PUTH) between December 2020 and August 2023, was separated for training and internal validation, and an external data set of 160 patients at the FU YANG People's Hospital from August 2022 to August 2023 was employed for external validation. Least absolute shrinkage and selection operator (LASSO) and multivariate logistic regression were used to screen predictor variables. Multiple ML classification models were integrated to analyze and identify the best models. Several evaluation indexes were used to compare the model performance, including the area under the receiver-operating-characteristic curve (AUC) and decision curve analysis (DCA). SHapley Additive ex Planations (SHAP) is used to interpret ML models.

Results

400 critically ill patients were included in the analysis, with 117 developing ARDS during follow-up. The final model included gender, Lung Injury Prediction Score (LIPS), Hepatic Disease, Shock, and combined Lung Contusion. Based on the AUC and DCA in the validation group, the logistic model demonstrated excellent performance, achieving an AUC of 0.836 (95% CI: 0.762–0.910). For external validation, comprising 160 patients, 44 of whom developed ARDS, the AUC was 0.799 (95% CI: 0.723–0.875), significantly outperforming the LIPS score alone.

Conclusion

Combining the LIPS score with other clinical parameters in a logistic regression model provides a more accurate, clinically applicable, and user-friendly ARDS prediction tool than the LIPS score alone.

Background

ARDS is a syndrome characterized by acute respiratory failure resulting from various intrapulmonary or extrapulmonary factors, leading to increased permeability of the pulmonary vasculature and epithelium, pulmonary edema, gravity-dependent pulmonary atelectasis, and ultimately diffuse pulmonary inflammation and edema [1]. The LUNG SAFE study reported an incidence of ARDS in ICU patients at 10.4%, with hospital mortality rates of 34.9%, 40.3%, and 46.1% for mild, moderate, and severe ARDS [2]. A CHARDSnet study conducted in 18 ICUs across mainland China found ARDS accounted for 3.57% of total ICU admissions, with hospital mortality rates of 31.4%, 40.4%, and 56.2% for mild, moderate, and severe ARDS, respectively [3]. Current treatments primarily involve comprehensive supportive measures such as mechanical ventilation, with no drugs recognized as effective in treating ARDS, contributing to its high mortality rate. Thus, the urgent clinical need lies in identifying effective methods and approaches to improve ARDS prognosis. Early identification of high-risk ARDS patients and proactive intervention have been shown to mitigate the occurrence and progression of ARDS [4,5,6].

Gajic et al. constructed a predictor of clinical indicators of ARDS: the LIPS, which was validated in a multicenter study [7]. However, the positive predictive value (PPV) of LIPS was only 18%, thus limiting its clinical application. Other prediction models for ARDS (e.g., surgical lung injury prediction model (SLIP), emergency department lung injury prediction score (EDLIPS), sepsis ARDS prediction model, traumatic brain injury ARDS prediction model) have not been validated in clinical practice [8,9,10,11].

In this study, we observed the incidence of ARDS in high-risk patients. The LIPS score and other clinical parameters were used to develop and validate an ML-based ARDS prediction model for accurate patient identification.

Methods

Study population and diagnosis of ARDS

From December 2020 to August 2023, we prospectively enrolled 476 consecutive patients with identified risk factors for ARDS in the Department of Surgical Critical Care Medicine at the PUTH. Among these patients, 117 were ultimately diagnosed with ARDS. The diagnostic criteria for ARDS adhered to the 2012 Berlin definition [12]: new or worsening respiratory symptoms within 1 week attributed to known clinical triggers; bilateral diffuse exudates on chest imaging not entirely attributable to pleural effusion or lobar/lung collapse; respiratory failure induced by pulmonary edema not entirely attributable to cardiac failure or fluid overload; and an oxygenation index < 300.

Inclusion criteria comprised: (1) age ≥ 18 years; (2) presence of one or more risk factors for ARDS, including sepsis, shock, acute abdomen, pneumonia, pulmonary contusion, aspiration, high-risk trauma, or high-risk surgeries (such as spinal, abdominal, cardiac, and major vascular surgeries) [6]. Exclusion criteria encompassed: (1) patients with a pre-existing diagnosis of ARDS before initial assessment; (2) ICU admission duration shorter than 72 h; (3) patients deceased within 6 h of admission; (4) incomplete core information; and (5) abandonment of intensive care. Patients meeting any of these conditions were excluded from the study. Ultimately, 400 patients were included in our analysis (Fig. 1). The study protocol received approval from the Ethics Committee of the PUTH (Approval No. M2020278). Patient treatments adhered to the diagnosis and treatment protocols issued by the National Health Commission in real-time, and informed consent was obtained from patients or their families. Relevant clinical data were collected and analyzed with strict adherence to patient identity confidentiality protocols.

Fig. 1

Flowchart of screening

Data collection and processing

We utilized demographic characteristics, vital sign measurements, and laboratory data collected within the first 24 h after ICU admission to identify features and construct prediction models, as shown in Table 1. All data were extracted from the Electronic Medical Record (EMR) system. Given that nursing record system entries on the first day after ICU admission were documented from admission time until 7 am the subsequent day, we utilized the recorded data during this period for constructing the prediction model. The LIPS, Acute Physiology and Chronic Health Evaluation (APACHE) II score, Sequential Organ Failure Assessment(SOFA) score, and Neutrophil-to-Lymphocyte Ratio (NLR) were computed based on physiological parameters documented within the first 24 h of ICU admission.

Table 1 Baseline demographics, clinical characteristics, and outcomes

To address multicollinearity, a test was performed to identify variables with a variance inflation factor (VIF) above 5, indicating multicollinearity (as shown in Table S1). A total of 35 features were retained, demographic factors including age, sex, body mass index (BMI); medical history including chronic obstructive pulmonary disease (COPD), hypertension, coronary artery disease (CAD), diabetes mellitus (DM), hepatic disease, malignancy, chronic kidney disease (CKD); use of vasoactive drugs, continuous renal replacement therapy (CRRT), or massive blood transfusion; admission type and diagnoses including shock, trauma, brain injury, sepsis, acute abdomen, pulmonary contusion, pneumonia, aspiration, orthopedic spinal, spinal cord injury (SCI); and clinical parameters and evaluations such as 24-h fluid balance upon admission (calculated as fluid intake minus fluid output), NLR, hemoglobin (HGB), red blood cell distribution width (RDW_CV), lactate, international normalized ratio (INR), urea, albumin (ALB), APACHE II scores, LIPS scores, and SOFA scores (Table 1).

External validation cohort

An external dataset comprising 160 patients from the ICU of FU YANG People’s Hospital, collected between August 2022 and August 2023, was incorporated to validate the derivation cohort. The inclusion and exclusion criteria mirrored those applied to the training cohort.

Model development and comparison

First, R software (glmnet4.1.2) was used to conduct the least absolute shrinkage and selection operator (LASSO) regression analysis and adjust the variable screening and complexity. Subsequently, the outcomes of the LASSO regression analyses were utilized to conduct multifactor logistic regression analyses, ultimately identifying characteristic factors with P-values < 0.05. The Python software (version 0.22.1) was then employed to randomly partition the data from the PUTH into 70% for training and 30% for validation to mitigate potential overfitting issues. Additionally, an external dataset was employed for testing purposes (external validation).

The final screened features were used to develop predictive models. Five ML models namely K Nearest Neighbours (KNN), Logistic Regression (LR), Random Forest (RF), Support Vector Machine (SVM), and eXtreme Gradient Boosting (XGboost) were used for predicting the occurrence of ARDS in critically ill patients. To optimize the prediction model, grid search combined with manual fine-tuning was applied to obtain the final hyperparameters.

Several commonly used evaluation indexes, such as AUC, sensitivity, specificity, PPV, negative predictive value(NPV), accuracy, and F1 score, were used to evaluate the reliability of these models. Furthermore, a DCA was carried out to evaluate the utility of the decision models by quantifying the net benefit across different threshold probabilities.

Model explanation

The SHapley Additive exPlanations (SHAP) method offered global and local explanations for the model explanation. The global explanation could give consistent and accurate attribution values for each feature within a model to show the associations between input features and ARDS. The local explanation could demonstrate a specific prediction for individual patients by inputting the specific data.

Webpage deployment tool based on shiny

To enhance the clinical utility of the model, a web-based interactive dynamic column chart application was developed using Shiny (version 0.13.2.26). This application facilitates the prediction of the probability of ARDS by inputting the values of the corresponding features from the final model.

Statistical analysis

The data were analyzed using Python version 3.6.5 (https://www.python.org) and SPSS version 27.0 (https://www.ibm.com/spss). Categorical data were presented as frequencies (percentages) and compared using chi-square tests. Continuous variables were presented as median (interquartile range) and compared using the Kruskal–Wallis test. The predictive power was evaluated using the AUC, with the optimal cut-off value determined by maximizing the Youden index (sensitivity + specificity − 1). DCA and Precision-Recall (P-R) curve analysis was conducted using R version 4.1.2 (https://www.r-project.org). A significance level of P < 0.05 was considered statistically significant.

Results

Patient characteristics

The study utilized 400 patients from the PUTH as the derivation cohort, with 117 patients diagnosed with ARDS, yielding an incidence rate of 29.3% during hospitalization. The derivation cohort was randomly divided into two subsets using the random number method: 70% of patients comprised the training cohort, while the remaining 30% constituted the internal validation cohort for constructing and screening the best predictive model. Additionally, an external cohort comprising 160 patients from FU YANG People's Hospital was employed for external validation. Details of the study design are displayed in Fig. 1

As shown in Table 1, among the 400 patients in the derivation cohort, a notably higher proportion of patients in the ARDS group compared to the non-ARDS group were male. Significant differences between the ARDS and non-ARDS groups were observed in terms of gender, fluid balance, liver function abnormality, and urea levels (P < 0.05). Furthermore, within the initial 24 h of admission to the ICU, patients in the ARDS group exhibited significantly higher LIPS scores, APACHE II scores, and SOFA scores (P < 0.01). Variables were compared between the derivation cohort and the external validation cohort (as shown in Table S2). Patients in the external validation cohort were older than those in the derivation cohort. Patients in the external validation cohort had higher LIPS scores, SOFA scores, APACHE II scores, and higher NLR and RDW_CV compared with the derivation cohort.

Factor selection for the predictive model

LASSO regression analysis was conducted on the remaining independent variables, with ARDS serving as the dependent variable (Fig. 2). LASSO can compress variable coefficients to prevent overfitting and solve severe collinearity problems [13]. The results showed (lambda = 0.013 for minimum mean square error) that 35 independent variables were reduced to 21, including gender, admission type, MBT, Fluid balance, RDW_CV, ALB, APACHE II, LIPS, SOFA, shock, Trauma, Brain Injury, Lung contusion, Pneumonia, Lung contusion, Aspiration, SCI, Hypertensive, CAD, DM, HepaticDisease, and Tumor. To further control for the effects of confounders, multiple logistic regression was used to analyze the above 21 independent variables were analyzed [14]. Finally, only Gender, LIPS, shock, lung contusion, and Hepatic disease were identified as characteristic factors (P < 0.05) (as Table 2).

Fig. 2

Results of the LASSO regression analysis. A Plot of the LASSO coefficient profiles. B Tuning parameter (λ) selection cross-validation error curve

Table 2 Multivariate logistic regression analysis

Comprehensive analysis of classified multi-model

XGBoost, LR, RF, SVM, and KNN were trained and repeated 10 times. The models were evaluated using AUC values [15], which showed that XGBoost and RF were highest in the training set, and LR was highest in the set of internal validation sets (Fig. 3a–c). The AUC indicator focuses on the predictive accuracy of the model and does not tell whether the model is clinically usable or which one of the two is preferable [16]. Therefore, the DCA, calibration curves, and PR curves were analyzed. The DCA assessment revealed that the Logistic Regression model exhibited better clinical applicability (Fig. 3d). Calibration curves demonstrated higher accuracy in the predictions of the Logistic model (Fig. 3e). Moreover, in the validation set, the Logistic model demonstrated higher Average Precision (AP) values (Fig. 3f). Taken together, these findings suggest that the Logistic Regression model was the most desirable model for this study (Details are displayed in Tables S3 and S4).

Fig. 3

ML model comprehensive analysis. A Training sets ROC and AUC B Validation sets ROC and AUC. Patients were sampled 10 times in a 7:3 ratio. C The validation set AUC forest plot D Calibration curves for the validation set, with the horizontal coordinates being the average predicted probability, the case coordinates being the actual probability of the event, the dashed diagonal being the reference line, and the rest of the solid lines representing the different models. E The validation set DCA where the black dotted line is the assumption that all patients are treated and the red represents the assumption that all patients are not treated. The remaining solid lines represent different models. F Validation set PR curve and AP The y-axis is precision and the x-axis is recall. If the PR curve of one model is completely covered by the PR curve of another model, it can be concluded that the latter is better than the former, and the higher the AP value, the better the model performance. The different colors in the picture represent the corresponding model, and the values are represented by the average and 95% CI

The best model building and external evaluation

Logistic regression analysis and tenfold cross-validation were performed on the training set. The results show an average AUC of 0.824 (0.779–0.869) for the training set, 0.808 (0.662–0.951) for the internal validation set, and 0.799 (0.723–0.875) for the external validation set (Fig. 4a–c) (see Table S5 S6and S7 for more details). Given that the performance of the validation set under the AUC metric did not surpass that of the test set, or the exceedance ratio was less than 10%, the fit can be considered successful [16]. This demonstrates that the model exhibited strong performance in both internal and external validation. Additionally, the DCA assessment model demonstrated good clinical applicability in the external test set (Fig. 4d).

Fig. 4

Logistic regression model training, validation, and testing. A Training set ROC and AUC and B Validation set ROC and AUC. Training and cross-validation of patients in the derivation cohort. Different colored solid lines represent 10 different outcomes. C Test set ROC and AUC. test results from patients of the external test cohort. D Test set clinical decision curves, where the black dashed line represents the hypothesis that all patients have ARDS and the red represents the hypothesis that no patients have ARDS. The solid line represents the logistic model

Model explanation

To visually elucidate the selected variables, SHAP was utilized to illustrate how these variables predict the occurrence of ARDS in the model [17]. Figure 5a depicts the five most important features in our model. Each feature significance line showcases all patient attributions for the outcome, represented by differently colored dots: red dots indicate high-risk values, while blue dots denote low-risk values. Elevated LIPS score, male gender, shock, pulmonary contusion, and combined liver disease were found to increase the risk of ARDS in high-risk patients. Figure 5b displays the ranking of the five risk factors assessed by mean absolute SHAP value, with the x-axis SHAP value indicating the importance of the predictive model. Additionally, two typical examples are provided to illustrate the interpretability of the model. One example pertains to patients who developed ARDS with a higher SHAP prediction score (0.82) (Fig. 5c), while the other example involves patients who did not develop ARDS, exhibiting a lower SHAP prediction score (0.24) (Fig. 5d).

Fig. 5

SHAP interprets the mode. A Attributes of characteristics in SHAP. Each line represents a feature and the abscissa is the SHAP value. Red dots represent higher eigenvalues and blue dots represent lower eigenvalues. B Feature importance ranking as indicated by SHAP. The matrix diagram describes the importance of each covariate in the development of the final predictive model. C Individual efforts by patients with ARDS and D without ARDS. The SHAP value represents the predicted characteristics of an individual patient and the contribution of each characteristic to the predicted ARDS. The number in bold is the probability forecast value (f(x)), while the base value is the predicted value without providing input to the model. F(x) is the logarithmic ratio of each observation. Red features indicate an increased risk of ARDS and blue features indicate a reduced risk of ARDS. The length of the arrows helps visualize the extent to which the prediction is affected. The longer the arrow, the greater the effect

Convenient application for clinical utility

The final prediction model was integrated into a web application to enhance its utility in clinical settings, as depicted in Fig. 6. This application seamlessly predicts a patient's risk of developing ARDS upon entering the actual values of the five features required by the model. The web application is accessible online at https://prediction-model-for-ards.shinyapps.io/Prediction_ards/

Fig. 6

Online dynamic nomogram. An illustrative example is described for predicting the probability of ARDS after ICU admission in a woman presenting in shock with comorbid liver disease with a LIPS score of 10

Discussion

The findings of this study revealed that among ICU patients with high-risk factors for ARDS, the incidence of ARDS was 29.3%. In comparison, the LUNG SAFE study in 2016 reported that approximately 10% of ICU inpatients and 23.4% of mechanically ventilated patients developed ARDS [2]. Additionally, the CHARDS net study highlighted that the incidence rate of ARDS in 18 ICUs across mainland China was 3.57%. However, notable disparities were observed among different ICUs, with incidence rates varying significantly from one ICU to another, ranging from 1.0% to 16.7% [3]. The slightly higher incidence rate of ARDS observed in this study compared to the national study could potentially be attributed to the prospective observational nature of this study, which exclusively included patients with high-risk factors for ARDS meeting admission criteria, thus potentially skewing the incidence rate. Furthermore, differences in research methodologies and treatment management among various research institutions may also contribute to this discrepancy in incidence rates.

Despite advancements, the morbidity and mortality rates of ARDS patients remain substantial, reaching as high as 40.0%, with these rates escalating with the severity of ARDS [18]. However, clinical diagnosis of ARDS is frequently delayed or overlooked. In the Lung Safe study, only 60.2% of all ARDS patients were identified by clinicians, with recognition rates ranging from 51.3% for mild ARDS to 78.5% for severe ARDS [2]. Additionally, less than two-thirds of ARDS patients receive the recommended tidal volume of 8 mL/kg or less of their expected body weight [2]. Delayed recognition of ARDS can fail to implement strategies known to improve ARDS survival, such as protective mechanical ventilation [19], fluid restriction [20], and prone ventilation [21, 22], consequently leading to increased mortality [23]. Despite ongoing research, there remains a lack of a widely accepted predictive model to identify individuals at risk of ARDS. One notable strength of this study is its prospective design, wherein patients were screened daily until the 7th day of ICU admission, effectively mitigating missed or misdiagnosed cases of ARDS. Moreover, by employing well-established ARDS risk prediction models and disease severity scoring systems from previous studies, this study has developed a prediction model for ARDS incidence that aligns with clinical practice and exhibits improved general applicability.

In this study, we employed LASSO regression and multifactorial logistic regression analyses to identify five key variables (Gender, LIPS, shock, lung contusion, and Hepatic disease) out of 35 factors associated with the risk of ARDS in patients. The LIPS score model, proposed by Gajic and Trillo-Alvarez in 2011, has been widely used in recent years to assess the risk stratification of patients with acute lung injury (ALI). It systematically grades the extent and degree of lung injury based on factors such as susceptibility, high-risk surgeries and traumas, and other risk-modifying factors. The LIPS score has demonstrated good predictive value for ALI, with AUC ranging from 0.80 to 0.84 [7, 24]. Studies conducted in China by Xiejianfeng et al. reported an AUC of 0.770 for the LIPS score [25], while Soto et al. demonstrated a close association between the LIPS score and the occurrence of ARDS, with an AUC of 0.740 [26]. The sensitivity of the LIPS score ≥ 4 in predicting ALI was 90.3%, the specificity was 30.9%, the positive predictive value was 17.3%, and the negative predictive value was 95.2% [26]. The specificity of the LIPS score in predicting the occurrence of ARDS is low, and it may not be accurate for clinical risk prediction alone. Our model combines the LIPS score with other clinical data, and the final model had a sensitivity of 63.6%, a specificity of 83.6%, a positive predictive value of 41.9%, and a negative predictive value of 92.5% in the test set. (Table S8).

Furthermore, male patients are more predisposed to ARDS compared to females. In a secondary analysis of the LUNG SAFE study, it was found that ARDS occurred more commonly in men than in women, with 905 (38%) women compared to 1472 (62%) men out of 2377 ARDS patients [27]. A similar male predisposition to ARDS has been reported in patients with COVID-19-associated ARDS [28]. Patients with liver disease are also at a higher risk of developing ARDS [29]. In patients with COVID-19, advanced liver disease has been associated with worse respiratory outcomes, including increased mortality and the need for mechanical ventilation [30]. This could be attributed to impaired function of the hepatic reticuloendothelial system and hepatocytes, which compromises systemic and pulmonary defenses, thereby impacting the onset and remission of ARDS [31, 32]. In our study, combined liver disease was identified as an independent risk factor for the development of ARDS. Additionally, shock and pulmonary contusion were identified as risk factors for ARDS in both the Lung Safe study and our study [2]. However, other risk factors such as sepsis, pneumonia, and trauma did not show predictive value for the development of ARDS in our multifactorial logistic regression analysis, possibly due to covariance with factors included in the LIPS scoring system.

In this study, we used several ML models and found that logistic regression models outperformed other models after analyzing AUC, DCA, calibration curves, and PR curves. However, it has been a challenge to interpret machine learning predictive models more comprehensively and to present the predictions visually to clinicians. Therefore, the SHAP approach was used to interpret the logistic model, which ensures model performance and clinical interpretability. This will help physicians better understand the decision-making process of the model and facilitate the use of the prediction results. In the external validation cohort, when the prediction specificity was fixed at 83.6%, the negative predictive value was 92.5% and the positive predictive value was only 41.9%. Therefore, the model may not be able to fully provide decision support to clinicians. In clinical practice, it is necessary to evaluate the benefits of early identification of people at risk of ARDS and their additional costs.

Indeed, our study has several limitations that warrant consideration. Firstly, it is a prospective single-center study with a relatively small sample size. Consequently, the generalizability of our findings may be limited, and validation in larger, multicenter studies is necessary to confirm the robustness of our results. Secondly, while we validated our model using an external cohort, it is essential to acknowledge that this cohort consisted of retrospective data. This may introduce selection bias and affect the generalizability of our model's performance to other settings. Thirdly, our prediction model was based on data available within 24 h of ICU admission and failed to use data on dynamic changes in indicators, which may have somewhat overlooked subsequent events that altered the occurrence of ARDS and caused confounding factors.

Conclusions

In conclusion, our ML model, incorporating sex, LIPS, shock, pulmonary contusion, and liver disease as risk factors, effectively predicts ARDS in high-risk patients. Compared to the LIPS score, the model demonstrates a significantly higher positive predictive value (41.9% vs. 18%). In high-risk populations, the model enhances the identification of truly at-risk individuals, optimizes medical resource allocation, and provides additional intervention time for patients. Furthermore, the SHAP method was used to explain personalized ARDS risk, fostering greater trust among clinicians in the model’s predictions. However, these conclusions require validation through further randomized controlled trials.