A comparison of scoring systems for predicting mortality and sepsis in the emergency department patients with a suspected infection

Article information

Clin Exp Emerg Med. 2021;8(4):289-295
Publication date (electronic) : 2021 December 31
doi : https://doi.org/10.15441/ceem.20.145
Department of Emergency Medicine, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand
Correspondence to: Theerapon Tangsuwanaruk Department of Emergency Medicine, Faculty of Medicine, Chiang Mai University, 110 Inthawaroros road, Sribhumi, Amphoe Muang Chiang Mai, Chiang Mai 50200, Thailand E-mail: theerapon.t@cmu.ac.th
Received 2020 November 1; Revised 2021 February 16; Accepted 2021 March 8.

Abstract

Objective

We aimed to compare the modified National Early Warning Score (mNEWS), quick Sequential Organ Failure Assessment (qSOFA) score, modified Systemic Inflammatory Response Syndrome (mSIRS) score, and modified Search Out Severity (mSOS) score in predicting mortality and sepsis among patients suspected of first observed infections in the emergency department. The modified scores were created by removing variables for simplicity.

Methods

This was a prospective cohort study that enrolled adult patients presenting at the emergency department with signs and symptoms suggesting infection. The mNEWS, qSOFA score, mSIRS score, and mSOS score were calculated using triage data. The SOFA score was a reference standard for sepsis diagnosis. All patients were monitored for up to 30 days after the initial visit to measure each scoring system’s ability to predict 30-day mortality and sepsis.

Results

There were 260 patients included in the study. The 30-day mortality prediction with mNEWS ≥5 had the highest sensitivity (91.18%). The highest area under the receiver operating characteristic curve (AUC) for the 30-day mortality prediction was mNEWS (0.607), followed by qSOFA (0.605), mSOS (0.550), and mSIRS (0.423). The sepsis prediction with mNEWS ≥5 had the highest sensitivity (96.48%). The highest AUC for the sepsis prediction was also mNEWS (0.685), followed by qSOFA (0.605), mSOS (0.480), and mSIRS (0.477).

Conclusion

mNEWS was an acceptable scoring system screening tool for predicting mortality and sepsis in patients with a suspected infection.

INTRODUCTION

Sepsis is one of the most challenging global health issues. In 2017, the total sepsis-related mortality worldwide was 19.7% and was higher in males, especially those in low, low-middle, or middle sociodemographic index areas [1]. Rapidly diagnosing sepsis results in early treatment, which decreases mortality. Currently, the Surviving Sepsis Campaign provides guidelines for early detection and treatment [2].

There are several screening tools for predicting sepsis and mortality. The quick Sequential Organ Failure Assessment (qSOFA) is used as a screening tool among patients with suspected infection in the emergency department (ED) or general hospital ward settings [3,4]. Previous studies showed that qSOFA was better than the Systemic Inflammatory Response Syndrome (SIRS) for predicting mortality [5,6]. qSOFA also had higher specificity but lower sensitivity compared to SIRS [5,7]. Further, the National Early Warning Score (NEWS) correlated with increased mortality [8-10], and a score that increased by three or more indicated a risk of severe sepsis and septic shock [11]. NEWS was hypothesized to be comparable with qSOFA and SIRS as a sepsis screening tool, but a later study showed that NEWS was the most accurate screening tool for sepsis [12]. Another study reported that the Search Out Severity (SOS) score could also identify patients with an infection, but the study was performed only on admitted hospital patients [13,14].

We found these scoring systems challenging to use at triage owing to immediate data unavailability, including laboratory results, supplemental oxygen decisions, and urine output records. Therefore, we removed some variables to simplify these scores, then compared each modified scoring system’s ability to predict the 30-day mortality and sepsis in ED patients with a suspected infection. This study aimed to identify the quickest screening tool to use in triage.

METHODS

Study design

This was a prospective cohort study. Data were collected from adult patients who met the inclusion criteria. The patient or a legally authorized representative gave informed consent according to ethical conduct. The institutional review board of the Faculty of Medicine, Chiang Mai University approved the study (470/2017), which is registered with the Thai Clinical Trials Registry (TCTR20200526012, https://www.thaiclinicaltrials.org). We followed the Strengthening the Reporting of Observational Studies in Epidemiology statement recommendations.

Study setting and population

The study was conducted in a university-based, tertiary care hospital ED, with more than 33,000 visits annually. Adult patients with signs and symptoms of suspected infection who visited the ED between December 15, 2017 and March 31, 2018, were enrolled. The signs and symptoms of suspected infection included (but were not limited to): fever, productive cough, turbid urine, and swelling/redness of the skin. The exclusion criteria were patients admitted due to reinfection, pregnancy, patients transferred from other clinics or hospitals, patients lost to follow-up, or incomplete medical records.

Study protocol

Registered nurses trained in triage informed eligible patients, obtained informed consent, and recorded data to calculate the scores. As this study aimed to identify the quickest tool to use in triage, we removed variables from the screening assessments that were not readily available at triage. For NEWS, the oxygen supplement parameter was removed owing to the subjectivity of decision-making by health care personnel. Moreover, patients with sepsis could present with increased respiratory rate, decreased oxygen saturation, and declined consciousness level, all listed as NEWS parameters. Additionally, we removed the white blood cell count and the urine output record from the SIRS and SOS scores, respectively, owing to their unavailability at triage. In this study, the scoring systems were termed modified SIRS (mSIRS), modified NEWS (mNEWS), and modified SOS (mSOS) (Supplementary Tables 1-5) [4,9,14]. The qSOFA variables were already practical, so variables were not removed. The scoring variables were recorded in conventional and electronic medical records.

Outcome measures

The patient characteristics and scoring variables for each patient were obtained from the medical records by a researcher. Each patient’s Sequential Organ Failure Assessment (SOFA) score was calculated as a reference standard for sepsis diagnosis as there is no gold standard. A SOFA score of 2 points or more was considered sepsis according to the standard definition in Sepsis-3 [3]. The patient’s baseline SOFA score was zero unless the patient had a documented preexisting organ dysfunction. Cutoff points of mNEWS ≥5, qSOFA ≥2, mSIRS ≥2, and mSOS ≥4 were used to calculate each scoring system’s ability to identify patients with sepsis, which are the same cutoff points as in the original scoring system [4,10,14]. All patients were monitored for 30 days to evaluate each scoring system’s ability to predict 30-day mortality. All incomplete data affecting mNEWS, qSOFA, mSIRS, mSOS, and SOFA score calculations were excluded. However, other missing data were handled with multiple imputation methods.

Data analyses

The sample size was calculated by the infinite population proportion to ensure an adequate number of patients in the study. A 172 patient sample size was adequate with an alpha error of 0.05, a maximum tolerated error of 0.075, and a dropout rate of 20%, based on a previously reported sensitivity of 72.3% for SIRS ≥2 [7].

Descriptive data were reported as counts and percentages. The mean, standard deviation, median, and interquartile ranges were used for continuous variables. Categorical variables were analyzed using the chi-square or Fisher exact tests. Continuous variables were compared using an independent samples t-test for normally distributed data or a Mann-Whitney U-test for non-normally distributed data. The Shapiro-Wilk test was used to determine the data distribution. Diagnostic accuracy was reported as the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio, negative likelihood ratio, accuracy, and the area under the receiver operating characteristic curve (AUROC). IBM SPSS Statistics ver. 22 (IBM Corp., Armonk, NY, USA) was used for general statistical analyses, and Stata ver. 16 (StataCorp., College Station, TX, USA) for AUROC comparisons. Statistical significance was designated as P<0.05.

RESULTS

In total, 260 patients were included in the study (Fig. 1). Of these, 142 (54.6%) were diagnosed with sepsis and had no missing data. Age, oxygen saturation, consciousness, the partial pressure of arterial oxygen, the platelet count, the bilirubin and creatinine levels, the mNEWS and qSOFA scores, and intensive care unit admission significantly differed between patients with and without sepsis (all P<0.05) (Table 1). There were more patients with alteration of consciousness among the sepsis patients (P<0.001). All patients were hospitalized. Seventeen patients (11.97%) with sepsis were admitted to the intensive care unit. Death within 30 days occurred in 23 patients with sepsis (16.20%) and 11 patients without (9.32%).

Fig. 1.

The study flowchart. SOFA, Sequential Organ Failure Assessment score.

The characteristics of patients

Table 2 compares the scoring systems’ ability to predict 30-day mortality. mNEWS ≥5 had the highest sensitivity (91.18%), while qSOFA ≥2 had the highest specificity (63.27%), PPV (19.42%), and NPV (91.08%). The highest AUROC for the 30-day mortality prediction was mNEWS (0.607), followed by qSOFA (0.605), mSOS (0.550), and mSIRS (0.423) (Fig. 2). Pairwise comparison of AUROC showed differences between mNEWS versus mSIRS and qSOFA versus mSIRS (Table 3).

Comparing the scoring systems’ ability to predict 30-day mortality and sepsis

Fig. 2.

Receiver operating characteristic curves of the scoring systems’ ability to predict 30-day mortality. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Area under the receiver operating characteristic curve comparisons of 30-day mortality and sepsis

Table 2 also shows a comparison of the scoring systems’ ability to predict sepsis. mNEWS ≥5 had the highest sensitivity (96.48%) and NPV (80%). qSOFA ≥2 had the highest specificity (69.49%) and PPV (65.05%). The highest AUROC for predicting sepsis was observed in mNEWS (0.685), followed by qSOFA (0.605) (Fig. 3). Pairwise comparison of AUROC showed differences between mNEWS versus qSOFA, mNEWS versus mSIRS, mNEWS versus mSOS, qSOFA versus mSIRS, and qSOFA versus mSOS (Table 3). The subsequent calculation results showed that the cutoff point for mNEWS with the best accuracy was ≥7, which had higher specificity (45.76%) and accuracy (66.54%) in sepsis predictions. However, the ≥7 cutoff point had a lower sensitivity (83.80%) than ≥5.

Fig. 3.

Receiver operating characteristic curves of each scoring systems’ ability to predict sepsis. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

DISCUSSION

Many studies have shown that NEWS is the most accurate scoring system [12]. Our NEWS modification, which removed the need for supplemental oxygen at triage, was also the most accurate for predicting 30-day mortality and sepsis. The mNEWS cutoff point ≥5 had high sensitivity but low specificity, implying that mNEWS may be used as a triage screening tool. Although mNEWS had the highest AUROC for the 30-day mortality and sepsis predictions, the value was less than 0.7, suggesting poor discrimination ability. Redfern et al. [15] reported that NEWS ≥5 had a higher AUROC than qSOFA ≥2 in terms of in-hospital mortality predictions, whereas Goulden et al. [10] found that the NEWS ≥5 AUROC was similar to qSOFA ≥2, but had a higher value than SIRS in in-hospital mortality predictions. However, the mortality endpoints differed from those in our study.

We found that mNEWS was better than qSOFA in predicting sepsis. Similarly, Usman et al. [12] found NEWS was the most accurate in detecting severe sepsis or septic shock compared to SIRS ≥2 and qSOFA ≥2. They reported a higher AUROC than ours, which might be due to population differences and the use of unmodified NEWS and SIRS.

In this study, qSOFA had the highest specificity and was the second-best way for predicting 30-day mortality and sepsis. However, its low sensitivity makes it unfit for its use as a screening tool. Similar studies also yielded the same result. Brink et al. [16] found that qSOFA ≥2 had a higher AUROC and specificity than SIRS ≥2 in 30-day mortality predictions. Usman et al. [12] also reported that qSOFA ≥2 had the highest specificity in severe sepsis or septic shock detection. Additionally, Redfern et al. [15] and Churpek et al. [17] showed that qSOFA ≥2 had a higher specificity than SIRS ≥2 in in-hospital mortality predictions. However, the latter studies had different mortality endpoints. One study proposed using the qSOFA score as a sepsis screening tool [3]. However, our study, similar to Usman et al. [12] and Churpek et al. [18] found that qSOFA had low sensitivity and might not be suitable as a sepsis screening tool.

SIRS was used for formally diagnosing sepsis, and a subsequent study found it to have high false positives [19]. Our study showed that modified SIRS had very low specificity and was not suitable as a sepsis screening tool. The mSOS scoring system was also not fit to use at triage owing to its low sensitivity and accuracy.

Our study has some limitations. First, patients’ known medical conditions, which might affect disease progression, were not included in the scoring system calculations. Second, 30 patients (11.54%) were referred out to other hospitals for continuing treatment and were considered alive at 30 days, and 23 patients (8.85%) who refused treatment and returned home were considered dead at 30 days. These patients might have different outcomes, which could alter the study results. Third, the number of patients included in our study was relatively small. Extending the enrollment period to include additional patients should be considered for future studies. Fourth, the infection sources were not documented, although several baseline characteristics were explored in this study.

In summary, mNEWS is an acceptable scoring system screening tool for predicting 30-day mortality and sepsis and could provide quicker evaluation at triage.

Notes

No potential conflict of interest relevant to this article was reported.

SUPPLEMENTARY MATERIAL

Supplementary Tables are available from: https://doi.org/10.15441/ceem.20.145.

Supplementary Table 1.

Comparisons of NEWS, qSOFA, SIRS, SOS

Supplementary Table 2.

National Early Warning Score

Supplementary Table 3.

Quick sequential (sepsis-related) organ failure assessment score

Supplementary Table 4.

Systemic inflammatory response syndrome score

Supplementary Table 5.

Search out severity (SOS) score

References

1. Rudd KE, Johnson SC, Agesa KM, et al. Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the Global Burden of Disease Study. Lancet 2020;395:200–11.
2. Levy MM, Evans LE, Rhodes A. The Surviving Sepsis Campaign Bundle: 2018 update. Crit Care Med 2018;46:997–1000.
3. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis: for the Third International Consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:762–74.
4. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:801–10.
5. Herwanto V, Shetty A, Nalos M, et al. Accuracy of quick sequential organ failure assessment score to predict sepsis mortality in 121 studies including 1,716,017 individuals: a systematic review and meta-analysis. Crit Care Explor 2019;1:e0043.
6. Finkelsztein EJ, Jones DS, Ma KC, et al. Comparison of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sepsis outside the intensive care unit. Crit Care 2017;21:73.
7. Williams JM, Greenslade JH, McKenzie JV, Chu K, Brown AF, Lipman J. Systemic inflammatory response syndrome, quick sequential organ function assessment, and organ dysfunction: insights from a prospective database of ED patients with infection. Chest 2017;151:586–96.
8. Corfield AR, Lees F, Zealley I, et al. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg Med J 2014;31:482–7.
9. Royal College of Physicians. National Early Warning Score (NEWS): standardizing the assessment of acute illness severity in the NHS London: Royal College of Physicians; 2012.
10. Goulden R, Hoyle MC, Monis J, et al. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J 2018;35:345–9.
11. Keep JW, Messmer AS, Sladden R, et al. National early warning score at Emergency Department triage may allow earlier identification of patients with severe sepsis and septic shock: a retrospective observational study. Emerg Med J 2016;33:37–41.
12. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department. Am J Emerg Med 2019;37:1490–7.
13. Khwannimit B, Bhurayanontachai R, Vattanavanit V. Comparison of the accuracy of three early warning scores with SOFA score for predicting mortality in adult sepsis and septic shock patients admitted to intensive care unit. Heart Lung 2019;48:240–4.
14. Pisitsak C, Champunot R, Morakul S. The role of the hospitalists in the workforce to address the shortages of intensivists in hospitals here in Thailand. J Med Assoc Thai 2014;97 Suppl 1:S132–6.
15. Redfern OC, Smith GB, Prytherch DR, Meredith P, Inada-Kim M, Schmidt PE. A comparison of the quick sequential (sepsis-related) organ failure assessment score and the national early warning score in non-ICU patients with/without infection. Crit Care Med 2018;46:1923–33.
16. Brink A, Alsma J, Verdonschot RJ, et al. Predicting mortality in patients with suspected sepsis at the emergency department: a retrospective cohort study comparing qSOFA, SIRS and National Early Warning Score. PLoS One 2019;14:e0211133.
17. Churpek MM, Snyder A, Han X, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195:906–11.
18. Churpek MM, Snyder A, Sokol S, Pettit NN, Edelson DP. Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores. Crit Care Med 2017;45:1805–12.
19. Kaukonen KM, Bailey M, Bellomo R. Systemic inflammatory response syndrome criteria for severe sepsis. N Engl J Med 2015;373:881.

Article information Continued

Notes

Capsule Summary

What is already known

The National Early Warning Score (NEWS), the quick Sequential Organ Failure Assessment, and the Systemic Inflammatory Response Syndrome scores are useful in sepsis diagnosis. The Search Out Severity score is also used to identify patients with infection. However, these scores are difficult to use as a screening tool at triage owing to their complex variables.

What is new in the current study

The modified NEWS, which removed supplemental oxygen at triage as a variable, is a simplified alternative scoring system screening tool for predicting 30-day mortality and sepsis. Modified NEWS had better sensitivity compared with quick Sequential Organ Failure Assessment, modified Systemic Inflammatory Response Syndrome, and modified Search Out Severity.

Fig. 1.

The study flowchart. SOFA, Sequential Organ Failure Assessment score.

Fig. 2.

Receiver operating characteristic curves of the scoring systems’ ability to predict 30-day mortality. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Fig. 3.

Receiver operating characteristic curves of each scoring systems’ ability to predict sepsis. CI, confidence interval; mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Table 1.

The characteristics of patients

Variable Patients with sepsis (SOFA ≥ 2) (n = 142) Patients without sepsis (SOFA < 2) (n = 118) P-value
Age (yr) 69.5 (59.0–82.0) 64.0 (51.0–77.0) 0.009a)
Sex, male 72 (50.70) 56 (47.46) 0.602b)
Oxygen saturation (%) 93 (90–95) 96 (94–98) < 0.001a)
Temperature (°C) 38.6 (38.0–39.3) 38.5 (38.1–39.4) 0.598a)
Systolic blood pressure (mmHg) 126 (104–148) 128 (112–148) 0.462a)
Diastolic blood pressure (mmHg) 70 (57–83) 73 (63–85) 0.106a)
Pulse rate (beats/min) 111 ± 26 114 ± 19 0.166c)
Respiratory rate (breaths/min) 28 (24–32) 24 (22–32) 0.386a)
Consciousness 0.004b)
Awake 75 (52.82) 83 (70.34)
Responds to verbal stimulation 52 (36.62) 33 (27.97)
Responds to painful stimulation 11 (7.75) 2 (1.69)
Unresponsive 4 (2.82) 0 (0)
Partial pressure of arterial Oxygen (mmHg) 67 (57–80) 84 (71–91) < 0.001a)
Platelet count (× 103/mm3) 194 (112–268) 226 (169–285) 0.006a)
Bilirubin (mg/dL) 0.87 (0.50–1.71) 0.60 (0.42–1.09) < 0.001a)
Creatinine (mg/dL) 1.18 (0.85–1.84) 0.94 (0.75–1.29) 0.002a)
Patients with altered mental status 40 (28.17) 10 (8.47) < 0.001b)
Scoresa)
mNEWS 9 (7–10) 7 (5–9) < 0.001
qSOFA 1 (1–2) 1 (1–2) 0.002
mSIRS 3 (2–3) 3 (2–3) 0.459
mSOS 5 (4–6) 5 (4–6) 0.569
Patients admitted to the intensive care unit 17 (11.97) 2 (1.69) 0.002b)
Death within 30 days 23 (16.20) 11 (9.32) 0.102b)

Values are presented as median (interquartile range), number (%), or mean±standard deviation.

SOFA, Sequential Organ Failure Assessment; mNEWS, modified National Early Warning Score; qSOFA, quick SOFA score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

a)

Analyzed by Mann-Whitney U-test.

b)

Analyzed by chi-square test or the Fisher exact test.

c)

Analyzed by an independent samples t-test.

Table 2.

Comparing the scoring systems’ ability to predict 30-day mortality and sepsis

Category mNEWS ≥5 qSOFA ≥2 mSIRS ≥2 mSOS ≥4
30-Day mortality prediction
 Sensitivity 91.18 55.82 82.35 76.47
 Specificity 9.73 63.27 4.87 20.80
 Positive predictive value 13.19 19.42 11.52 12.68
 Negative predictive value 88.00 91.08 64.71 85.45
 Positive likelihood ratio 1.01 1.60 0.87 0.97
 Negative likelihood ratio 0.91 0.65 3.63 1.13
 Accuracy 20.38 62.69 15.00 28.08
Sepsis prediction
 Sensitivity 96.48 47.18 90.85 78.17
 Specificity 16.95 69.49 3.39 20.34
 Positive predictive value 58.30 65.05 53.09 54.15
 Negative predictive value 80.00 52.23 23.53 43.64
 Positive likelihood ratio 1.16 1.55 0.94 0.98
 Negative likelihood ratio 0.21 0.76 2.70 1.07
 Accuracy 60.38 57.31 51.15 51.92

mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

Table 3.

Area under the receiver operating characteristic curve comparisons of 30-day mortality and sepsis

Scoring systemsa) mNEWS qSOFA mSIRS
30-Day mortality prediction
 qSOFA 0.961 - -
 mSIRS < 0.001 0.001 -
 mSOS 0.508 0.504 0.108
Sepsis prediction
 qSOFA 0.001 - -
 mSIRS < 0.001 0.003 -
 mSOS < 0.001 0.012 0.954

mNEWS, modified National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment score; mSIRS, modified Systemic Inflammatory Response Syndrome score; mSOS, modified Search Out Severity score.

a)

Area under the receiver operating characteristic curve comparisons as P-value.