A targeted real-time early warning score (TREWScore) for septic shock
Science Translational Medicine • 5 Aug 2015 • Vol 7, Issue 299 • p. 299ra122 • DOI: 10.1126/scitranslmed.aab3719
Evening the score against sepsis
Sepsis is a major cause of death, which remains difficult to treat despite modern antibiotics. Early aggressive treatment of this disease improves patient mortality, but the tools currently available in the clinic do not predict who will develop sepsis and its late manifestation, septic shock, until the patients are already in advanced stages of the disease. Henry et al. used readily available data from patient monitors and medical records to develop TREWScore, a targeted real-time early warning score that predicts in advance which patients are at risk for septic shock. With a median lead time of over 24 hours, this scoring algorithm may allow clinicians enough time to intervene before the patients suffer the most damaging effects of sepsis.
Abstract
Sepsis is a leading cause of death in the United States, with mortality highest among patients who develop septic shock. Early aggressive treatment decreases morbidity and mortality. Although automated screening tools can detect patients currently experiencing severe sepsis and septic shock, none predict those at greatest risk of developing shock. We analyzed routinely available physiological and laboratory data from intensive care unit patients and developed “TREWScore,” a targeted real-time early warning score that predicts which patients will develop septic shock. TREWScore identified patients before the onset of septic shock with an area under the ROC (receiver operating characteristic) curve (AUC) of 0.83 [95% confidence interval (CI), 0.81 to 0.85]. At a specificity of 0.67, TREWScore achieved a sensitivity of 0.85 and identified patients a median of 28.2 [interquartile range (IQR), 10.6 to 94.2] hours before onset. Of those identified, two-thirds were identified before any sepsis-related organ dysfunction. In comparison, the Modified Early Warning Score, which has been used clinically for septic shock prediction, achieved a lower AUC of 0.73 (95% CI, 0.71 to 0.76). A routine screening protocol based on the presence of two of the systemic inflammatory response syndrome criteria, suspicion of infection, and either hypotension or hyperlactatemia achieved a lower sensitivity of 0.74 at a comparable specificity of 0.64. Continuous sampling of data from the electronic health records and calculation of TREWScore may allow clinicians to identify patients at risk for septic shock and provide earlier interventions that would prevent or mitigate the associated morbidity and mortality.
Get full access to this article
View all available purchase options and get full access to this article.
Already a Subscriber?Sign In
Supplementary Material
Summary
Materials and Methods
Table S1. Sample feature coefficients learned by TREWScore for a single imputation of the development data set.
Table S2. Patient characteristics.
Resources
File (7-299ra122_sm.pdf)
REFERENCES AND NOTES
1
Angus D. C., Linde-Zwirble W. T., Lidicker J., Clermont G., Carcillo J., Pinsky M. R., Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit. Care Med. 29, 1303–1310 (2001).
2
HCUP Facts and Figures: Statistics on Hospital-based Care in the United States, 2009 (Agency for Healthcare Research and Quality, Rockville, MD, 2011); www.hcup-us.ahrq.gov/reports.jsp.
3
Kumar G., Kumar N., Taneja A., Kaleekal T., Tarima S., McGinley E., Jimenez E., Mohan A., Khan R. A., Whittle J., Jacobs E., Nanchal R.Milwaukee Initiative in Critical Care Outcomes Research Group of Investigators, Nationwide trends of severe sepsis in the 21st century (2000–2007). Chest 140, 1223–1231 (2011).
4
Rivers E., B. Nguyen, S. Havstad, J. Ressler, A. Muzzin, B. Knoblich, E. Peterson, M. Tomlanovich; Early Goal-Directed Therapy Collaborative Group, Early goal-directed therapy in the treatment of severe sepsis and septic shock. N. Engl. J. Med. 345, 1368–1377 (2001).
5
Nguyen H. B., Corbett S. W., Steele R., Banta J., Clark R. T., Hayes S. R., Edwards J., Cho T. W., Wittlake W. A., Implementation of a bundle of quality indicators for the early management of severe sepsis and septic shock is associated with decreased mortality. Crit. Care Med. 35, 1105–1112 (2007).
6
Sebat F., Musthafa A. A., Johnson D., Kramer A. A., Shoffner D., Eliason M., Henry K., Spurlock B., Effect of a rapid response system for patients in shock on time to treatment and mortality during 5 years. Crit. Care Med. 35, 2568–2575 (2007).
7
Kumar A., Roberts D., Wood K. E., Light B., Parrillo J. E., Sharma S., Suppes R., Feinstein D., Zanotti S., Taiberg L., Gurka D., Kumar A., Cheang M., Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit. Care Med. 34, 1589–1596 (2006).
8
Coba V., Whitmill M., Mooney R., Horst H. M., Brandt M. M., Digiovine B., Mlynarek M., McLellan B., Boleski G., Yang J., Conway W., Jordan J.The Henry Ford Hospital Sepsis Collaborative Group, Resuscitation Bundle Compliance in severe sepsis and septic shock: Improves survival, is better late than never. J. Intensive Care Med. 26, 304–313 (2011).
9
Castellanos-Ortega A., Suberviola B., García-Astudillo L. A., Holanda M. S., Ortiz F., Llorca J., Delgado-Rodríguez M., Impact of the Surviving Sepsis Campaign protocols on hospital length of stay and mortality in septic shock patients: Results of a three-year follow-up quasi-experimental study. Crit. Care Med. 38, 1036–1043 (2010).
10
ProCESS Investigators; Yealy D. M., Kellum J. A., Huang D. T., Barnato A. E., Weissfeld L. A., Pike F., Terndrup T., Wang H. E., Hou P. C., LoVecchio F., Filbin M. R., Shapiro N. I., Angus D. C., A randomized trial of protocol-based care for early septic shock. N. Engl. J. Med. 370, 1683–1693 (2014).
11
ARISE InvestigatorsANZICS Clinical Trials GroupPeake S. L., Delaney A., Bailey M., Bellomo R., Cameron P. A., Cooper D. J., Higgins A. M., Holdgate A., Howe B. D., Webb S. A., Williams P., Goal directed resuscitation for patients with early septic shock. N. Engl. J. Med. 371, 1496–1506 (2014).
12
Mouncey P. R., Osborn T. M., Power G. S., Harrison D. A., Sadique M. Z., Grieve R. D., Jahan R., Harvey S. E., Bell D., Bion J. F., Coats T. J., Singer M., Young J. D., Rowan K. M.ProMISe Trial Investigators, Trial of early, goal-directed resuscitation for septic shock. N. Engl. J. Med. 372, 1301–1311 (2015).
13
Levy M. M., Early goal directed therapy: What do we do now? Crit. Care 18, 705 (2014).
14
Ghanem-Zoubi N. O., Vardi M., Laor A., Weber G., Bitterman H., Assessment of disease-severity scoring systems for patients with sepsis in general internal medicine departments. Crit. Care 15, R95 (2011).
15
Barriere S. L., Lowry S. F., An overview of mortality risk prediction in sepsis. Crit. Care Med. 23, 376–393 (1995).
16
Jones A. E., Trzeciak S., Kline J. A., The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit. Care Med. 37, 1649–1654 (2009).
17
Vorwerk C., Loryman B., Coats T. J., Stephenson J. A., Gray L. D., Reddy G., Florence L., Butler N., Prediction of mortality in adult emergency department patients with sepsis. Emerg. Med. J. 26, 254–258 (2009).
18
Herasevich V., Pieper M. S., Pulido J., Gajic O., Enrollment into a time sensitive clinical study in the critical care setting: Results from computerized septic shock sniffer implementation. J. Am. Med. Inform. Assoc. 18, 639–644 (2011).
19
Acutely Ill Patients in Hospital: Recognition of and Response to Acute Illness in Adults in Hospital (NICE clinical guideline no. 50) (National Institute for Health and Clinical Excellence, London, UK, 2007); www.nice.org.uk/CG050).
20
Nguyen S. Q., Mwakalindile E., Booth J. S., Hogan V., Morgan J., Prickett C. T., Donnelly J. P., Wang H. E., Automated electronic medical record sepsis detection in the emergency department. PeerJ 2, e343 (2014).
21
Umscheid C. A., Betesh J., VanZandbergen C., Hanish A., Tait G., Mikkelsen M. E., French B., Fuchs B. D., Development, implementation, and impact of an automated early warning and response system for sepsis. J. Hosp. Med. 10, 26–31 (2015).
22
Nelson J., Smith B., Jared J., Younger J., Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann. Emerg. Med. 57, 500–504 (2011).
23
Morgan R. J. M., Williams F., Wright M. N., An early warning scoring system for detecting developing critical illness. Clin. Intens. Care Nurs. 8, 100 (1997).
24
Bates D. W., Saria S., Ohno-Machado L., Shah A., Escobar G., Big data in health care: Using analytics to identify and manage high-risk and high-cost patients. Health Aff. 33, 1123–1131 (2014).
25
Donzé J., Aujesky D., Williams D., Schnipper J. L., Potentially avoidable 30-day hospital readmissions in medical patients: Derivation and validation of a prediction model. JAMA Intern. Med. 173, 632–638 (2013).
26
Osborn T. M., Phillips G., Lemeshow S., Townsend S., Schorr C. A., Levy M. M., Dellinger R. P., Sepsis Severity Score: An internationally derived scoring system from the Surviving Sepsis Campaign database. Crit. Care Med. 42, 1969–1976 (2014).
27
Saria S., Rajani A., Gould J., Koller D., Penn A., Integration of early physiological responses predicts later illness severity in preterm infants. Sci. Transl. Med. 2, 48ra65 (2010).
28
Wiens J., Horvitz E., and , Guttag J. V., Patient risk stratification for hospital-associated C. diff as a time-series classification task. Adv. in Neural Inf. Process. Syst. 25, 476–484 (2012).
29
Gajic O., Dabbagh O., Park P. K., Adesanya A., Chang S. Y., Hou P., Anderson H., Hoth J. J., Mikkelsen M. E., Gentile N. T., Gong M. N., Talmor D., Bajwa E., Watkins T. R., Festic E., Yilmaz M., Iscimen R., Kaufman D. A., Esper A. M., Sadikot R., Douglas I., Sevransky J., Malinchoc M.U.S. Critical Illness and Injury Trials Group: Lung Injury Prevention Study Investigators (USCIITG-LIPS), Early identification of patients at risk of acute lung injury: Evaluation of Lung Injury Prediction Score in a multicenter cohort study. Am. J. Respir. Crit. Care Med. 183, 462–470 (2011).
30
Stanculescu I., Williams C. K. I., Freer Y., Autoregressive hidden Markov models for the early detection of neonatal sepsis. IEEE J. Biomed. Health Inform. 18, 1560–1570 (2014).
31
Griffin M. P., Lake D. E., O’Shea T. M., Moorman J. R., Heart rate characteristics and clinical signs in neonatal sepsis. Pediatr. Res. 61, 222–227 (2007).
32
Gultepe E., Green J. P., Nguyen H., Adams J., Albertson T., Tagkopoulos I., From vital signs to clinical outcomes for patients with sepsis: A machine learning basis for a clinical decision support system. J. Am. Med. Inform. Assoc. 21, 315–325 (2014).
33
Giuliano K. K., Physiological monitoring for critically ill patients: Testing a predictive model for the early detection of sepsis. Am. J. Crit. Care 16, 122–130 (2007).
34
Thiel S. W., Rosini J. M., Shannon W., Doherty J. A., Micek S. T., Kollef M. H., Early prediction of septic shock in hospitalized patients. J. Hosp. Med. 5, 19–25 (2010).
35
Henry K., Paxton C., Kim K. S., Pham J., Saria S., REWS: Real-time early warning score for septic shock. Crit. Care Med. 42, abstract 63 (2014).
36
D. Shavdia, thesis, Septic shock: Providing early warnings through multivariate logistic regression models, Harvard-MIT Division of Health Sciences and Technology (2007).
37
Ho J. C., Lee C. H., Ghosh J., Septic shock prediction for patients with missing data. ACM Trans. Manag. Inf. Syst. 5, Article No. 1 (2014).
38
Paxton C., Niculescu-Mizil A., Saria S., Developing predictive models using electronic medical records: Challenges and pitfalls. AMIA Annu. Symp. Proc. 2013, 1109–1115 (2013).
39
K. Dyagilev, S. Saria, Learning (predictive) risk scores in the presence of censoring due to interventions. arXiv:1507.07295 (2015).
40
Goldberger A. L., Amaral L. A., Glass L., Hausdorff J. M., Ivanov P. C., Mark R. G., Mietus J. E., Moody G. B., Peng C. K., Stanley H. E., PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation 101, e215–e220 (2000).
41
A. Hendrich, J. Byrd, V. Naylor, T. Daniels, R. Weakland, C. Wagner, Leading Edge Advanced Practice Topic (LEAPT): Severe Sepsis and Septic Shock (Health Research & Educational Trust, Chicago, IL, 2014); www.hret-hen.org/index.php?option=com_phocadownload&view=category&id=370&Itemid=369).
42
S. Townsend, T. Clemmer, D. Jacobsen, IHI Expedition: Treating Sepsis in the Emergency Department and Beyond, Session 6 (Institute for Healthcare Improvement, Cambridge, MA, 2013); www.ihi.org/Engage/Memberships/Passport/Pages/Treating-Sepsis.aspx).
43
Subbe C. P., Kruger M., Rutherford P., Gemmel L., Validation of a modified Early Warning Score in medical admissions. QJM 94, 521–526 (2001).
44
Smith G. B., D. R. Prytherch, P. E. Schmidt, P. I. Featherstone, Review and performance evaluation of aggregate weighted ‘track and trigger’ systems. Resuscitation 77, 170–179 (2008).
45
Dellinger R. P., Levy M. M., Rhodes A., Annane D., Gerlach H., Opal S. M., Sevransky J. E., Sprung C. L., Douglas I. S., Jaeschke R., Osborn T. M., Nunnally M. E., Townsend S. R., Reinhart K., Kleinpell R. M., Angus D. C., Deutschman C. S., Machado F. R., Rubenfeld G. D., Webb S. A., Beale R. J., Vincent J. L., Moreno R.Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup, Surviving Sepsis Campaign: International guidelines for management of severe sepsis and septic shock: 2012. Crit. Care Med. 41, 580–637 (2013).
46
Berger T., Green J., Horeczko T., Hagar Y., Garg N., Suarez A., Panacek E., Shapiro N., Shock index and early recognition of sepsis in the emergency department: Pilot study. West. J. Emerg. Med. 14, 168–174 (2013).
47
H. D. Humes, in Kelley’s Essentials of Internal Medicine (Lippincott Williams & Wilkins, Philadelphia, ed. 2, 2001), chap. 18.
48
Rachoin J. S., Daher R., Moussallem C., Milcarek B., Hunter K., Schorr C., Abboud M., Henry P., Weisberg L. S., The fallacy of the BUN:creatinine ratio in critically ill patients. Nephrol. Dial. Transplant. 27, 2248–2254 (2012).
49
C. Hug, “Detecting hazardous intensive care patient episodes using real-time mortality models” (MIT Technical Report, 2009).
50
van der Sijs H., Aarts J., Vulto A., Berg M., Overriding of drug safety alerts in computerized physician order entry. J. Am. Med. Inform. Assoc. 13, 138–147 (2006).
51
Gu W. J., Wang F., Bakker J., Tang L., Liu J. C., The effect of goal-directed therapy on mortality in patients with sepsis - earlier is better: A meta-analysis of randomized controlled trials. Crit. Care 18, 570 (2014).
52
Campbell J., Payne T., A comparison of four schemes for codification of problem lists. Proc. Annu. Symp. Comput. Appl. Med. Care 201–205 (1994).
53
Whittaker S. A., Mikkelsen M. E., Gaieski D. F., Koshy S., Kean C., Fuchs B. D., Severe sepsis cohorts derived from claims-based strategies appear to be biased toward a more severely ill patient population. Crit. Care Med. 41, 945–953 (2013).
54
Saria S., McElvain G., Rajani A. K., Penn A. A., Koller D. L., Combining structured and free-text data for automatic coding of patient outcomes. AMIA Annu. Symp. Proc. 2010, 712–716 (2010).
55
Levy M. M., Dellinger R. P., Townsend S. R., Linde-Zwirble W. T., Marshall J. C., Bion J., Schorr C., Artigas A., Ramsay G., Beale R., Parker M. M., Gerlach H., Reinhart K., Silva E., Harvey M., Regan S., Angus D. C.Surviving Sepsis Campaign, The Surviving Sepsis Campaign: Results of an international guideline-based performance improvement program targeting severe sepsis. Crit. Care Med. 38, 367–374 (2010).
56
Pan W., A multiple imputation approach to Cox regression with interval-censored data. Biometrics 56, 199–203 (2000).
57
Zhang Z., Sun L., Sun J., Finkelstein D. M., Regression analysis of failure time data with informative interval censoring. Stat. Med. 26, 2533–2546 (2007).
58
R. Little, D. Rubin, Statistical Analysis with Missing Data (Wiley Series in Probability and Mathematical Statistics, Wiley, New York, NY, 2002).
59
T. M. Therneau, T. Lumley, The Survival Package (The Comprehensive R Archive Network, 2014).
60
Friedman J., Hastie T., Tibshirani R., Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33, 1–22 (2010).
61
Simon N., Friedman J., Hastie T., Tibshirani R., Regularization paths for Cox’s proportional hazards model via coordinate descent. J. Stat. Softw. 39, 1–13 (2011).
62
Tibshirani R., The lasso method for variable selection in the cox model. Stat. Med. 16, 385–395 (1997).
63
Heagerty P. J., Lumley T., Pepe M. S., Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics 56, 337–344 (2000).
64
Blanche P., Dartigues J.-F., Jacqmin-Gadda H., Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom. J. 55, 687–704 (2013).
Information & Authors
Information
Published In
Science Translational Medicine
Volume 7 | Issue 299
August 2015
August 2015
Copyright
Copyright © 2015, American Association for the Advancement of Science.
Submission history
Received: 17 April 2015
Accepted: 14 July 2015
Acknowledgments
We thank M. Wei and C. Paxton for their help in setting up the database and a first prototype of the feature extraction algorithms; J. Pham and K. S. Kim for fruitful clinical discussions that led to REWS, a prototype that inspired this work; and C. Venghaus for managing the secure server where the analyses were conducted. Funding: This research was supported by National Science Foundation Graduate Research Fellowship award ID 1232825, Google Research grant ID 1202463721, the Gordon and Betty Moore Foundation, and the Johns Hopkins University Whiting School of Engineering faculty start-up funds. The funders had no role in the study design, data analysis, decision to publish, or preparation of the manuscript. Author contributions: S.S., K.E.H., and D.N.H. designed the study. K.E.H. and S.S. designed the model, undertook the statistical analysis, and wrote the manuscript. D.N.H. provided clinical analysis and wrote the manuscript. P.J.P. provided clinical input and edited the manuscript. Competing interests: The authors declare that they have no competing interests.
Authors
Metrics & Citations
Metrics
Article Usage
Altmetrics
Citations
Export citation
Select the format you want to export the citation of this publication.
Cited by
- Tensor learning of pointwise mutual information from EHR data for early prediction of sepsis, Computers in Biology and Medicine, 134, (104430), (2021).https://doi.org/10.1016/j.compbiomed.2021.104430
- Generalization in Clinical Prediction Models: The Blessing and Curse of Measurement Indicator Variables, Critical Care Explorations, 3, 7, (e0453), (2021).https://doi.org/10.1097/CCE.0000000000000453
- undefined, 2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), (522-527), (2021).https://doi.org/10.1109/ICCECE51280.2021.9342457
- Kinematics approach with neural networks for early detection of sepsis (KANNEDS), BMC Medical Informatics and Decision Making, 21, 1, (2021).https://doi.org/10.1186/s12911-021-01529-3
- A systematic review of rapid response applications based on early warning score for early detection of inpatient deterioration, Informatics for Health and Social Care, 46, 2, (148-157), (2021).https://doi.org/10.1080/17538157.2021.1873349
- Predicting postoperative opioid use with machine learning and insurance claims in opioid-naïve patients, The American Journal of Surgery, 222, 3, (659-665), (2021).https://doi.org/10.1016/j.amjsurg.2021.03.058
- Non-occlusive mesenteric ischemia: Diagnostic challenges and perspectives in the era of artificial intelligence, World Journal of Gastroenterology, 27, 26, (4088-4103), (2021).https://doi.org/10.3748/wjg.v27.i26.4088
- An Explainable Machine Learning Model for Early Prediction of Sepsis Using ICU Data, Infectious Diseases and Sepsis [Working Title], (2021).https://doi.org/10.5772/intechopen.94701
- Diagnostic and prognostic capabilities of a biomarker and EMR‐based machine learning algorithm for sepsis, Clinical and Translational Science, 14, 4, (1578-1589), (2021).https://doi.org/10.1111/cts.13030
- HeMA: A hierarchically enriched machine learning approach for managing false alarms in real time: A sepsis prediction case study, Computers in Biology and Medicine, 131, (104255), (2021).https://doi.org/10.1016/j.compbiomed.2021.104255
- See more
Loading...
View Options
Get Access
Log in to view the full text
AAAS login provides access to Science for AAAS members, and access to other journals in the Science family to users who have purchased individual subscriptions.
- Become a AAAS Member
- Activate your Account
- Purchase Access to Other Journals in the Science Family
- Account Help
Log in via OpenAthens.
Log in via Shibboleth.
More options
Register for free to read this article
As a service to the community, this article is available for free. Login or register for free to read this article.
View options
PDF format
Download this article as a PDF file
Download PDF