Jump to content

Choose locationCalifornia Pacific Medical Center
  • Sign in or Register
    • I want to choose my medical group or hospital
    • Clear my location
X Close Window

We've expanded your view

You are now viewing all services in the Sutter Health network. You can change your location above to narrow your view to a medical group, hospital, city or zip.
Sutter Health
  • Find Doctors
  • Find Locations
  • Treatments & Services
  • Locations
  • Sign in or Register
    • Find Doctors
    • Find Locations
    • Treatments & Services
    • Pay a Bill
    • Symptom Checker
    • Get Care Today
    • Diseases & Conditions
    • Health & Wellness
    • Classes & Events
    • Research & Clinical Trials
    • For Patients
    • About Sutter Health
    • Ways to Give
    • Careers
    • News
    • For Medical Professionals
    • Other Business Services
Close Search
  • Home
  • CPMC
  • Research
  • Aging and Longevity
Content

Validation of prediction models for critical care outcomes using natural language processing of electronic health record data.

Description

Marafino BJ, Park M, Davies JM, Thombley R, Luft HS, Sing DC, Kazi DS, DeJong C, Boscardin WJ, Dean ML, Dudley RA., JAMA Netw Open. 1(8):e185097. doi: 10.1001/jamanetworkopen.2018.5097., 2018 Dec 07

Investigators

Harold (Hal) Luft, Ph.D., Senior Scientist

Abstract

Importance: Accurate prediction of outcomes among patients in intensive care units (ICUs) is important for clinical research and monitoring care quality. Most existing prediction models do not take full advantage of the electronic health record, using only the single worst value of laboratory tests and vital signs and largely ignoring information present in free-text notes. Whether capturing more of the available data and applying machine learning and natural language processing (NLP) can improve and automate the prediction of outcomes among patients in the ICU remains unknown.

Objectives: To evaluate the change in power for a mortality prediction model among patients in the ICU achieved by incorporating measures of clinical trajectory together with NLP of clinical text and to assess the generalizability of this approach.

Design, Setting, and Participants: This retrospective cohort study included 101?196 patients with a first-time admission to the ICU and a length of stay of at least 4 hours. Twenty ICUs at 2 academic medical centers (University of California, San Francisco [UCSF], and Beth Israel Deaconess Medical Center [BIDMC], Boston, Massachusetts) and 1 community hospital (Mills-Peninsula Medical Center [MPMC], Burlingame, California) contributed data from January 1, 2001, through June 1, 2017. Data were analyzed from July 1, 2017, through August 1, 2018.

Main Outcomes and Measures: In-hospital mortality and model discrimination as assessed by the area under the receiver operating characteristic curve (AUC) and model calibration as assessed by the modified Hosmer-Lemeshow statistic.Results: Among 101?196 patients included in the analysis, 51.3% (n = 51?899) were male, with a mean (SD) age of 61.3 (17.1) years; their in-hospital mortality rate was 10.4% (n = 10?505). A baseline model using only the highest and lowest observed values for each laboratory test result or vital sign achieved a cross-validated AUC of 0.831 (95% CI, 0.830-0.832). In contrast, that model augmented with measures of clinical trajectory achieved an AUC of 0.899 (95% CI, 0.896-0.902; P < .001 for AUC difference). Further augmenting this model with NLP-derived terms associated with mortality further increased the AUC to 0.922 (95% CI, 0.916-0.924; P < .001). These NLP-derived terms were associated with improved model performance even when applied across sites (AUC difference for UCSF: 0.077 to 0.021; AUC difference for MPMC: 0.071 to 0.051; AUC difference for BIDMC: 0.035 to 0.043; P < .001) when augmenting with NLP at each site.

Conclusions and Relevance: Intensive care unit mortality prediction models incorporating measures of clinical trajectory and NLP-derived terms yielded excellent predictive performance and generalized well in this sample of hospitals. The role of these automated algorithms, particularly those using unstructured data from notes and other sources, in clinical research and quality improvement seems to merit additional investigation.

Pubmed Abstract

Pubmed AbstractOpens New Window

Associated Topics

  • Aging and Longevity
  • Medical Informatics

Related Publications

Predicting need for advanced illness or palliative care in a primary care population using electronic health record data.

Jung K, Sudat SEK, Kwon N, Stewart WF, Shan NH.
J Biomed Inform. 92:103115.
2019 Apr 01

Impact of home-based, patient-centered support for people with advanced illness in an open health system: a retrospective claims analysis of health expenditures, utilization, and quality of care at end of life.

Sudat SEK, Franco A, Pressman AR, Rosenfeld K, Gornet E, Stewart W.
Palliat Med. 2018 Feb;32(2):485-492. doi: 10.1177/0269216317711824. Epub 2017 Jun 7.
2018 Feb 01

Adherence to placebo and mortality in the Beta Blocker Evaluation of Survival Trial (BEST).

Pressman A, Avins AL, Neuhaus J, Ackerson L, Rudd P.
Contemp Clin Trials. 33(3):492-8. doi: 10.1016/j.cct.2011.12.003. Epub 2012 Jan 12.
2012 May 01

Impact of home-based, patient-centered support for people with advanced illness in an open health system: a retrospective claims analysis of health expenditures, utilization, and quality of care at end of life.OLD

Sudat SEK, Franco A, Pressman AR, Rosenfeld K, Gornet E, Stewart W.
Palliat Med. 2018 Feb;32(2):485-492. doi: 10.1177/0269216317711824. Epub 2017 Jun 7.
2018 Feb 01

Predicting need for advanced illness or palliative care in a primary care population using electronic health record data.OLD

Jung K, Sudat SEK, Kwon N, Stewart WF, Shan NH.
J Biomed Inform. 92:103115.
2019 Apr 01
The Sutter Health Network of Care
Expertise to fit your needs
Primary Care

Check-ups, screenings and sick visits for adults and children.

Specialty Care

Expertise and advanced technologies in all areas of medicine.

Emergency Care

For serious accidents, injuries and conditions that require immediate medical care.

Urgent Care

After-hours, weekend and holiday services.

Walk-In Care

Convenient walk-in care clinics for your non-urgent health needs.

About Sutter

  • About Our Network
  • Annual Report
  • Awards
  • Community Benefit
  • Contact Us
  • News Room
  • Ways to Give
  • Find Care

  • Birth Centers
  • Care Centers
  • Emergency Rooms
  • Hospitals
  • Imaging
  • Labs
  • Surgery Centers
  • Urgent Care
  • Walk-In Care
  • View All >
  • Featured Services

  • Behavioral Health
  • Cancer Services
  • Family Medicine
  • Home Health and Hospice
  • Orthopedics
  • Pediatrics
  • Pregnancy
  • Primary Care
  • Women's Health
  • View All >
  • Patient Resources

  • Accepted Health Plans
  • Classes and Events
  • Estimate Costs
  • Flu Resources
  • Health and Wellness
  • Medical Records
  • Medicare
  • My Health Online
  • Pay a Bill
  • Symptom Checker
  • Our Team

  • For Employees
  • Physician Careers
  • Recruiting Events
  • Sutter Careers
  • Vendors
  • Volunteers
    • ADA Accessibility
    • Contact
    • Privacy Policy
    • Terms of Use

    • LinkedIn Opens new window
    • YouTube Opens new window
    • Facebook Opens new window
    • Twitter Opens new window
    • Glassdoor Opens new window
    • Instagram Opens new window

    Copyright © 2019 Sutter Health. All rights reserved. Sutter Health is a registered trademark of Sutter Health ®, Reg. U.S. Patent & Trademark office.