--- title: "Using FRAX in large healthcare delivery system | Sutter Health" description: "Multiple methods should be used to evaluate risk tool performance with less reliance on area under the receiver operating characteristics alone." url: "https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749" --- 1. [Home](https://www.sutterhealth.org/) / 2. [Research](https://www.sutterhealth.org/research) / 3. [Research publications](https://www.sutterhealth.org/research/publications) / 4. Using FRAX in large healthcare delivery system # Methods for assessing fracture risk prediction models: experience with FRAX in a large integrated health care delivery system. Pressman AR, Lo JC, Chandra M, & Ettinger B. _J Clin Densitom. 14(4):407-15. doi: 10.1016/j.jocd.2011.06.006. Epub 2011 Oct 1._ 2011-10-01 ## Abstract Area under the receiver operating characteristics (AUROC) curve is often used to evaluate risk models. However, reclassification tests provide an alternative assessment of model performance. We performed both evaluations on results from FRAX (World Health Organization Collaborating Centre for Metabolic Bone Diseases, University of Sheffield, UK), a fracture risk tool, using Kaiser Permanente Northern California women older than 50yr with bone mineral density (BMD) measured during 1997-2003. We compared FRAX performance with and without BMD in the model. Among 94,489 women with mean follow-up of 6.6yr, 1579 (1.7%) sustained a hip fracture. Overall, AUROCs were 0.83 and 0.84 for FRAX without and with BMD, suggesting that BMD did not contribute to model performance. AUROC decreased with increasing age, and BMD contributed significantly to higher AUROC among those aged 70yr and older. Using an 81% sensitivity threshold (optimum level from receiver operating characteristic curve, corresponding to 1.2% cutoff), 35% of those categorized above were reassigned below when BMD was added. In contrast, only 10% of those categorized below were reassigned to the higher risk category when BMD was added. The net reclassification improvement was 5.5% (p<0.01). Two versions of this risk tool have similar AUROCs, but alternative assessments indicate that addition of BMD improves performance. Multiple methods should be used to evaluate risk tool performance with less reliance on AUROC alone. [PubMed abstract](https://www.ncbi.nlm.nih.gov/pubmed/21958955/) ## Research topics - [Biostatistics](https://www.sutterhealth.org/research-search-results?query=%22%5C%22Biostatistics%5C%22%22) - [Musculoskeletal Disorders](https://www.sutterhealth.org/research-search-results?query=%22%5C%22Musculoskeletal+Disorders%5C%22%22) - [Women's Health](https://www.sutterhealth.org/research-search-results?query=%22%5C%22Women%27s+Health%5C%22%22) ## Related publications [View all publications](https://www.sutterhealth.org/research-search-results?verticalKey=research_publication) - Publication ### [Initiation of osteoporosis treatment after bone mineral density testing.](https://www.sutterhealth.org/research/publications/osteoporosis-tx-after-bone-mineral-density-testing-1055312705) Pressman A, Forsyth B, Ettinger B, Tosteson AN. Osteoporos Int. 12(5):337-42. 2001 May 01 - Publication ### [Randomized placebo-controlled placebo trial to determine the placebo effect size.](https://www.sutterhealth.org/research/publications/randomize-placebo-trial-determine-the-placebo-effect-size-1055312404) Gerdesmeyer L, Klueter T, Rahlfs VW, Muderis MA, Saxena A, Gollwitzer H, Harrasser N, Stukenberg M, Prehn-Kristensen A. Pain Physician. 20(5):387-396. 2017 Jul 01 - Publication ### [Naviculocuneiform coalition: case reports of two sibling soccer players.](https://www.sutterhealth.org/research/publications/naviculocuneiform-coalition-2-sib-soccer-players-1055312396) Saxena A, Fournier M. J Foot Ankle Surg. pii: S1067-2516(15)00414-7. doi: 10.1053/j.jfas.2015.09.001. \[Epub ahead of print\] 2015 Oct 17 - Publication ### [Fracture risk tool validation in an integrated healthcare delivery system.](https://www.sutterhealth.org/research/publications/fracture-risk-tool-validation-in-healthcare-delivery-system-1055312714) Lo JC, Pressman AR, Chandra M, Ettinger B. Am J Manag Care. 17(3):188-94. 2011 Mar 01 - Publication ### [Correlates of sexual abuse and subsequent risk taking.](https://www.sutterhealth.org/research/publications/correlates-sexual-abuse-subsequent-risk-taking-1055312680) Brown NL, Wilson SR, Kao Y-M, Luna V, Kuo ES, Rodriguez C, Lavori PW. Hispan J Behav Sci. 25(3):331-351. 2003 Aug 01 [View all publications](https://www.sutterhealth.org/research-search-results?verticalKey=research_publication) ```json {"@context":"https://schema.org","@type":"MedicalOrganization","name":"Sutter Health","@id":"https://www.sutterhealth.org/#medicalorganization","url":"https://www.sutterhealth.org","logo":"https://www.sutterhealth.org/static/images/sutter-health-logo.svg","address":{"@type":"PostalAddress","streetAddress":"2200 River Plaza Drive","addressLocality":"Sacramento","addressRegion":"CA","postalCode":"95833","addressCountry":"US"},"areaServed":{"@type":"AdministrativeArea","name":"California"},"sameAs":["https://www.facebook.com/SutterHealth","https://www.instagram.com/sutterhealth/","https://x.com/SutterHealth","https://www.youtube.com/sutterhealth","https://www.linkedin.com/company/sutter-health","https://en.wikipedia.org/wiki/Sutter_Health"]} ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","@id":"https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749#breadcrumblist","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.sutterhealth.org/"},{"@type":"ListItem","position":2,"name":"Research","item":"https://www.sutterhealth.org/research"},{"@type":"ListItem","position":3,"name":"Research publications","item":"https://www.sutterhealth.org/research/publications"},{"@type":"ListItem","position":4,"name":"Using FRAX in large healthcare delivery system"}]} ``` ```json {"@context":"https://schema.org","@type":"MedicalScholarlyArticle","@id":"https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749#medicalscholarlyarticle","name":"Methods for assessing fracture risk prediction models: experience with FRAX in a large integrated health care delivery system.","headline":"Using FRAX in large healthcare delivery system","abstract":"Area under the receiver operating characteristics (AUROC) curve is often used to evaluate risk models. However, reclassification tests provide an alternative assessment of model performance. We performed both evaluations on results from FRAX (World Health Organization Collaborating Centre for Metabolic Bone Diseases, University of Sheffield, UK), a fracture risk tool, using Kaiser Permanente Northern California women older than 50yr with bone mineral density (BMD) measured during 1997-2003. We compared FRAX performance with and without BMD in the model. Among 94,489 women with mean follow-up of 6.6yr, 1579 (1.7%) sustained a hip fracture. Overall, AUROCs were 0.83 and 0.84 for FRAX without and with BMD, suggesting that BMD did not contribute to model performance. AUROC decreased with increasing age, and BMD contributed significantly to higher AUROC among those aged 70yr and older. Using an 81% sensitivity threshold (optimum level from receiver operating characteristic curve, corresponding to 1.2% cutoff), 35% of those categorized above were reassigned below when BMD was added. In contrast, only 10% of those categorized below were reassigned to the higher risk category when BMD was added. The net reclassification improvement was 5.5% (p\u003c0.01). Two versions of this risk tool have similar AUROCs, but alternative assessments indicate that addition of BMD improves performance. Multiple methods should be used to evaluate risk tool performance with less reliance on AUROC alone.","description":"Multiple methods should be used to evaluate risk tool performance with less reliance on area under the receiver operating characteristics alone.","keywords":["Biostatistics","Musculoskeletal Disorders","Women's Health"],"datePublished":"2011-10-01","url":"https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749","inLanguage":"en-US","identifier":[{"@type":"PropertyValue","propertyID":"PMID","value":"21958955"},{"@type":"PropertyValue","propertyID":"DOI","value":"10.1016/j.jocd.2011.06.006"}],"sameAs":["https://pubmed.ncbi.nlm.nih.gov/21958955/","https://doi.org/10.1016/j.jocd.2011.06.006"],"isPartOf":{"@type":"Periodical","name":"J Clin Densitom"},"author":[{"@type":"Person","name":"Pressman AR"},{"@type":"Person","name":"Lo JC"},{"@type":"Person","name":"Chandra M"},{"@type":"Person","name":"Ettinger B"}],"sourceOrganization":{"@type":"MedicalOrganization","name":"Palo Alto Medical Foundation Research Institute"},"about":[{"@type":"DefinedTerm","name":"Biostatistics","url":"https://www.sutterhealth.org/research-search-results?facets=%5B%7B%22i%22%3A%22c_researchTopics.name%22%2C%22v%22%3A%5B%22Biostatistics%22%5D%7D%5D&verticalKey=research_publication"},{"@type":"DefinedTerm","name":"Musculoskeletal Disorders","url":"https://www.sutterhealth.org/research-search-results?facets=%5B%7B%22i%22%3A%22c_researchTopics.name%22%2C%22v%22%3A%5B%22Musculoskeletal+Disorders%22%5D%7D%5D&verticalKey=research_publication"},{"@type":"DefinedTerm","name":"Women's Health","url":"https://www.sutterhealth.org/research-search-results?facets=%5B%7B%22i%22%3A%22c_researchTopics.name%22%2C%22v%22%3A%5B%22Women%27s+Health%22%5D%7D%5D&verticalKey=research_publication"}],"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749","breadcrumb":{"@id":"https://www.sutterhealth.org/research/publications/using-frax-in-large-healthcare-delivery-system-1055312749#breadcrumblist"}}} ```