Dissecting racial bias in an algorithm used to manage the health of populations
Racial bias in health algorithms
The U.S. health care system uses commercial algorithms to guide health decisions. Obermeyer et al. find evidence of racial bias in one widely used algorithm, such that Black patients assigned the same level of risk by the algorithm are sicker than White patients (see the Perspective by Benjamin). The authors estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. Bias occurs because the algorithm uses health costs as a proxy for health needs. Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients. Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care.
Abstract
Health systems rely on commercial prediction algorithms to identify and help patients with complex health needs. We show that a widely used algorithm, typical of this industry-wide approach and affecting millions of patients, exhibits significant racial bias: At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%. The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients. Thus, despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise. We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.
Get full access to this article
View all available purchase options and get full access to this article.
Already a Subscriber?Sign In
Supplementary Material
Summary
Materials and Methods
Figs. S1 to S5
Tables S1 to S4
Resources
File (aax2342_obermeyer_sm.pdf)
References and Notes
1
J. Angwin, J. Larson, S. Mattu, L. Kirchner, “Machine Bias,” ProPublica (23 May 2016); www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
2
S. Barocas, A. D. Selbst, Big data’s disparate impact. Calif. Law Rev. 104, 671 (2016).
3
A. Chouldechova, A. Roth, The frontiers of fairness in machine learning. arXiv:1810.08810 [cs.LG] (20 October 2018).
4
A. Datta, M. C. Tschantz, A. Datta, Automated experiments on ad privacy settings. Proc. Privacy Enhancing Technol. 2015, 92–112 (2015).
5
L. Sweeney, Discrimination in online ad delivery. Queue 11, 1–19 (2013).
6
M. Kay, C. Matuszek, S. A. Munson, in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (ACM, 2015), pp. 3819–3828.
7
B. F. Klare, M. J. Burge, J. C. Klontz, R. W. Vorder Bruegge, A. K. Jain, Face Recognition Performance: Role of Demographic Information. IEEE Trans. Inf. Forensics Security 7, 1789–1801 (2012).
8
J. Buolamwini, T. Gebru, in Proceedings of the Conference on Fairness, Accountability and Transparency (PMLR, 2018), pp. 77–91.
9
A. Caliskan, J. J. Bryson, A. Narayanan, Semantics derived automatically from language corpora contain human-like biases. Science 356, 183–186 (2017).
10
S. Corbett-Davies, S. Goel, The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv:1808.00023 [cs.CY] (31 July 2018).
11
M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, A. T. Kalai, Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. arXiv:1901.09451 [cs.IR] (27 January 2019).
12
M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, S. Venkatasubramanian, in Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2015), pp. 259–268.
13
J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, S. Mullainathan, Human decisions and machine predictions. Q. J. Econ. 133, 237–293 (2018).
14
C. S. Hong, A. L. Siegel, T. G. Ferris, Caring for high-need, high-cost patients: What makes for a successful care management program? Issue Brief (Commonwealth Fund) 19, 1–19 (2014).
15
N. McCall, J. Cromwell, C. Urato, “Evaluation of Medicare Care Management for High Cost Beneficiaries (CMHCB) Demonstration: Massachusetts General Hospital and Massachusetts General Physicians Organization (MGH)” (RTI International, 2010).
16
J. Hsu, M. Price, C. Vogeli, R. Brand, M. E. Chernew, S. K. Chaguturu, E. Weil, T. G. Ferris, Bending The Spending Curve By Altering Care Delivery Patterns: The Role Of Care Management Within A Pioneer ACO. Health Aff. 36, 876–884 (2017).
17
L. Nelson, “Lessons from Medicare’s demonstration projects on disease management and care coordination” (Working Paper 2012-01, Congressional Budget Office, 2012).
18
C. Vogeli, A. E. Shields, T. A. Lee, T. B. Gibson, W. D. Marder, K. B. Weiss, D. Blumenthal, Multiple chronic conditions: Prevalence, health consequences, and implications for quality, care management, and costs. J. Gen. Intern. Med. 22 (suppl. 3), 391–395 (2007).
19
D. W. Bates, S. Saria, L. Ohno-Machado, A. Shah, G. Escobar, Big data in health care: Using analytics to identify and manage high-risk and high-cost patients. Health Aff. 33, 1123–1131 (2014).
20
J. Kleinberg, J. Ludwig, S. Mullainathan, Z. Obermeyer, Prediction Policy Problems. Am. Econ. Rev. 105, 491–495 (2015).
21
G. Hileman, S. Steele, “Accuracy of claims-based risk scoring models” (Society of Actuaries, 2016).
22
J. Kleinberg, S. Mullainathan, M. Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv:1609.05807 [cs.LG] (19 September 2016).
23
A. Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data 5, 153–163 (2017).
24
V. de Groot, H. Beckerman, G. J. Lankhorst, L. M. Bouter, How to measure comorbidity. a critical review of available methods. J. Clin. Epidemiol. 56, 221–229 (2003).
25
J. J. Gagne, R. J. Glynn, J. Avorn, R. Levin, S. Schneeweiss, A combined comorbidity score predicted mortality in elderly patients better than existing scores. J. Clin. Epidemiol. 64, 749–759 (2011).
26
A. K. Parekh, M. B. Barton, The challenge of multiple comorbidity for the US health care system. JAMA 303, 1303–1304 (2010).
27
D. Ettehad, C. A. Emdin, A. Kiran, S. G. Anderson, T. Callender, J. Emberson, J. Chalmers, A. Rodgers, K. Rahimi, Blood pressure lowering for prevention of cardiovascular disease and death: a systematic review and meta-analysis. Lancet 387, 957–967 (2016).
28
K.-T. Khaw, N. Wareham, R. Luben, S. Bingham, S. Oakes, A. Welch, N. Day, Glycated haemoglobin, diabetes, and mortality in men in Norfolk cohort of European Prospective Investigation of Cancer and Nutrition (EPIC-Norfolk). BMJ 322, 15 (2001).
29
K. Fiscella, P. Franks, M. R. Gold, C. M. Clancy, Inequality in quality: Addressing socioeconomic, racial, and ethnic disparities in health care. JAMA 283, 2579–2584 (2000).
30
N. E. Adler, K. Newman, Socioeconomic disparities in health: Pathways and policies. Health Aff. 21, 60–76 (2002).
31
N. E. Adler, W. T. Boyce, M. A. Chesney, S. Folkman, S. L. Syme, Socioeconomic inequalities in health. No easy solution. JAMA 269, 3140–3145 (1993).
32
M. Alsan, O. Garrick, G. C. Graziani, “Does diversity matter for health? Experimental evidence from Oakland” (National Bureau of Economic Research, 2018).
33
K. Armstrong, K. L. Ravenell, S. McMurphy, M. Putt, Racial/ethnic differences in physician distrust in the United States. Am. J. Public Health 97, 1283–1289 (2007).
34
M. Alsan, M. Wanamaker, Tuskegee and the health of black men. Q. J. Econ. 133, 407–455 (2018).
35
M. van Ryn, J. Burke, The effect of patient race and socio-economic status on physicians’ perceptions of patients. Soc. Sci. Med. 50, 813–828 (2000).
36
K. M. Hoffman, S. Trawalter, J. R. Axt, M. N. Oliver, Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites. Proc. Natl. Acad. Sci. U.S.A. 113, 4296–4301 (2016).
37
J. J. Escarce, F. W. Puffer, “Black-White Differences in the Use of Medical Care by the Elderly: A Contemporary Analysis” in Racial and Ethnic Differences in the Health of Older Americans (National Academies Press, 1997), chap. 6; www.ncbi.nlm.nih.gov/books/NBK109841/.
38
S. Passi, S. Barocas, Problem Formulation and Fairness. arXiv:1901.02547 [cs.CY] (8 January 2019).
39
S. Mullainathan, Z. Obermeyer, Does Machine Learning Automate Moral Hazard and Error? Am. Econ. Rev. 107, 476–480 (2017).
40
K. E. Joynt Maddox, M. Reidhead, J. Hu, A. J. H. Kind, A. M. Zaslavsky, E. M. Nagasako, D. R. Nerenz, Adjusting for social risk factors impacts performance and penalties in the hospital readmissions reduction program. Health Serv. Res. 54, 327–336 (2019).
41
K. E. Joynt Maddox, M. Reidhead, A. C. Qi, D. R. Nerenz, Association of Stratification by Dual Enrollment Status With Financial Penalties in the Hospital Readmissions Reduction Program. JAMA Intern. Med. 179, 769–776 (2019).
42
K. Lum, W. Isaac, To predict and serve? Significance 13, 14–19 (2016).
43
I. Ajunwa, “The Paradox of Automation as Anti-Bias Intervention,” available at SSRN (2016); https://ssrn.com/abstract=2746078.
44
S. DellaVigna, M. Gentzkow, “Uniform pricing in US retail chains” (National Bureau of Economic Research, 2017).
45
C. A. Gomez-Uribe, N. Hunt, The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Manag. Inf. Syst. 6, 13 (2016).
46
Z. Obermeyer, S. Mullainathan, in Proceedings of the Conference on Fairness, Accountability, and Transparency (ACM, 2019), p. 89.
47
G. Weiss, L. T. Goodnough, Anemia of chronic disease. N. Engl. J. Med. 352, 1011–1023 (2005).
48
M. Tonelli, N. Wiebe, B. Culleton, A. House, C. Rabbat, M. Fok, F. McAlister, A. X. Garg, Chronic kidney disease and mortality risk: A systematic review. J. Am. Soc. Nephrol. 17, 2034–2047 (2006).
49
H. Ujiie, M. Kawasaki, Y. Suzuki, M. Kaibara, Influence of age and hematocrit on the coagulation of blood. J. Biorheol. 23, 111–114 (2009).
50
M. G. Silverman, B. A. Ference, K. Im, S. D. Wiviott, R. P. Giugliano, S. M. Grundy, E. Braunwald, M. S. Sabatine, Association Between Lowering LDL-C and Cardiovascular Risk Reduction Among Different Therapeutic Interventions: A Systematic Review and Meta-analysis. JAMA 316, 1289–1297 (2016).
51
B. Nowok, G. M. Raab, C. Dibben, synthpop: Bespoke creation of synthetic data in R. J. Stat. Softw. 74, 1–26 (2016).
Information & Authors
Information
Published In
Copyright
Copyright © 2019 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works.
This is an article distributed under the terms of the Science Journals Default License.
History
Received: 8 March 2019
Accepted: 4 October 2019
Acknowledgments
We thank S. Lakhtakia, Z. Li, K. Lin, and R. Mahadeshwar for research assistance and D. Buefort and E. Maher for data science expertise. Funding: This work was supported by a grant from the National Institute for Health Care Management Foundation. Author contributions: Z.O. and S.M. designed the study, obtained funding, and conducted the analyses. All authors contributed to reviewing findings and writing the manuscript. Competing interests: The analysis was completely independent: None of the authors had any contact with the algorithm’s manufacturer until after it was complete. No authors received compensation, in any form, from the manufacturer or have any commercial interests in the manufacturer or competing entities or products. There were no confidentiality agreements that limited reporting of the work or its results, no material transfer agreements, no oversight in the preparation of this article (besides ethical oversight from the approving IRB, which was based at a non-profit academic health system), and no formal relationship of any kind between any of the authors and the manufacturer. Data and materials availability: Because the data used in this analysis are protected health information, they cannot be made publicly available. We provide instead a synthetic dataset (using the R package synthpop) and all code necessary to reproduce our analyses at https://gitlab.com/labsysmed/dissecting-bias.
Authors
Funding Information
National Institute for Health Care Management Foundation:
RE: tautology in AI predictor
The inattentive reader will miss a crucial acknowledgement by Obermeyer et al(1) of a serious tautology inherent in the described analytic enterprise. It goes like this:
If you build a model to distribute future health care resources based upon billing of present consumption.
If African American have less access to resource either through functional denial with inadequate resources in their community or self-denial through a lifetime of trained hopelessness to access such services in a meaningful way
Then, future distributions directed by that model will give more to Whites independent of need.
To those to whom much has been given, more will be given.
Obermeyer correctly identifies that the programs enlisting these models "primarily work to prevent acute health decompensations that lead to catastrophic health care utilization". They are not focused on long term population health that would be more properly invoked 15 years prior to the "catastrophic sentinel event" and directed against prevalent risk factors of hypertension, diabetes, and elevated lipids poorly controlled in African American populations. The unfairness hinted at in the article is the focus on short term benefit of cost reduction in the thirty days post acute decompensation while ignoring the much broader health care needs of the population. Unfortunately, it is the goal that is deviant from health care justice. It is lifelong health care missing management(2) that is the true problem not a single poorly focused AI directed predictor.
Artificial Intelligence is not the problem. It is the master who has directed the AI to distribute resources using the wrong metric for the wrong goal.
Sincerely,
Eran Bellin, M.D.
Professor of Epidemiology and Population Health and Medicine
Albert Einstein College of Medicine Bronx, N.Y.
Vice President Clinical IT Research and Development
Montefiore Information Technology
1. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453.
2. Bellin E. Missing Management: Health-Care Analytic Discovery in a Learning Health System. South Carolina: Kindle direct publishing; 2019.
RE: Dissecting racial bias in an algorithm used to manage the health of populations
Recent news that an algorithm deciding care for 200 million patients was racially biased is highly concerning. This cautionary tale provides valuable lessons. A hasty rejection of Artificial Intelligence (AI) would be wrong though. AI has immense potential to improve health outcomes across society.
AI is already transforming health for the better. British researchers have built systems that can accurately diagnose breast cancer and detect eye conditions faster than ever before. In hospitals, we are using AI to proactively identify sepsis cases, thereby saving lives. Reducing readmissions and creating more accurate staffing forecasts is also saving money – essential for cash-strapped health services.
This story reminds companies to follow AI best practices. Set the right goals – the AI will follow your lead. Train the AI on your data and beware of bias. Avoid using third-party opaque "black-box" tools. Make sure decisions are easy to explain and justifiable. For sensitive areas, keep people involved. Despite the recent news, AI is ethical and trustworthy when used properly.
James Lawson, AI Evangelist, DataRobot