Beware explanations from AI in health care
Abstract
Artificial intelligence and machine learning (AI/ML) algorithms are increasingly developed in health care for diagnosis and treatment of a variety of medical conditions (1). However, despite the technical prowess of such systems, their adoption has been challenging, and whether and how much they will actually improve health care remains to be seen. A central reason for this is that the effectiveness of AI/ML-based medical devices depends largely on the behavioral characteristics of its users, who, for example, are often vulnerable to well-documented biases or algorithmic aversion (2). Many stakeholders increasingly identify the so-called black-box nature of predictive algorithms as the core source of users' skepticism, lack of trust, and slow uptake (3, 4). As a result, lawmakers have been moving in the direction of requiring the availability of explanations for black-box algorithmic decisions (5). Indeed, a near-consensus is emerging in favor of explainable AI/ML among academics, governments, and civil society groups. Many are drawn to this approach to harness the accuracy benefits of noninterpretable AI/ML such as deep learning or neural nets while also supporting transparency, trust, and adoption. We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable.
Get full access to this article
View all available purchase options and get full access to this article.
Already a Subscriber?Sign In
References and Notes
1
S. Benjamens, P. Dhunnoo, B. Meskó, NPJ Digit. Med. 3, 118 (2020).
2
B. J. Dietvorst, J. P. Simmons, C. Massey, Manage. Sci. 64, 1155 (2018).
3
A. F. Markus, J. A. Kors, P. R. Rijnbeek, J. Biomed. Inform. 113, 103655 (2021).
4
M. T. Ribeiro, S. Singh, C. Guestrin, in KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2016), pp. 1135–1144.
5
S. Gerke, T. Minssen, I. G. Cohen, in Artificial Intelligence in Healthcare, A. Bohr, K. Memarzadeh, Eds. (Elsevier, 2020), pp. 295–336.
6
Y. Lou, R. Caruana, J. Gehrke, in KDD '12: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2012), pp. 150–158.
7
Z. C. Lipton, ACM Queue 16, 1 (2018).
8
C. Rudin, Nat. Mach. Intell. 1, 206 (2019).
9
D. Martens, F. Provost, Manage. Inf. Syst. Q. 38, 73 (2014).
10
S. Wachter, B. Mittelstadt, C. Russell, Harv. J. Law Technol. 31, 841 (2018).
11
R. M. Hamm, S. L. Smith, J. Fam. Pract. 47, 44 (1998).
12
S. Gerke, B. Babic, T. Evgeniou, I. G. Cohen, NPJ Digit. Med. 3, 53 (2020).
13
U. J. Muehlematter, P. Daniore, K. N. Vokinger, Lancet Digit. Health 3, e195 (2021).
14
U. Guillen, H. Kirpalani, Semin. Fetal Neonatal Med. 23, 25 (2018).
15
Biofourmis, RhythmAnalytics (2020); www.biofourmis.com/solutions/.
Information & Authors
Information
Published In
Science
Volume 373 | Issue 6552
16 July 2021
16 July 2021
Copyright
Copyright © 2021, American Association for the Advancement of Science.
This is an article distributed under the terms of the Science Journals Default License.
Submission history
Published in print: 16 July 2021
Acknowledgments
We thank S. Wachter for feedback on an earlier version of this manuscript. All authors contributed equally to the analysis and drafting of the paper. Funding: S.G. and I.G.C. were supported by a grant from the Collaborative Research Program for Biomedical Innovation Law, a scientifically independent collaborative research program supported by a Novo Nordisk Foundation grant (NNF17SA0027784). I.G.C. was also supported by Diagnosing in the Home: The Ethical, Legal, and Regulatory Challenges and Opportunities of Digital Home Health, a grant from the Gordon and Betty Moore Foundation (grant agreement number 9974). Competing interests: S.G. is a member of the Advisory Group–Academic of the American Board of Artificial Intelligence in Medicine. I.G.C. serves as a bioethics consultant for Otsuka on their Abilify MyCite product. I.G.C. is a member of the Illumina ethics advisory board. I.G.C. serves as an ethics consultant for Dawnlight. The authors declare no other competing interests.
Authors
Metrics & Citations
Metrics
Article Usage
Altmetrics
Citations
Export citation
Select the format you want to export the citation of this publication.
View Options
Get Access
Log in to view the full text
AAAS login provides access to Science for AAAS members, and access to other journals in the Science family to users who have purchased individual subscriptions.
- Become a AAAS Member
- Activate your Account
- Purchase Access to Other Journals in the Science Family
- Account Help
Log in via OpenAthens.
Log in via Shibboleth.
More options
Purchase digital access to this article
Download and print this article for your personal scholarly, research, and educational use.
Buy a single issue of Science for just $15 USD.
View options
PDF format
Download this article as a PDF file
Download PDF
RE: Beware explanations from AI in health care
<p>To the Editor: <br />Babic et al. (1) recognize the emerging consensus favoring explainable artificial intelligence (AI) over cryptic (black-box) AI. We agree that it is detrimental to require transparency in AI if the system requires post hoc rationalizations of black-box diagnoses. We offer an alternate path for AI that both improves transparency and enables gains in system performance.</p> <p>This emerging image processing technique fuses visible structure information with black-box AI parameters in the classifier network to improve diagnostic accuracy for images in various diagnostic classes (2-5).</p> <p>If conventional image processing techniques can recognize a structure, and the diagnostic system allows the fusion of AI information with this structure, the output can obtain higher diagnostic accuracy and explain the correct diagnosis. The fusion concept can be extended to any additional structural data or clinical metadata that the black-box AI system lacks, such as a dimple sign for a benign dermatofibroma. Fusion of the AI information with the clinical sign again offers a boost in diagnostic accuracy and transparency. Such clinical signs are beneficial for a so-called long-tailed diagnosis such as dermatofibroma, which lacks sufficient training examples (6).</p> <p>The optimal role for AI in the diagnostic process is to serve as a diagnostic adjunct that assists but does not replace the clinician by providing supportive information for diagnostic assessment. The clinician is responsible for the care of the patient, not the AI software.</p> <p>The fusion of structure identification with AI offers a significant advantage in diagnostic performance for atypical or uncommon diagnostic subtypes that clinicians and AI systems have difficulty identifying, such as amelanotic melanoma (7, 8). Fusion of vessel structure parameters with AI techniques improves diagnostic performance for melanoma (2). Ultimately, the bridge between AI and the clinician requires identifying all critical information, including recognizable structures and metadata, thereby improving the final diagnostic accuracy.</p> <p> </p> <p><strong>REFERENCES AND NOTES</strong><br />1. B. Babic et al., Science 37, 284 (2021). <br />2. J. Hagerty et al., IEEE J. Biomed. Health. Inf. 23, 1385 (2019). <br />3. S. Li et al.,Phys. Med. Biol. 64, 175012 (2019). <br />4. N. Antropova et al., Med. Phys. 44, 162 (2017). <br />5. P. Guo et al., IEEE J. Biomed. Health. Inf. 20, 1595 (2016). <br />6. International Skin Imaging Collaboration; https://www.isic-archive.com/#!/topWithHeader/wideContentTop/main. <br />7. W. V. Stoecker, W. Stolz, Arch Dermatol. 144, 1207 (2008). <br />8. S. W. Menzies et al., Arch Dermatol. 144, 1120 (2008).</p>