In defense of the black box
Abstract
The science fiction writer Douglas Adams imagined the greatest computer ever built, Deep Thought, programmed to answer the deepest question ever asked: the Great Question of Life, the Universe, and Everything. After 7.5 million years of processing, Deep Thought revealed its answer: Forty-two (1). As artificial intelligence (AI) systems enter every sector of human endeavor—including science, engineering, and health—humanity is confronted by the same conundrum that Adams encapsulated so succinctly: What good is knowing the answer when it is unclear why it is the answer? What good is a black box?
Get full access to this article
View all available purchase options and get full access to this article.
Already a Subscriber?Sign In
References and Notes
1
D. Adams, Hitchhiker's Guide to the Galaxy (Harmony Books, 1980).
2
D. S. Char, N. H. Shah, D. Magnus, N. Engl. J. Med. 378, 981 (2018).
3
Regulation (EU) 2016/679 of the European Parliament and of the Council, 27 April 2016; https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN.
4
F. Doshi-Velez, B. Kim, arXiv:1702.08608v2 [stat.ML] (2017).
5
R. E. Nisbett, T. D. Wilson, Psychol. Rev. 84, 231 (1977).
6
J. I. Gold, M. N. Shadlen, Annu. Rev. Neurosci. 30, 535 (2007).
7
D. S. Evans, J. Econ. Perspect. 23, 37 (2009).
8
B. L. DeCost, B. Lei, T. Francis, E. A. Holm, Microsc. Microanal. 25, 21 (2019).
9
A. Jalalian et al., Clin. Imaging 37, 420 (2013).
10
N. Kalra, S. M. Paddock, Transport. Res. A 94, 182 (2016).
11
G. Marcus, arXiv:1801.00631 [cs.AI] (2018).
12
V. Gulshan et al., JAMA 316, 2402 (2016).
13
J. Krause et al., Ophthalmology 125, 1264 (2018).
14
R. Poplin et al., Nat. Biomed. Eng. 2, 158 (2018).
Information & Authors
Information
Published In
Science
Volume 364 | Issue 6435
5 April 2019
5 April 2019
Copyright
Copyright © 2019 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works.
This is an article distributed under the terms of the Science Journals Default License.
Submission history
Published in print: 5 April 2019
Acknowledgments
This work has been funded by U.S. National Science Foundation grants DMR-1507830 and CMMI-1826218 and by the Carnegie Mellon Manufacturing Futures Initiative through a grant from the Richard King Mellon Foundation.
Authors
Metrics & Citations
Metrics
Article Usage
Altmetrics
Citations
Export citation
Select the format you want to export the citation of this publication.
View Options
Get Access
Log in to view the full text
AAAS login provides access to Science for AAAS members, and access to other journals in the Science family to users who have purchased individual subscriptions.
- Become a AAAS Member
- Activate your Account
- Purchase Access to Other Journals in the Science Family
- Account Help
Log in via OpenAthens.
Log in via Shibboleth.
More options
Purchase digital access to this article
Download and print this article for your personal scholarly, research, and educational use.
Buy a single issue of Science for just $15 USD.
View options
PDF format
Download this article as a PDF file
Download PDF
Black-box in AI is not black box
Elizabeth A. Holm wrote an article entitled "In defense of the black box " (1). Holm of CMU is confused with intentional or ignorant black box problems. Many researchers don't know how to convert deep learning which is called black-box into an explainable decision tree. Geoffrey Everest Hinton, former faculty of CMU (1982–1987) has proposed an idea on how to eliminate the black-box problem (2). Hinton's algorithm, called the soft decision tree converted from deep learning has been implemented in several open source sites (3, 4). Not only in deep learning based on GPU computing, but also in ensemble methods based on CPU computing, the explainable decision tree function has been implemented in open source machine learning including scikit-learn. In other words, the black box problem in AI can be eliminated if we would like to do. The intentional black box in AI system is another issue.
References:
1. E. A. Holm, Science 05 Apr 2019: Vol. 364, Issue 6435, pp. 26-27
2. https://arxiv.org/abs/1711.09784
3. https://github.com/kimhc6028/soft-decision-tree
4. https://github.com/AaronX121/Soft-Decision-Tree
RE: In defense of the black box
Elizabeth Holm nods to Douglas Adams and his notional 'Deep Thought' computer in her defense of 'black box' computation. However, Professor Holm makes little effort to help users of 'black box' algorithms to distinguish between trustworthiness and truthiness. Even official regulation of commercial 'black box' implementations can fail (q. v., Boeing's 737 Max flight control software) if appropriate engineering choices are overridden by managerial demands.
At the very least, any and all 'black box' software should be distributed and utilized in a state of maximum transparency. This means the developer should provide software documentation as well as use cases. And the user, particularly for publication of results, should provide a full suite of parameter selection and analysis choices. In other words, scientists and engineers must be willing to provide the clarity that that the scientific method demands.
RE: In defense of the black box
Dear Colleagues,
As with many things, that esteemed philosopher of science, Richard Feynman, said it best. On numerous occasions, he remarked on how much fun it is to think about the puzzles and problems of science. Will AI take the fun out of thinking about the puzzles of science if we don't understand why the answer which the black box give us is the way it is?