Advertisement

Abstract

Diagnostic systems of several kinds are used to distinguish between two classes of events, essentially "signals" and "noise." For them, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy. It is the only measure available that is uninfluenced by decision biases and prior probabilities, and it places the performances of diverse systems on a common, easily interpreted scale. Representative values of this measure are reported here for systems in medical imaging, materials testing, weather forecasting, information retrieval, polygraph lie detection, and aptitude testing. Though the measure itself is sound, the values obtained from tests of diagnostic systems often require qualification because the test data on which they are based are of unsure quality. A common set of problems in testing is faced in all fields. How well these problems are handled, or can be handled in a given field, determines the degree of confidence that can be placed in a measured value of accuracy. Some fields fare much better than others.

Get full access to this article

View all available purchase options and get full access to this article.

Already a subscriber or AAAS Member? Log In

References

ABRAMS, H.L., COMPUTED-TOMOGRAPHY VERSUS ULTRASOUND OF THE ADRENAL-GLAND - A PROSPECTIVE-STUDY, RADIOLOGY 143: 121 (1982).
BACUS, J. W., ANALYTICAL AND QUANTITATIVE CYTOLOGY AND HISTOLOGY 6: 121 (1984).
Bacus, J. W., Application of Digital Image Processing Tehniques to Cytology Automation (1982).
BEGG, C.B., ASSESSMENT OF RADIOLOGIC TESTS - CONTROL OF BIAS AND OTHER DESIGN CONSIDERATIONS, RADIOLOGY 167: 565 (1988).
BENSHAKHAR, G, J APPL PSYCHOL 67: 701 (1986).
BERBAUM, K.S., TENTATIVE DIAGNOSES FACILITATE THE DETECTION OF DIVERSE LESIONS IN CHEST RADIOGRAPHS, INVESTIGATIVE RADIOLOGY 21: 532 (1986).
BOOKSTEIN, A, ANOMALOUS BEHAVIOR OF PRECISION IN SWETS MODEL, AND ITS RESOLUTION, JOURNAL OF DOCUMENTATION 30: 374 (1974).
BROOKES, B.C., MEASURES OF INFORMATION RETRIEVAL EFFECTIVENESS PROPOSED BY SWETS, JOURNAL OF DOCUMENTATION 24: 41 (1968).
Brunet, N., Tenth Conference on Probability and Statistics in Atmospheric Sciences: 12 (1987).
CLEVERDON, C, ASS SPEC LIBR INF BU 19: 6 (1967).
Cleverdon, C., Factors Determining the Performance of Indexing Systems: Test Results 2 (1966).
Egan, R. L., Mammography (1964).
GARLAND, L.H., ON THE SCIENTIFIC EVALUATION OF DIAGNOSTIC PROCEDURES - 34TH ANNUAL MEETING OF THE RADIOLOGICAL-SOCIETY-OF-NORTH-AMERICA, RADIOLOGY 52: 309 (1949).
GETTY, D.J., ENHANCED INTERPRETATION OF DIAGNOSTIC IMAGES, INVESTIGATIVE RADIOLOGY 23: 240 (1988).
GOHAGAN, J.K., ROC ANALYSIS OF MAMMOGRAPHY AND PALPATION FOR BREAST SCREENING, INVESTIGATIVE RADIOLOGY 19: 587 (1984).
GOIN, J.E., ANALYSIS OF MAMMOGRAPHY - A BLIND INTERPRETATION OF BCDDP RADIOGRAPHS, RADIOLOGY 148: 393 (1983).
GRAY, R, CONSTRUCTION OF RECEIVER OPERATING CHARACTERISTIC CURVES WHEN DISEASE VERIFICATION IS SUBJECT TO SELECTION BIAS, MEDICAL DECISION MAKING 4: 151 (1984).
Green, D. M., Signal Detection Theory and Psychophysics (1966).
Griffith, B., Key Papers in Information Science (1980).
HEINE, M.H., INVERSE RELATIONSHIP OF PRECISION AND RECALL IN TERMS OF SWETS MODEL, JOURNAL OF DOCUMENTATION 29: 81 (1973).
HEINE, M.H., MEASURES OF LANGUAGE EFFECTIVENESS AND SWETSIAN HYPOTHESES, JOURNAL OF DOCUMENTATION 31: 283 (1975).
HERRNSTEIN, R.J., TEACHING THINKING SKILLS, AMERICAN PSYCHOLOGIST 41: 1279 (1986).
Kochen, M., The Growth of Knowledge (1967).
KOSHLAND, D.E., RETROACTIVE PROPHETS, SCIENCE 238: 727 (1987).
Lusted, L. B., Introduction to Medical Decision Making (1968).
MACMAHON, H, DIGITAL RADIOGRAPHY OF SUBTLE PULMONARY ABNORMALITIES - AN ROC STUDY OF THE EFFECT OF PIXEL SIZE ON OBSERVER PERFORMANCE, RADIOLOGY 158: 21 (1986).
MASON, I, AUSTRALIAN METEOROLOGICAL MAGAZINE 30: 291 (1982).
Mason, I., Ninth Conference on Weather Forcasting and Analysis: 169 (1982).
MCCOY, M.C., SEVERE-STORM-FORECAST RESULTS FROM THE PROFS 1983 FORECAST EXPERIMENT, BULLETIN OF THE AMERICAN METEOROLOGICAL SOCIETY 67: 155 (1986).
McCoy, M. C., Ninth Conference on Probability and Statistics in Atmospheric Sciences: 423 (1985).
METZ, C.E., ROC METHODOLOGY IN RADIOLOGIC IMAGING, INVESTIGATIVE RADIOLOGY 21: 720 (1986).
PETERSON, W.W., TRANSACTIONS OF THE IRE PROFESSIONAL GROUP ON INFORMATION THEORY 4: 171 (1954).
RANSOHOFF, D.F., PROBLEMS OF SPECTRUM AND BIAS IN EVALUATING EFFICACY OF DIAGNOSTIC TESTS, NEW ENGLAND JOURNAL OF MEDICINE 299: 926 (1978).
REVESZ, G, THE EFFECT OF VERIFICATION ON THE ASSESSMENT OF IMAGING TECHNIQUES, INVESTIGATIVE RADIOLOGY 18: 194 (1983).
SALTON, G, COMPUTER EVALUATION OF INDEXING AND TEXT PROCESSING, JOURNAL OF THE ACM 15: 8 (1968).
Saracevic, T., Introduction to Information Sciencce (1970).
SAXE, L, AMERICAN PSYCHOLOGIST 40: 794 (1985).
Swanson, L., Armed Services Vocational Aptitude Battery, Forms 6 and 7, Validation Against School Performance in Navy Enlisted Schools, Navy Personnel Research and Development Center Technical Report 80-1 (1979).
SWETS, J.A., EFFECTIVENESS OF INFORMATION RETRIEVAL METHODS, AMERICAN DOCUMENTATION 20: 72 (1969).
Swets, J. A., Evaluation of Diagnostic Systems: Methods from Signal Detection Theory (1982).
SWETS, J.A., ROC ANALYSIS APPLIED TO THE EVALUATION OF MEDICAL IMAGING TECHNIQUES, INVESTIGATIVE RADIOLOGY 14: 109 (1979).
SWETS, J.A., ASSESSMENT OF NDT SYSTEMS .1. THE RELATIONSHIP OF TRUE AND FALSE DETECTIONS, MATERIALS EVALUATION 41: 1294 (1983).
SWETS, J.A., INDEXES OF DISCRIMINATION OR DIAGNOSTIC-ACCURACY - THEIR ROCS AND IMPLIED MODELS, PSYCHOLOGICAL BULLETIN 99: 100 (1986).
SWETS, J.A., FORM OF EMPIRICAL ROCS IN DISCRIMINATION AND DIAGNOSTIC TASKS - IMPLICATIONS FOR THEORY AND MEASUREMENT OF PERFORMANCE, PSYCHOLOGICAL BULLETIN 99: 181 (1986).
SWETS, J.A., IS THERE A SENSORY THRESHOLD - WHEN EFFECTS OF OBSERVERS RESPONSE CRITERION ARE ISOLATED, A SENSORY LIMITATION IS NOT EVIDENT, SCIENCE 134: 168 (1961).
SWETS, J.A., INFORMATION RETRIEVAL SYSTEMS, SCIENCE 141: 245 (1963).
SWETS, J.A., RELATIVE OPERATING CHARACTERISTIC IN PSYCHOLOGY, SCIENCE 182: 990 (1973).
SWETS, J.A., ASSESSMENT OF DIAGNOSTIC TECHNOLOGIES, SCIENCE 205: 753 (1979).
Swets, J. A., Signal Detection and Recognition by Human Obserervers (1964).
SZUCKO, J.J., STATISTICAL VERSUS CLINICAL LIE DETECTION, AMERICAN PSYCHOLOGIST 36: 488 (1981).
Williams, G., Ninth Conference on Probability and Statistics in Atmospheric Sciences: 214 (1985).
YERUSHALMY, J, THE ROLE OF DUAL READING IN MASS RADIOGRAPHY, AMERICAN REVIEW OF TUBERCULOSIS 61: 443 (1950).

Information & Authors

Information

Published In

View large Science cover image
Science
Volume 240 | Issue 4857
3 June 1988

Submission history

Published in print: 3 June 1988

Permissions

Request permissions for this article.

Authors

Affiliations

John A. Swets
BBN Laboratories Incorporated, Cambridge, MA 02238.

Metrics & Citations

Metrics

Article Usage
Altmetrics

Citations

Export citation

Select the format you want to export the citation of this publication.

Cited by

  1. Novel trophic interactions under climate change promote alpine plant coexistence, Science, 370, 6523, (1469-1473), (2021)./doi/10.1126/science.abd7015
    Abstract
  2. The Future of the Behavioral and Social Sciences, Science, 271, 5249, (611-612), (1996)./doi/10.1126/science.271.5249.611
    Abstract
Loading...

View Options

Check Access

Log in to view the full text

AAAS ID LOGIN

AAAS login provides access to Science for AAAS Members, and access to other journals in the Science family to users who have purchased individual subscriptions.

Log in via OpenAthens.
Log in via Shibboleth.
More options

Purchase digital access to this article

Download and print this article for your personal scholarly, research, and educational use.

Purchase this issue in print

Buy a single issue of Science for just $15 USD.

View options

PDF format

Download this article as a PDF file

Download PDF

Media

Figures

Multimedia

Tables

Share

Share

Share article link

Share on social media