HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Submit an electronic Letter to the Editor about this paper View responses Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (60) Citing Articles via Google Scholar Google Scholar Articles by Bruns, D. E. Articles by Young, D. S. Search for Related Content PubMed PubMed Citation Articles by Bruns, D. E. Articles by Young, D. S. Related Collections Laboratory Management General Clinical Chemistry Clinical Immunology Pediatric Clinical Chemistry Nutrition Molecular Diagnostics and Genetics Evidence Based Laboratory Medicine and Test Utilization Hemostasis and Thrombosis Proteomics and Protein Markers Lipids, Lipoproteins, and Cardiovascular Risk Factors Drug Monitoring and Toxicology Hematology Endocrinology and Metabolism

Toward a Checklist for Reporting of Studies of Diagnostic Accuracy of Medical Tests

¹ Department of Pathology, University of Virginia School of Medicine, Charlottesville, VA 22908.

² Annals of Internal Medicine, 190 N. Independence Mall West, Philadelphia, PA 19106-1572.

³ Department of Clinical Biochemistry, Amager Hospital, Italiensvej 1, DK2300 Copenhagen, Denmark.

⁴ Department of Pathology and Laboratory Medicine, University of Pennsylvania, 3400 Spruce St., Philadelphia, PA 19104-4283.
^a Author for correspondence. Fax 1-904-979-7599; e-mail dbruns{at}virginia.edu

	Abstract

Top Abstract Introduction Checklist for Publications on... References

 
Background: "Diagnostic accuracy" refers to the ability of medical tests to provide accurate information about diagnosis, prognosis, risk of disease, and other clinical issues. Published reports on diagnostic accuracy of medical tests frequently fail to adhere to minimal clinical epidemiological standards, and such failures lead to overly optimistic assessments of evaluated tests. Our aim was to enumerate key items for inclusion in published reports on diagnostic accuracy, with a related aim of making the reports more useful for systematic reviews.

Methods: We examined published reports on shortcomings of studies of diagnostic accuracy. We prepared an initial draft of a checklist to address common errors and presented it at a meeting of editors. After incorporation of comments from editors, we published a revised version in Clinical Chemistry in 1997 for comment from readers. One of us (E.M.) additionally circulated copies of the draft to methodologists and others interested in Evidence-Based Medicine. We updated the checklist with input from these sources.

Results: The updated document lists items for inclusion in the title, abstract, methods, results, and discussion sections of published papers. Depending on the nature of the study, the total number of items for a single paper is ~40. We invite comments on this document, which is freely available at Clinical Chemistry Online, where it can accessed readily from the Table of Contents for the July 2000 issue at www.clinchem.org/content/vol46/issue7/. Comments (eLetters) can be posted there for general reading.

Conclusions: The suggested revisions incorporated in this report appear useful to ensure inclusion of additional information that can allow assessment of the validity of the conclusions and the applicability of the study in other settings. The list can be useful in formulating guidelines and a checklist, which will require testing by authors and study of their effect on published studies of diagnostic accuracy.

	Introduction

Top Abstract Introduction Checklist for Publications on... References

 
Modern medical practice makes extensive use of laboratory tests, radiological imaging, and other technologies in decision-making regarding diagnosis, prognosis, monitoring, screening, and risk assessment. Despite remarkable technical testing methods and continuous improvement in analytical accuracy and precision of tests, the evaluation of the diagnostic accuracy of new tests often has received only modest attention. Thus, available tests may be analytically accurate but not provide reliable clinical information for diagnosis, risk stratification, and other clinical matters. Recent reports (1)(2) have documented that most published studies of diagnostic accuracy of clinical tests fail to meet (or fail to document adherence to) reasonable methodological standards. Moreover, the very recent study of Lijmer et al. (3) clearly demonstrated that failure to adhere to such standards produces overly optimistic estimates of diagnostic accuracy. This empirical evidence made clear the need for better clinical evaluations of medical tests (3).

In 1997, we proposed items to include in a checklist for reporting of studies of diagnostic accuracy (4). The proposal prompted valuable comments from numerous individuals, including statisticians, methodologists, editors, and researchers. Most comments have been positive, and all have been constructive. An eminent statistician correctly noted that most studies of diagnostic tests do not follow the model of a prospective clinical trial that appeared to guide the thinking in the proposed guidelines. He emphasized the need for "good information on exactly how patients were selected for inclusion in the study, and on clarity in reporting the results". We have tried to keep our eyes on those needs, and hope that further work will produce a document that best reflects the prevalent study designs in this field.

In the updated checklist below, we have incorporated the vast majority of suggestions that we received. The present authors, however, did not agree even among themselves on every item that we proposed to each other. The wording below thus reflects compromises and not a unanimous opinion of the authors. One concern is the term "diagnostic accuracy", which some feel will be confused with analytical accuracy. For better or worse, however, the term is well entrenched in the literature, and proposed alternatives seemed to have more problems than the present term.

"Diagnostic" is used here in the sense of a diagnostic system (5), with awareness that multiple factors (including medical history and physical examination) other than an evaluated test usually enter into a medical diagnosis or decision. In medicine, "diagnostic" tests find uses as aids for diagnosis, prognosis, monitoring, screening, and risk assessment, as mentioned above. Studies of "analytical" accuracy logically should precede studies of diagnostic accuracy; guidelines for such studies have been published [e.g., see Ref. (6)]. We do not address here how to report studies of nonanalytical sources of variation (including biological variation), which logically should be evaluated at an earlier stage of investigation, or studies of cost-effectiveness, which may require a distinct protocol.

	Checklist for Publications on Studies of Diagnostic Accuracy of Tests Used in Medical Case-Finding, Diagnosis, Prognosis, Risk Stratification, and Monitoring

Top Abstract Introduction Checklist for Publications on... References

 
Title: Identify the study as an evaluation of a test in screening, risk assessment, diagnosis, prognosis, or monitoring. Specify the disease or condition and the test(s) studied.

Abstract: Use a structured abstract.

Search terms: Use Medline-compatible terms for evaluated test or test(s), the disease or outcome (or both), the criterion (gold) standard test, and the type of study design. Include the terms diagnostic accuracy, sensitivity and specificity, and diagnosis.

Introduction: State the research question and why it came up. Cite a systematic review of the problem or provide a summary of how a search for prior studies was conducted. Summarize design of study to address unresolved issues. Indicate considerations regarding fallibility of criterion (gold) standard and effect on study design (e.g., use of double-reading of histologic sections). State the hypothesis and specific objectives of study.

Study Protocol and Methods:

1. Study design: prospective cohort, retrospective cohort, randomized clinical trial, etc.
   
2. Patient-care setting (e.g., ambulatory, general or referral practice, inpatient, volunteers) and summary of factors, especially other tests, that channeled patients to have the test under evaluation.
   
3. Criteria for (a) inclusion and (b) exclusion of subjects, especially regarding the results of any other tests and the criteria used in the interpretation of those tests. (c) Consent procedures and approvals of study.
   
4. Planned sample size and subgroup analyses; statistical power and resource considerations.
   
5. Methods to avoid spectrum bias¹ (e.g., consecutive series, statistically selected random sample, stratified random sample) and to define spectrum¹ of disease.
   
6. Methods (and references) for (a) evaluated test(s) and (b) criterion (gold) standard test(s). References should include studies that validate the analytical performance of the tests. When no such studies have been published, provide key information. When an outcome is used as the criterion standard, indicate duration and methods of follow-up.
   
7. Indicate the masking (blinding) of those performing (a) evaluated test(s) and (b) criterion standard test(s) to avoid reviewer bias.² When multiple tests are to be evaluated, indicate whether the performance of each was without knowledge of results of the other(s).
   
8. Methods to avoid verification bias³ (usually by application of criterion standard to all subjects) or to deal with its consequences.
   
9. Methods (and references) for statistical analysis including steps to deal with (a) potential for diagnostic accuracy to be overestimated when diagnostic rules are constructed by use of statistical modeling or by examination of more than one cutoff value for continuous variables, (b) repeated or serial measures, and (c) outliers.
   
10. Cutoffs used for quantitative tests and how they were determined; subjective criteria for qualitative tests.
    
11. For studies of prognostic tests, indicate whether the criterion standard or the evaluated tests influenced therapy (treatment paradox⁴ ).
    
12. Design features aimed at ensuring comparability with other studies.
    
13. Indicate that these guidelines were followed.
    

Results:

A. Study Subjects:

1. Inclusive dates of accrual of subjects.
   
2. Sample size achieved.
   
3. Numbers of subjects who were excluded, reasons for exclusions, and their timing.
   
4. Number of indeterminate test results and their use (if any) in further data analysis.
   
5. Demographics of subjects and their clinical characteristics to include the presence or absence of disease and spectrum of disease (e.g., summary of signs/symptoms and disease stage).
   
6. Deviations from study protocol (e.g., loss to follow-up) and reasons.
   

B. Study Findings:

1. Data on reproducibility of evaluated test (e.g., analytical CV at relevant concentrations and during an appropriate temporal interval; measures of intraobserver or interobserver variability as appropriate for the way the tests were performed in the study).
   
2. Appropriate tabulation of key results (e.g., 2 x 2 contingency table).
   
3. Measures of diagnostic accuracy of test(s) and confidence intervals (e.g., areas under ROC curves, sensitivity/specificity pairs, likelihood ratios). When the diagnostic marker is not naturally dichotomous, posttest probabilities should be emphasized over methods relying on arbitrary cutoffs such as sensitivity and specificity. When a diagnostic rule was derived from statistical modeling of multiple potential predictors or from examination of multiple cutoffs for continuous variables, provide an unbiased validation of diagnostic accuracy on a data set not used to develop the diagnostic rule, or use a resampling technique that corrects for overfitting.
   
4. Repository where original data may be obtained (e.g., for use in systematic reviews).
   

Discussion:

1. Internal validity of study: Assess imprecision and bias, e.g., fallibility of criterion standard, losses to follow-up, verification bias, and reviewer bias.
   
2. Applicability to other settings, e.g., spectrum of disease, demographics of subjects. Positive and negative predictive values for relevant populations when appropriate.
   
3. Adequacy of sample size for potential uses of the report.
   
4. Interpretation of study results in context of other reports, e.g., potential utility of test as an addition to existing tests and clinical observation or as a replacement for other tests.
   
5. For studies of prognostic tests, discuss treatment paradox when appropriate.
   
6. Conclusions and implications for further research and clinical practice.
   

	Acknowledgments

 
Supported in part by AACC, Clinical Chemistry, and the International Federation of Clinical Chemistry and Laboratory Medicine. This article is freely available at Clinical Chemistry Online (www.clinchem.org), where it can be accessed readily from the Table of Contents for the July 2000 issue at www.clinchem.org/content/vol46/issue7/. Comments (eLetters) can be posted there in the form of eLetters.

	Footnotes

 
¹ Spectrum bias: "Spectrum" refers to spectrum of disease. Bias may be introduced when, for example, only patients with advanced (presumably readily diagnosed) disease are included.

² Reviewer bias occurs when the performance or evaluation of a test is influenced by knowledge of the results of other tests.

³ Verification bias may occur when the criterion standard test is applied to only a subset of the patients or subjects.

⁴ Treatment paradox occurs when a positive test result successfully identifies patients at risk who then receive an effective therapy and thus have a favorable outcome despite the positive test result.

	References

Top Abstract Introduction Checklist for Publications on... References

 

1. Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA 1995;274:645-651. [Abstract]
2. Bogardus ST, Concato J, Feinstein AR. Clinical epidemiological quality in molecular genetic research. The need for methodological standards. JAMA 1999;281:1919-1926. [Abstract/Free Full Text]
3. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JHP, Bossuyt PMM. Empirical evidence of deign-related bias in studies of diagnostic tests. JAMA 1999;282:1061-1066. [Abstract/Free Full Text]
4. Bruns DE. The Clinical Chemist. Clin Chem 1997;43:2211-2212. [Free Full Text]
5. Swets JA. Measuring the accuracy of diagnostic systems. Science 1988;240:1285-1293. [Abstract/Free Full Text]
6. Anonymous. Information for authors. Clin Chem 2000;46:1-5.

The following articles in journals at HighWire Press have cited this article:

				A Caselli, V Spallone, G A Marfia, C Battista, C Pachatz, A Veves, and L Uccioli Validation of the nerve axon reflex for the assessment of small nerve fibre dysfunction J. Neurol. Neurosurg. Psychiatry, August 1, 2006; 77(8): 927 - 932. [Abstract] [Full Text] [PDF]

				A. Carroccio, L. Di Prima, G. Pirrone, C. Scalici, A. M. Florena, M. Gasparin, G. Tolazzi, A. Gucciardi, C. Sciume, and G. Iacono Anti-Transglutaminase Antibody Assay of the Culture Medium of Intestinal Biopsy Specimens Can Improve the Accuracy of Celiac Disease Diagnosis Clin. Chem., June 1, 2006; 52(6): 1175 - 1180. [Abstract] [Full Text] [PDF]

				C. P. Price, R. G. Newall, and J. C. Boyd Use of Protein:Creatinine Ratio Measurements on Random Urine Samples for Prediction of Significant Proteinuria: A Systematic Review Clin. Chem., September 1, 2005; 51(9): 1577 - 1586. [Abstract] [Full Text] [PDF]

				G. Soletormos, A. Semjonow, P. E.C. Sibley, R. Lamerz, P. H. Petersen, W. Albrecht, P. Bialk, M. Gion, F. Junker, H.-P. Schmid, et al. Biological Variation of Total Prostate-Specific Antigen: A Survey of Published Estimates and Consequences for Clinical Practice Clin. Chem., August 1, 2005; 51(8): 1342 - 1351. [Abstract] [Full Text] [PDF]

				N. Smidt, A. W. S. Rutjes, D. A. W. M. van der Windt, R. W. J. G. Ostelo, J. B. Reitsma, P. M. Bossuyt, L. M. Bouter, and H. C. W. de Vet Quality of Reporting of Diagnostic Accuracy Studies Radiology, May 1, 2005; 235(2): 347 - 353. [Abstract] [Full Text] [PDF]

				R. Rej Clinical Chemistry through Clinical Chemistry: A Journal Timeline Clin. Chem., December 1, 2004; 50(12): 2415 - 2458. [Abstract] [Full Text] [PDF]

				J.-C. Wasmuth, B. Grun, B. Terjung, A. Homrighausen, and U. Spengler ROC Analysis Comparison of Three Assays for the Detection of Antibodies against Double-Stranded DNA in Serum for the Diagnosis of Systemic Lupus Erythematosus Clin. Chem., November 1, 2004; 50(11): 2169 - 2171. [Full Text] [PDF]

				N. A. Obuchowski, M. L. Lieber, and F. H. Wians Jr. ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions Clin. Chem., July 1, 2004; 50(7): 1118 - 1125. [Abstract] [Full Text] [PDF]

				G. Zou From Diagnostic Accuracy to Accurate Diagnosis: Interpreting a Test Result with Confidence Med Decis Making, June 1, 2004; 24(3): 313 - 318. [Abstract] [PDF]

				P. M.M. Bossuyt The Quality of Reporting in Diagnostic Test Research: Getting Better, Still Not Optimal Clin. Chem., March 1, 2004; 50(3): 465 - 466. [Full Text] [PDF]

				B. Lumbreras-Lacarra, J. M. Ramos-Rincon, and I. Hernandez-Aguado Methodology in Diagnostic Laboratory Test Research in Clinical Chemistry and Clinical Chemistry and Laboratory Medicine Clin. Chem., March 1, 2004; 50(3): 530 - 536. [Abstract] [Full Text] [PDF]

				C. Stephan, S. Wesseling, T. Schink, and K. Jung Comparison of Eight Computer Programs for Receiver-Operating Characteristic Analysis Clin. Chem., March 1, 2003; 49(3): 433 - 439. [Abstract] [Full Text] [PDF]

				D. E. Bruns The STARD Initiative and the Reporting of Studies of Diagnostic Accuracy Clin. Chem., January 1, 2003; 49(1): 19 - 20. [Full Text] [PDF]

				I. E.A. Hoffman, I. Peene, E. M. Veys, and F. De Keyser Detection of Specific Antinuclear Reactivities in Patients with Negative Anti-nuclear Antibody Immunofluorescence Screening Tests Clin. Chem., December 1, 2002; 48(12): 2171 - 2176. [Abstract] [Full Text] [PDF]

				I. Ryden, P. Pahlsson, and S. Lindgren Diagnostic Accuracy of {alpha}1-Acid Glycoprotein Fucosylation for Liver Cirrhosis in Patients Undergoing Hepatic Biopsy Clin. Chem., December 1, 2002; 48(12): 2195 - 2201. [Abstract] [Full Text] [PDF]

				A. Carroccio, G. Vitale, L. Di Prima, N. Chifari, S. Napoli, C. La Russa, G. Gulotta, M. R. Averna, G. Montalto, S. Mansueto, et al. Comparison of Anti-Transglutaminase ELISAs and an Anti-Endomysial Antibody Assay in the Diagnosis of Celiac Disease: A Prospective Study Clin. Chem., September 1, 2002; 48(9): 1546 - 1550. [Abstract] [Full Text] [PDF]

				M. J. Khoury Commentary: Epidemiology and the Continuum from Genetic Research to Genetic Testing Am. J. Epidemiol., August 15, 2002; 156(4): 297 - 299. [Full Text] [PDF]

				M. Martinez, F. Espana, M. Royo, J. M. Alapont, S. Navarro, A. Estelles, J. Aznar, C. D. Vera, and J. F. Jimenez-Cruz The Proportion of Prostate-specific Antigen (PSA) Complexed to {alpha}1-Antichymotrypsin Improves the Discrimination between Prostate Cancer and Benign Prostatic Hyperplasia in Men with a Total PSA of 10 to 30 {micro}g/L Clin. Chem., August 1, 2002; 48(8): 1251 - 1256. [Abstract] [Full Text] [PDF]

				J.-C. Wasmuth, D. Oliver y Minarro, A. Homrighausen, L. Leifeld, J. K. Rockstroh, T. Sauerbruch, and U. Spengler Phospholipid Autoantibodies and the Antiphospholipid Antibody Syndrome: Diagnostic Accuracy of 23 Methods Studied by Variation in ROC Curves with Number of Clinical Manifestations Clin. Chem., July 1, 2002; 48(7): 1004 - 1010. [Abstract] [Full Text] [PDF]

				E. J. Lee, E.-J. Oh, Y.-J. Park, H. K. Lee, and B. K. Kim Soluble Transferrin Receptor (sTfR), Ferritin, and sTfR/Log Ferritin Index in Anemic Patients with Nonhematologic Malignancy and Chronic Inflammation Clin. Chem., July 1, 2002; 48(7): 1118 - 1121. [Full Text] [PDF]

				J. Ricny, L. Simkova, and A. Vincent Determination of Anti-Acetylcholine Receptor Antibodies in Myasthenic Patients by Use of Time-resolved Fluorescence Clin. Chem., March 1, 2002; 48(3): 549 - 554. [Abstract] [Full Text] [PDF]

				N. Ferre, J. Camps, E. Prats, E. Vilella, A. Paul, L. Figuera, and J. Joven Serum Paraoxonase Activity: A New Additional Test for the Improved Evaluation of Chronic Liver Damage Clin. Chem., February 1, 2002; 48(2): 261 - 268. [Abstract] [Full Text] [PDF]

				R. F. Anton, C. Dominick, M. Bigelow, and C. Westby Comparison of Bio-Rad %CDT TIA and CDTect as Laboratory Markers of Heavy Alcohol Use and Their Relationships with {gamma}-Glutamyltransferase Clin. Chem., October 1, 2001; 47(10): 1769 - 1775. [Abstract] [Full Text] [PDF]

				N. Hayashi, T. Kawamoto, M. Mukai, A. Morinobu, M. Koshiba, S. Kondo, S. Maekawa, and S. Kumagai Detection of Antinuclear Antibodies by Use of an Enzyme Immunoassay with Nuclear HEp-2 Cell Extract and Recombinant Antigens: Comparison with Immunofluorescence Assay in 307 Patients Clin. Chem., September 1, 2001; 47(9): 1649 - 1659. [Abstract] [Full Text] [PDF]

				D. E. Bruns Laboratory-related Outcomes in Healthcare Clin. Chem., August 1, 2001; 47(8): 1547 - 1552. [Abstract] [Full Text] [PDF]

				N. Bizzaro, G. Mazzanti, E. Tonutti, D. Villalta, and R. Tozzoli Diagnostic Accuracy of the Anti-Citrulline Antibody Assay for Rheumatoid Arthritis Clin. Chem., June 1, 2001; 47(6): 1089 - 1093. [Abstract] [Full Text] [PDF]

				G. Soletormos and F. Bach Cerebrospinal Fluid Cytokeratins for Diagnosis of Patients with Central Nervous System Metastases from Breast Cancer Clin. Chem., May 1, 2001; 47(5): 948 - 950. [Full Text] [PDF]

				R. Claeys, C. Groven, and F. K. Gorus Capillary Zone Electrophoresis of Proteins in Body Fluids: Comparison of Capillary and Agarose Gel Electrophoresis Clin. Chem., May 1, 2001; 47(5): 967 - 970. [Full Text] [PDF]

eLetters:

This Article Abstract Full Text (PDF) Submit an electronic Letter to the Editor about this paper View responses Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (60) Citing Articles via Google Scholar Google Scholar Articles by Bruns, D. E. Articles by Young, D. S. Search for Related Content PubMed PubMed Citation Articles by Bruns, D. E. Articles by Young, D. S. Related Collections Laboratory Management General Clinical Chemistry Clinical Immunology Pediatric Clinical Chemistry Nutrition Molecular Diagnostics and Genetics Evidence Based Laboratory Medicine and Test Utilization Hemostasis and Thrombosis Proteomics and Protein Markers Lipids, Lipoproteins, and Cardiovascular Risk Factors Drug Monitoring and Toxicology Hematology Endocrinology and Metabolism