HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

Electronic Letters to:

Evidence-based Laboratory Medicine and Test Utilization: Carsten Stephan, Sebastian Wesseling, Tania Schink, and Klaus Jung Comparison of Eight Computer Programs for Receiver-Operating Characteristic Analysis Clin Chem 2003; 49: 433-439 [Abstract] [Full text] [PDF]

eLetters: Submit a response to this article

Electronic letters published:

Comparsion of ROC computer programs

Andrew KRAMAR, David Faraggi, Antoine Fortuné, Ben Reiser   (21 March 2003)

Comparison of programs for ROC analysis

Frank Schoonjans   (24 March 2003)

Comparsion of ROC computer programs 21 March 2003
Andrew KRAMAR, Biostatistician CRLC Val d'Aurelle, Montpellier, France, David Faraggi, Antoine Fortuné, Ben Reiser Send letter to journal: Re: Comparsion of ROC computer programs akramar{at}valdorel.fnclcc.fr Andrew KRAMAR, et al.	Eight computer programs for ROC analysis (1) have been compared for several criteria: data input (10%), data output (15%), correctness (40%), completeness (20%), software comfort (10%) and manual (5%). Since our mROC program (2) obtained the lowest score, we feel it necessary to comment on these weights since not enough detail is provided. First of all, all programs obtained the maximum score for correctness, however, we can see from the examples in table 4 that the results are not the same. Only MedCalc and mROC give “correct” non-symmetrical confidence intervals, although mROC is slightly more conservative. All the other programs use symmetric normal approximations to the confidence intervals, yet non- parametric methods are used for all the other calculations. As far as completeness is concerned, we feel it unjustified to attribute the lowest score to mROC, since the reason that mROC was developed in the first place was to be able to handle correlated multivariate data appropriately by selecting the best linear combination of markers which maximizes the area under the curve, by first transforming the data into a gaussian distribution. All other programs only allow two-way comparisons, which will become very impractical and difficult to summarize once there are more than 4 diagnostic tests to be compared. Also, it should be remembered that there is a direct relationship between p-values obtained by a comparison of 2 AUC’s and their respective confidence intervals, which provide more information. As far as data input is concerned, we feel that any program using ROC methodology should be based on valid data. If clinicians will need to “correct” data within one of these programs, this means that there has been a neglect in the data management process of the study, and it is not within an analysis program that these corrections should be performed. By doing so, suspicions will be raised as to why data needs to be “modified” or “deleted”. Every computer program should have it’s own specificity and may need to be used by persons sensitive to the requirements of assessing and comparing the diagnostic validity of correlated laboratory tests within a multivariate environment. References 1. Stephan C, Wesseling S, Schink T, Jung K. Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem 2003;49:433-439. 2. Kramar A, Faraggi D, Fortuné A, Reiser B. mROC: a computer program for combining tumour markers in predicting disease states. Comput Methods Programs Biomed 2001, 66:199-207.
Comparison of programs for ROC analysis 24 March 2003
Frank Schoonjans University Hospital Gent, Dept. Endocrinology, Belgium Send letter to journal: Re: Comparison of programs for ROC analysis frank.schoonjans{at}rug.ac.be Frank Schoonjans	The authors of the paper comparing 8 computer programs for ROC analysis (1) seem to have missed a difference in the methods used by the different programs to calculate the confidence interval (CI) of the area under the ROC curve (AUC). In table 4 of this publication, one can distinguish 2 groups of programs: the first group containing MedCalc and mROC, who have a more narrow and non-symmetrical CI, and a second group of programs who give a symmetrical CI. The reason for this is that MedCalc calculates the CI using the Binomial distribution which is a more precise method than the method based on the Normal distribution used by the other programs (2). mROC calculates the CI of the AUC using the method of Mee (3) yielding a comparable CI as when using the Binomial distribution method. This difference in methodology may be less important for the CI of the example AUC of 0.702, but the difference becomes more important, and the Normal distribution method more inappropriate, when the AUC approaches 1. In addition, GraphROC is praised for its ability to compare ROC curves at a certain sensitivity or specificity cutoff. However, statistical testing at specific sensitivity or specificity values is known to be impossible in most cases using non-parametric methods (4), which GraphROC claims to use. We therefore suggest that the methodology used by GraphROC may be unsound. References 1. Stephan C, Wesseling S, Schink T, Jung K. Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem, 2003;49:433-9. 2. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 1982;143:29-36. 3. Mee RW. Confidence intervals for probabilities and tolerance regions based on a generalization of the Mann-Whitney statistic. J Amer Statist Assn, 1990;85:793-800. 4. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical chemistry. Clin Chem, 1993;39:561 -77.