Clinical Chemistry 43: 1838-1842, 1997;
(Clinical Chemistry. 1997;43:1838-1842.)
© 1997 American Association for Clinical Chemistry, Inc.
Receiver operating characteristic plots to evaluate Guthrie, Wallac, and Isolab phenylalanine kit performance for newborn phenylketonuria screening
Stephen T. Wanga,
Sam Pizzolato and
Helen P. Demshar
Laboratory Services Branch, Ontario Ministry of Health, Etobicoke, Ontario, Canada.
a Author for correspondence: Chemistry Section, Laboratory Services Branch, Ontario Ministry of Health, 81 Resources Rd., Etobicoke, Ontario, Canada M9P 3T1, or Box 9000, Terminal A, Toronto, Ontario, Canada M5W 1R5. Fax 416-235-6281.
 |
Abstract
|
|---|
We used ROC plots to evaluate the clinical performance of the Guthrie,
Wallac, and Isolab assays for newborn phenylketonuria (PKU) screening
and assessed the screening discriminatory power of these three assays
by the area under the ROC plot, Youden's J index, and the likelihood
ratio. The use of these plots not only allows us to pinpoint the exact
cutoff value in screening, but also provides a direct comparison of
these three different assays in clinical outcome performance. The
optimum cutoff for the newborn PKU screening is a blood phenylalanine
concentration of 0.30, 0.27, and 0.18 mmol/L for the Guthrie, Wallac,
and Isolab assays, respectively. We conclude that the Wallac and Isolab
kits, like the Guthrie assay, are suitable for newborn PKU
screening.
|
Introduction
|
|---|
Newborn screening of phenylketonuria
(PKU)1
with a bacterial inhibition assay (BIA) on a filter-paper
blood spot was established in Ontario in 1965. The BIA was developed by
Guthrie and Susi (1) and is still used worldwide. The
assay uses media formulated to grow Bacillus subtilis in an
amount that is proportional to the concentration of phenylalanine in
the blood spot. Recent development of quantitative determination of
phenylalanine with the Wallac Quantase phenylalanine screening assay
kit (2)(3) and the Isolab phenylalanine test
kit has provided other methods for newborn PKU screening. The Wallac
Quantase kit uses phenylalanine dehydrogenase to catalyze the
NAD+-dependent oxidative deamination of phenylalanine, and
the NADH product is measured colorimetrically. The Isolab phenylalanine
test kit is a modification of a fluorometric procedure involving a
ninhydrin reaction, as published by McCaman and Robins
(4). This procedure is based on the enhancement of the
fluorescence of a phenylalanine–ninhydrin reaction product by a
dipeptide and the copper reagent in the succinate buffer solution. The
method measures phenylalanine quantitatively in the presence of the
other amino acids. Both the Wallac and Isolab kits offer the potential
of an automated system. We evaluated the clinical performance of these
three different assay procedures for newborn PKU screening with ROC
plots. The ROC plot has become popular in recent years for evaluating
the discriminatory power of a test (5)(6)(7)(8)(9). The ROC plot
displays graphically the relationship between the true-positive rate
(TPR = sensitivity) and the false-positive rate (FPR = 1
- specificity) over all possible decision values. The decision value
is the variable test value that is used to distinguish apparently
healthy patients from diseased patients. We assessed the screening
accuracy of these three assay systems by the area under the ROC plot
(5), Youden's J index (10), and the
likelihood ratio (LR) (11).
|
Materials and Methods
|
|---|
materials
The Wallac Quantase phenylalanine kits, a DELFIA plate punch
(Model 1296-031D double-punch), a Multiscan microtiter plate reader
with filters of 570 and 690 nm, and Multicalc Data Management Software
were obtained from Wallac Canada. Millipore multiscreen special
microfiltration plates were obtained from Millipore. The dried blood
spot calibrators and controls on Schleicher & Schuell Type 903 Lot W932
paper were also supplied by Wallac Canada. An Isolab
NCSTM Neonatal Chemistry System, which includes a
fluorometer, a computer-controlled sample processor, and phenylalanine
reagent kits, was supplied by Isolab Inc.
All newborn blood specimens collected on Schleicher & Schuell Type 903
Lot W921 paper were obtained from our Newborn Screening Program. The
same Guthrie cards for 1076 apparently healthy and 21 diseased patient
samples were punched and analyzed with the Guthrie BIA, Wallac, and
Isolab kits. The individual diseased patient was diagnosed with
hyperphenylalaninemia, hospitalized for several laboratory tests, and
assessed by our physician consultants as having a definitive
classification of PKU and undergoing dietary management. External
quality-control specimens 421–424 from the CDC were used in all three
testing assays.
methods
Wallac Quantase phenylalanine kit.
The principle of this test
kit is a quantitative determination of phenylalanine in a blood spot
based on the use of phenylalanine dehydrogenase enzyme and its
colorimetric measurement. The assay procedures were performed following
the insert protocol of the company's kit. Two 3.2-mm dried blood spots
punched from calibrators, controls, and patient specimens were
extracted with 60 µL of trichloroacetic acid (30 g/L) in a well of
the Millipore microfilter plate. The plate was shaken for 30 min, and
the extracts were transferred to a microtiter plate by a multiscreen
Vacuum Manifold. The enzyme/coenzyme reagent was then added to each
well. After a 30-min shaking incubation, 100 µL of color reagent was
added. The absorbance was read bichromatically at 570/690 nm, at the
end of color development for 2–5 min. A complete assay run of three
microplates takes about 2.7 h.
Isolab phenylalanine kit.
The principle of this test kit is
based on a modification of the fluorometric procedure published by
McCaman and Robins (4), with enhancement of the
fluorescence of a phenylalanine–ninhydrin product by the
L-leucyl-L-alanine dipeptide. In the assay
procedure a 3.2-mm dried blood spot was punched into a microtitration
well, 15 µL of extraction solution was added to each well by the
automated Isolab Sampler Processor, and the sample was incubated at
37 °C for 30 min. Distilled-deionized water (40 µL) was added and
mixed, and 25 µL of the contents was transferred to the corresponding
plate by a Sampler Processor. The microplate was incubated for 2 h
at 37 °C after addition of 50 µL of the PKU reagent. At the end of
a 2-h incubation, 200 µL of copper reagent was added and allowed to
react for 45 min at room temperature before the microplate was read
with an Isolab Fluorescent Plate Reader. A complete assay of three
microplates takes about 4.5 h.
Guthrie BIA test.
The Guthrie BIA test is performed as the
procedure published by Guthrie and Susi (1) with PKU test
agar and B. subtilis (ATCC 6633) Spore Suspension 2. The
normal blood phenylalanine values are <0.24 mmol/L, and newborn
infants with screening values of 0.24 mmol/L or higher are tested. The
Guthrie test requires overnight incubation.
ROC plot.
The area under the ROC curve was estimated from the
Metz
program.2
The LR is defined as the probability (P) of
obtaining a positive test result in a patient with disease divided by
the probability of obtaining a positive test in a patient without the
disease [LR = P/(1 - P) or
P = LR/(LR + 1)]. A good LR is usually >2, which
corresponds to the probability of having the disease being >66.7%.
The LR (11) corresponds to the slope or tangent of a
single selected decision value on the ROC plot and is simply calculated
from the ratio of the TPR to the FPR, i.e., LR = 
TPR/FPR.
Youden's J index (10) is a simple approach of minimizing
error, equivalent to maximizing the sum of the sensitivity and
specificity; it is calculated as J = specificity +
sensitivity - 1, i.e., TPR - FPR. The index ranges from 0
for a worthless test to 1 for the perfect test.
|
Results
|
|---|
The TPR and FPR (%) for various blood phenylalanine
concentrations of the Guthrie, Wallac, and Isolab assays are listed in
Table 1
. The ROC areas of these three assays are 0.9982, 0.9972, and
0.9968, respectively. The same total number of 1076 apparently healthy
patients and 21 diseased patients used in all three assays are also
indicated in Table 1
. The ROC plots of TPR vs FPR for the assays are
shown in Fig. 1
. An expanded scale of Fig. 1
with the FPR to 2.5% is shown in
Fig. 1A
.
View larger version (15K):
[in this window]
[in a new window]
|
Figure 1. The ROC plots for the Guthrie, Wallac, and Isolab assays.
A is an expanded ROC plot of the inset with a FPR
to 2.5%;  , Guthrie; X, Wallac; •, Isolab.
|
|
Youden's J index plot for the three assays is shown in Fig. 2
. Youden's J index plot is to obtain the maximum index at the
selected decision values. The best index is 98.6%, 98.7%, and 98.5%
at the blood phenylalanine concentrations of 0.30, 0.27, and 0.18
mmol/L for the Guthrie, Wallac, and Isolab assays, respectively.
View larger version (18K):
[in this window]
[in a new window]
|
Figure 2. Plots of TPRs minus FPRs expressed as Youden's J index
for all three kits at various blood phenylalanine concentrations.
, Guthrie; X, Wallac; •, Isolab.
|
|
The LR plot of various phenylalanine concentrations of all three assays
is shown in Fig. 3
. The LR of these three assays shows the identical optimum
cutoff value of the blood phenylalanine concentrations as indicated in
Youden's J index at 0.30, 0.27, and 0.18 mmol/L. At these optimum
cutoff values, the LR is 10, 26, and 38 for the Guthrie, Wallac, and
Isolab assays, respectively. It indicates the probability of having PKU
is 90.9%, 96.3%, and 97.4% at the above-mentioned optimum cutoff
values for the three assays.
View larger version (10K):
[in this window]
[in a new window]
|
Figure 3. The LR for various blood phenylalanine concentrations for
all three assays.
, Guthrie; X, Wallac; •, Isolab.
|
|
|
Discussion
|
|---|
The ROC plot is a convenient way of expressing how well a test
performs clinically. In newborn screening the ROC plot is particularly
useful in evaluating the discriminatory ability of a test in displaying
the error rates for selecting various cutoff points. For newborn
screening, it is generally preferred to have a higher sensitivity test
at the cost of many false-positives (low specificity), which could then
be eliminated by an additional test as pointed out by Galen and Gambino
(12). Applying this principle to the ROC plot, we obtained
a sensitivity (TPR) of 100% for all three assays at cutoff points of
phenylalanine concentrations of 0.30, 0.27, and 0.18 mmol/L (Table 1
)
for the Guthrie, Wallac, and Isolab assays, respectively. At these
cutoff points with 100% for TPR, the FPR for the Guthrie, Wallac, and
Isolab assays are very close at 1.4%, 1.3%, and 1.5%, respectively.
Therefore, all three assays may have similar performances for PKU
screening. We had included 15 false-positive cases in the 1076
apparently healthy sample group for this evaluation. The actual FPR in
the routine newborn screening population is much lower than that. The
FPR for the Guthrie BIA is 0.07%, or ~100 cases per year in our
routine PKU Screening Program of 150 000 newborns per year.
Some conditions exist such that a test with a high specificity is
required at the expense of TPR when the disease is serious but is not
treatable, and the FPRs have tremendous financial and psychological
consequences for the patient, e.g., advanced carcinoma.
One indicator that quantifies the diagnostic accuracy of a laboratory
test is the area under the ROC plot. The area under the ROC plot would
be 1 for a perfect screening test. The ROC plot (Fig. 1
) and its area
(Table 1
) show that the Guthrie test has a slightly higher value
(0.9982) than the Wallac and Isolab kits, but the difference in area is
not statistically significant (P = 0.16). This is probably
the main reason that the Guthrie BIA test is still the most commonly
used screening procedure for detecting PKU in newborns throughout the
world, despite the fact it is a semiquantitative method. The areas
under the ROC plot of both the Wallac and the Isolab kits are close to
each other, with values of 0.9972 and 0.9968, respectively. Both the
Wallac and Isolab kits, however, have an advantage of easy automation
over the Guthrie test.
The area under the ROC plot provides only an overall index of test
accuracy, not a measure of test performance at the selected decision
values on the ROC plot. The ROC-related decision value plots, Youden's
J index and LR (Figs. 2
and 3
), are useful for selecting the optimum
decision value. Like the area under the ROC plot, Youden's J index and
the LR combine sensitivity and specificity into a single factor and are
extremely useful for assessing test performance. Unlike the
above-mentioned approach of Galen and Gambino (12) in
newborn screening, which gives overwhelming weight to TPR at the
expense of FPR, Youden's approach and LR give equal weight to TPR and
FPR. Comparing Youden's J index in Fig. 2
and LRs in Fig. 3
, we
reconfirmed that the optimum screening cutoff point of the blood
phenylalanine concentrations is identical to the one mentioned above
(Table 1
): 0.30, 0.27, and 0.18 mmol/L for the Guthrie, Wallac, and
Isolab assays, respectively. These different cutoff points may be a
result of the different procedures used in preparation of the
phenylalanine calibrators (13).
The optimum screening cutoff point of 0.30 mmol/L of the Guthrie test
is slightly higher than the normal value of 0.24 mmol/L used in our
routine PKU screening. We use the lower than optimum value in the
screening to ensure 100% sensitivity as discussed before. This should
not affect the result of the comparison because the same specimens were
applied to all three assays studied.
In conclusion, a ROC plot is particularly useful when comparing two
or more testing kits. A test with a curve that lies statistically
significantly above the curve of another will be clearly better. The
results of this study suggest that both the Wallac and Isolab kits are
suitable for PKU screening. The addition of the Wallac and the Isolab
kits for PKU screening enable the laboratory to have a choice of
screening assays, depending on the environment, manpower, and the
requirement of automation. From the ROC plot, Youden's J index, and
the LR studies, the optimum decision value of phenylalanine
concentration on newborn PKU screening can be assessed easily.
|
Acknowledgments
|
|---|
The instruments and kits provided by Wallac and Isolab are
gratefully acknowledged. We thank the Reference Bacteriology Laboratory
and the Chemistry Section, Ontario Ministry of Health, for the use of
their Guthrie results.
|
Footnotes
|
|---|
1 Nonstandard abbreviations: PKU, phenylketonuria; BIA, bacterial inhibition assay; TPR, true-positive rate; FPR, false-positive rate; LR, likelihood ratio. 
2 Metz programs. Charles E. Metz, Department of Radiology, MC2026, The University of Chicago Medical Center, 5841 South Maryland Ave., Chicago, IL 60637-1470. Fax 312-702-6779; Internet c-metz[@] u-chicago.edu].
|
References
|
|---|
-
Guthrie R, Susi A. A simple phenylalanine method for detecting phenylketonuria in large populations of newborn infants. Pediatrics 1963;32:338-343.
[Abstract/Free Full Text]
-
Elvers LH, Diependaal GAM, Blonk HJ, Loeber JG. Phenylketonuria screening using the Quantase phenylalanine kit in combination with a microfilter system and the dye Tartrazine. Screening 1995;3:209-223.
-
Dhondt JL, Paux E. Evaluation of an enzymatic, colorimetric method for the neonatal screening of phenylketonuria. Screening 1993;2:141-147.
-
McCaman MW, Robins E. Fluorometric method for the determination of phenylalanine in serum. J Clin Med 1962;59:885-890.
-
Swets JA. Measuring the accuracy of diagnostic systems. Science 1988;240:1285-1293.
[Abstract/Free Full Text]
-
Zweig MH, Campbell G. Receiver operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine [Review]. Clin Chem 1993;39:561-577.
[Abstract/Free Full Text]
-
Henderson AR. Assessing test accuracy and its clinical consequences: a primer for receiver operating characteristic curve analysis. Ann Clin Biochem 1993;30:521-539.
-
. National Committee for Clinical Laboratory Standards. Approved Guideline GP10A. Assessment of the clinical accuracy of laboratory tests using receiver operating characteristic (ROC) plots 1995 NCCLS Wayne, PA. .
-
Wang ST, Pizzolato S, Demshar HP. Receiver operating characteristic curve of Wallac DELFIA neonatal thyrotropin system for newborn congenital hypothyroidism. Proceedings of the 10th National Neonatal Screening Symposium 1994:245-246 ASTPHLD Washington, DC. .
-
Youden WJ. Index for rating diagnostic tests. Cancer 1950;3:32-35.
[ISI][Medline]
[Order article via Infotrieve]
-
Vander Helm HJ, Hische EAH. Application of Bayes's theorem to results of quantitative clinical chemical determinations. Clin Chem 1979;25:985-988.
[Abstract/Free Full Text]
-
Galen RS, Gambino SR. Beyond normality: the predictive value and efficiency of medical diagnoses 1975 John Wiley & Sons New York. .
-
Dhondt JL, Paux E, Farriaux JP. Need for a standardized procedure in the preparation of phenylalanine calibrators. Early Hum Dev 1996;45:277-285.
[ISI][Medline]
[Order article via Infotrieve]
Top
Abstract
Introduction
