| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Technical Briefs |
(1 Nanogen, Point of Care Diagnostics Division, Toronto, Ontario, Canada;2 Department of Laboratory Medicine, San Francisco General Hospital/University of California, San Francisco, CA;3 Bayer Healthcare Corporation, Tarrytown, NY;4 Department of Medicine, Division of Cardiology, Duke University Medical Center, Durham, NC;5 Department of Pathology, University of Maryland Medical Center, Baltimore, MD;
aaddress correspondence to this author at: Department of Laboratory Medicine, San Francisco General Hospital, 1001 Potrero Ave., San Francisco, CA 94110; fax 415-206-3045, e-mail wualan{at}labmed2.ucsf.edu)
Abstract
Background: There has been considerable debate regarding the impact of assay imprecision on the performance of cardiac biomarkers for diagnosis of acute coronary syndromes (ACS) and risk stratification for future adverse cardiac events.
Methods: Using existing data from 2 published clinical trials, we used a resampling method to statistically introduce 5%, 10%, and 20% imprecision to results for B-type natriuretic peptide (BNP) and cardiac troponin I (cTnI) and examined its impact on ROC curve analysis.
Results: Superimposition of artificial imprecision produced no significant difference in the area under the ROC curve observed for BNP for diagnosis of heart failure or for cTnI for 30-day risk stratification of patients with ACS.
Conclusion: Assay imprecision does not appear to be a critical determinant in the interpretation of cardiac marker results for patients with heart disease.
The issue of assay imprecision as a potential factor affecting the clinical utility of a diagnostic device has been raised in recent literature. For example, the joint European Society of Cardiology/American College of Cardiology Committee for the Redefinition of Myocardial Infarction determined that a maximum concentration of cardiac troponin I (cTnI) or troponin T (cTnT) exceeding the 99th percentile of the distribution of the marker in a reference control group is a valid indicator of myocardial necrosis (1). Subsequent guidelines have recommended that assays measuring cTnI or cTnT have total measures of imprecision (i.e., CVs) not exceeding 10% at this percentile cutoff, but this recommendation has been difficult to achieve in practice (2)(3).
We previously showed that a change in the theoretical imprecision profile of a cardiac troponin assay (i.e., an increase in the CV at the 99th percentile cutoff concentration from 10% to 25%) had minimal impact on the number of cases falsely classified for diagnosis of acute coronary syndromes (ACS) (4). In the present study, we evaluated the effect of increased assay imprecision on the clinical performance of B-type natriuretic peptide (BNP) for assessment of heart failure (HF) and on cardiac troponin for assessment of short-term risk stratification of patients with ACS. We used a statistical resampling methodology to assess the standard error of the area under the corresponding ROC curve (AUC).
Two different data sets were used in the study. To examine the role of imprecision on diagnostic accuracy, we obtained data from a multisite clinical trial on the use of the ADVIA Centaur BNP assay (5). This trial included BNP results from 722 patients with HF and 983 without HF. The clinical sensitivity and specificity were determined at various cutoffs for the diagnosis of HF. This data set represents a test with high diagnostic efficiency. To examine the role of imprecision on risk stratification, we obtained data from the multisite GUSTO IIa Trial (6). This trial included cTnI (Stratus II; Dade Diagnostics) and cTnT (ES300; Roche Diagnostics) results from 770 patients for whom 30-day mortality data were available. The clinical sensitivity and specificity were determined at various cutoffs for prediction of adverse short-term outcomes (death and myocardial infarction). The use of troponin for risk stratification produced lower diagnostic efficiencies than use of troponin for diagnosis of myocardial infarction or BNP for diagnosis of HF.
For each data set, the ROC curve was generated and the nominal AUC was computed, along with the associated SE and 95% confidence interval (7). A total of 500 replicates of each data set were created by stratified resampling with replacement (8). To create a resampled data set, the subset of values from persons with a designated outcome measure were resampled with replacement and the subset of values from persons without an outcome measure were also resampled with replacement; these 2 resampled subsets were then combined to form a resampled data set the same size as the original data set, with the same number of values associated with and without a given outcome measure. The AUC associated with each resampled data set was computed, and the empirical distribution of the AUC was obtained, from which the mean AUC, associated SE, and 95% confidence interval were determined.
To assess the effect of increased assay imprecision on the estimate of the AUC and the SE of the AUC, we assumed that replicate measurements by a given assay of a sample with a fixed concentration of analyte followed a gaussian distribution and that the increase in assay CV was a constant percentage value across all concentrations within the measuring range of the assay. A two-stage resampling procedure was used. In stage 1, 50 replicates of each data set were created by stratified resampling with replacement. Let {xi1, ... , xin} denote the ith such resampled data set (i = 1–50). In stage 2, for each resampled data set i, each observation xij (j = 1–n) was perturbed by sampling randomly 50 times from the N(xij, cxij) distribution, where c represents the CV of the superimposed source of imprecision, expressed in decimal form. The AUC was computed for each of the 50 x 50 = 2500 resampled data sets obtained by use of a given superimposed CV. A random-effects ANOVA (9) was performed with the AUC as the response variable, with stage 2 replicates nested within stage 1 replicates, and the estimated variances V1 and V2 were computed, where V1 represents the estimated variance in the AUC with contributions from both the inherent imprecision of the assay and from between-subject variability, and V2 represents the estimated additional variance in the AUC as induced by the superimposed noise. The corresponding standard errors SE1 [
(V1)] and SE2 [(V2)] were computed, and the ratio SE2/SE1 was obtained to determine the relative impact on the variability of the AUC by a given superimposed assay imprecision. Superimposed CVs of 5%, 10%, and 20% were assessed in this manner.
With respect to the BNP data set, the 95% interval estimate of the AUC obtained by resampling (0.909–0.939) was almost identical to the parametric estimate (0.909–0.937). With respect to the ACS data set, the 95% interval estimates of the AUCs obtained by resampling for both cTnI (0.445–0.637) and cTnT (0.454–0.653) were slightly wider than the respective parametric estimates (0.437–0.656 for cTnI and 0.449–0.669 for cTnT). For both the BNP and ACS data sets, the effect of inducing increased assay imprecision on the mean estimate of the AUC was negligible (Table 1
). [Note: to investigate the impact of increased assay imprecision on the variability of the AUC with respect to the ACS data set, we performed the two-stage resampling procedure with the cTnI data, as the clinical effectiveness of cTnT in predicting short-term mortality was statistically equivalent to that of cTnI]. The effect of inducing increased assay imprecision on the overall variability of the AUC also appeared to be relatively minor. For both data sets, SE2 (the SE associated with the superimposed source of imprecision with a CV of 5%) was 
3% of SE1 (the SE associated with existing assay imprecision and between-subject variability). A very large superimposed source of imprecision with a CV of 20% gave a SE2/SE1 ratio of only 12% for both data sets (Table 1
).
|
When we used cardiac biomarker assays to discriminate between cases and controls for a given clinical endpoint, using the AUC of the corresponding ROC curve, an increase in assay imprecision appeared to make a relatively minor contribution to the overall imprecision of the AUC. We demonstrated this for cardiac tests that produce both high and low areas under the ROC curve. For diagnosis of cardiac diseases, changes to BNP and cardiac troponin concentrations are typically much >20% of the values from nondiseased individuals, which explains why there was little influence of imprecision in these models. The findings in this study regarding imprecision and clinical decision-making are limited to the use of cardiac biomarkers. Assay imprecision for other analytes, e.g., electrolytes, may have greater influence on ROC curve analysis, with regard to diminishing clinical sensitivity and specificity for the corresponding diseases, as more subtle changes in analyte concentrations may indicate the presence of disease. Nevertheless, any change in clinical sensitivity and specificity attributable to imprecision should be considered for each analyte in the context of how the analyte is used in the clinical setting.
References
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  P. O Collinson, G. H Gaynor, and D. C Gaze Cardiac troponin I measurement using the ACS:180 to predict four-year cardiac event rate Ann Clin Biochem, March 1, 2008; 45(2): 184 - 188. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Choi, D. Park, S. Lee, Y. Hong, S. Kim, and J. Lee Cut-off values of B-type natriuretic peptide for the diagnosis of congestive heart failure in patients with dyspnoea visiting emergency departments: a study on Korean patients visiting emergency departments Emerg. Med. J., May 1, 2007; 24(5): 343 - 347. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Sakhuja, S. Green, E. M. Oestreicher, P. M. Sluss, E. Lee-Lewandrowski, K. B. Lewandrowski, and J. L. Januzzi Jr. Amino-Terminal Pro-Brain Natriuretic Peptide, Brain Natriuretic Peptide, and Troponin T for Prediction of Mortality in Acute Heart Failure Clin. Chem., March 1, 2007; 53(3): 412 - 420. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. T. Holmes and K. Buhr Mathematical Modeling: Assumptions Affect Results. Clin. Chem., August 1, 2006; 52(8): 1606 - 1608. [Full Text] [PDF]
|
|
|
|
|
|
C. A. Parvin and F. S. Apple The authors of the article cited above respond: Clin. Chem., August 1, 2006; 52(8): 1608 - 1609. [Full Text] [PDF]
|
|
|
|
|
|
S. Masson, R. Latini, I. S. Anand, T. Vago, L. Angelici, S. Barlera, E. D. Missov, A. Clerico, G. Tognoni, J. N. Cohn, et al. Direct Comparison of B-Type Natriuretic Peptide (BNP) and Amino-Terminal proBNP in a Large Population of Patients with Chronic and Symptomatic Heart Failure: The Valsartan Heart Failure (Val-HeFT) Data Clin. Chem., August 1, 2006; 52(8): 1528 - 1538. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
