Clinical Chemistry 53: 735-741, 2007.
First published February 15, 2007; 10.1373/clinchem.2006.081174
(Clinical Chemistry. 2007;53:735-741.)
© 2007 American Association for Clinical Chemistry, Inc.
|
Automation and Analytical Techniques |
Evaluation of Assigned-Value Uncertainty for Complex Calibrator Value Assignment Processes: A Prealbumin Example
John Middleton1,a and
Jeffrey E. Vaks2
1 Department of Clinical Chemistry Development, Beckman Coulter Inc., Fullerton, CA.
2 Math/Stats Group, Beckman Coulter, Inc., Fullerton, CA.
aAddress correspondence to this author at: Department of Clinical Chemistry Development, Beckman Coulter Inc., Fullerton, CA. 92821. Fax 714-961-3740; email jsmiddleton{at}beckman.com.
 |
Abstract
|
|---|
Background: Errors of calibrator-assigned values lead to errors in the testing of patient samples. The ability to estimate the uncertainties of calibrator-assigned values and other variables minimizes errors in testing processes. International Organization of Standardization guidelines provide simple equations for the estimation of calibrator uncertainty with simple value-assignment processes, but other methods are needed to estimate uncertainty in complex processes.
Methods: We estimated the assigned-value uncertainty with a Monte Carlo computer simulation of a complex value-assignment process, based on a formalized description of the process, with measurement parameters estimated experimentally. This method was applied to study uncertainty of a multilevel calibrator value assignment for a prealbumin immunoassay.
Results: The simulation results showed that the component of the uncertainty added by the process of value transfer from the reference material CRM470 to the calibrator is smaller than that of the reference material itself (<0.8% vs 3.7%). Varying the process parameters in the simulation model allowed for optimizing the process, while keeping the added uncertainty small. The patient result uncertainty caused by the calibrator uncertainty was also found to be small.
Conclusions: This method of estimating uncertainty is a powerful tool that allows for estimation of calibrator uncertainty for optimization of various value assignment processes, with a reduced number of measurements and reagent costs, while satisfying the requirements to uncertainty. The new method expands and augments existing methods to allow estimation of uncertainty in complex processes.
|
Introduction
|
|---|
For a clinical measuring system, the purpose of calibration and calibrator value assignment is to provide for diagnostically accurate patient test results. Along with the reference ranges and clinical cutoffs, accurate patient test results provide data for valid medical decisions and effective treatment. We studied value-assignment processes for the Beckman Coulter, Inc. SYNCHRON® Prealbumin (transthyretin) immunoassay. Prealbumin (PAB)1
is an important marker for protein malnutrition and is used for assessment of patient nutritional status. Early detection of, and response to, protein malnutrition can shorten patient hospital stays and improve outcomes (1).
To support global quality improvements in clinical diagnostics, the European Union has created the In Vitro Diagnostics Medical Devices Directive, which requires that calibrator products offered in Europe have defined traceability and calibrator uncertainty (2). In addition, International Organization of Standardization (ISO) document 17511 provides guidance to manufacturers on how to establish assigned-value traceability (3). For determining calibrator-assigned value uncertainty, the ISO document refers to the Guide to the Expression of Uncertainty in Measurement (GUM) (4). Unfortunately, the GUM does not describe uncertainty analysis methods for complex value-transfer processes that involve multilevel calibrators with correlated errors. One approach to estimating the uncertainty of value assignment in such situations is by simultaneous building of multiple lots of calibrators and calculating SDs of the assigned value for each calibrator level. This approach, although it is conceptually simple and enables unbiased estimation, is prohibitively expensive. A more feasible approach is Monte Carlo simulation of the process.
Aside from fulfilling In Vitro Diagnostics Medical Devices compliance requirements, the clinical utility of estimating and reporting uncertainty in diagnostic assays has been debated (5)(6). We found Monte Carlo simulations to be useful in the design and optimization of the value-assignment process, because this method allows for changing simulated process parameters such as testing replication and number of instruments. In essence the process of PAB calibrator value assignment consists of transferring the value from the reference material CRM470 (7)(8) to a multilevel set of internal working calibrators (IWC), and then from the IWCs to the end-use calibrators during their production. The value-assignment process transfers the value from CRM470 to the production calibrator with additional uncertainty that is much smaller than that of the published uncertainty of CRM470 itself.
|
Materials and Methods
|
|---|
assay method
The Beckman Coulter reagent used to measure PAB concentration is a turbidimetric method. In the analytical reaction, serum PAB combines with a specific antibody to form light-scattering antigen-antibody complexes. The system monitors the changes in absorbances at several wavelengths and uses a nonlinear calibration curve obtained with the multilevel calibrator to calculate the concentrations of PAB in patient samples.
formulation of calibrators and standards
Delipidized human serum (9) containing endogenous PAB was used to formulate the IWC and production calibrators. High PAB concentrations were achieved by ultrafiltration concentration, and lower concentrations were created through dilutions of the concentrated ultrafiltrate with phosphate buffered saline (PBS). For the IWC, the dilution rations were quantified gravimetrically. Sodium azide at 1 mL/L is added as an antimicrobial. The upper limit of the analytical range for the PAB assay is 600 mg/L, which determines the requirement for the highest concentration of the production calibrator. To transfer reference values to the production calibrator from the IWC, the latter spans the range from 0 to 700 mg/L.
iwc value assignment
The IWC are directly traceable to CRM470 through a value-assignment process. The highest concentration calibrator is directly value assigned using CRM470, and then the values of the lower concentrations are determined from dilution ratios.
We initially considered a stochastic approximation approach to value assignment (10). However, because of the complexity of this method and accumulation of dilution errors with each iteration, we developed a method that is a modification of the stochastic approach. Both the Beckman Coulter and stochastic approximation methods assume that the logarithm of the signal vs the reciprocal dilution ratio (for narrow signal ranges) is approximately linear. In our modified method, however, rather than doing a number of calculations over a number of iterations, the value-assignment testing and calculations are performed in 1 step.
To compare the 2 value-assignment methods, we used the same dilutions and data from the stochastic approximation evaluation. For the modified approach, to determine the PAB concentration for the highest-level (level 8) calibrator, (1/dilution) vs ln(signal), a linear equation was fitted to data (Fig. 1
) in a narrow interval around the ln (signal) obtained with CRM470. The fitted line was used to estimate dilution of the highest-level calibrator needed to obtain the same signal as the one obtained with CRM470. The assigned value of the highest-level calibrator is equal to the reciprocal of the dilution of the highest-level calibrator multiplied by the assigned value of CRM470 (243 mg/L). This analysis is done for each of the 3 instruments used for data collection. The final value assigned to the highest level PAB IWC was the average across the instruments. The assigned values of the lower concentrations are determined by use of the respective dilution ratios applied to the assigned value of the highest concentration calibrator.
View larger version (16K):
[in this window]
[in a new window]
|
Figure 1. Reciprocal dilution (of IWC level 8) vs natural logarithm of the signal.
The IWC level 8 assigned value is calculated using the best fit line and the CRM470 signal. The natural log of the CRM470 signal is input into the line equation to calculate a reciprocal dilution value. That value is multiplied by the CRM470 assigned value (24.3 mg/L) to obtain the IWC level 8 assigned value.
|
|
production calibrator value assignment
The production calibrators form a 6-level calibrator with PAB concentrations ranging from 0 mg/L to 
600 mg/L. For the value assignment of the production calibrators, 18 measurements per concentration of the production calibrator and the IWC were performed on each of 4 SYNCHRON CX instruments with each of 2 reagent lots (144 measurements per test sample). Within a reagent lot, the data were averaged over all the instruments, and an IWC standard curve was determined by fitting a 5-parameter logit function to the averages of instrument responses vs assigned concentrations:
This standard curve was then used to calculate production calibrator PAB concentrations. The assigned values of the production calibrators are the average concentrations across 2 reagent lots, 4 instruments, and 18 replicates per reagent lot per instrument.
Estimation of Uncertainty
In estimating the uncertainty of the values assigned to the calibrators, it is useful to consider the traceability chain (See Fig. 1 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue4). Each process step in the traceability chain is considered when determining the total uncertainty of the production calibrator. At the beginning of the chain is the international protein reference material CRM470, with an uncertainty, Ustd, characterized by the SD of 9.0 mg/L, as published in the CRM470 product insert. Uiwc is the uncertainty in the assigned values for the IWC. Lastly, Upc denotes the uncertainties of the production calibrator assigned values. If these were single calibrator level process steps with uncorrelated additive error variance components, then the uncertainty of each step could be combined into 1 value representing the total uncertainty (4):
 |
However, for multilevel value transfers with cross-correlated errors of assigned values of calibrator levels, deriving closed form equations for total uncertainty for each calibrator level is practically impossible. Therefore, we evaluated the uncertainties of the IWC, as well as of the production calibrators, by Monte Carlo simulation of the value transfer process. To be performed correctly, the simulation process must be well described, including accurate estimation of error contributions and the correlations of the errors in various stages of the process. In addition, all the variance/covariance components used in the simulation are evaluated from the real value assignment process data. The concepts of Monte Carlo simulation are described in (11). The program for the simulation for this study was developed in S-Plus on a PC platform (12); it is included in the Supplemental Appendix. A distribution of the assigned values is generated through simulations of 1000 value assignment events. The SD of these simulated assigned values is the assigned value uncertainty. The 95% confidence intervals of the uncertainty were determined as 2.5th and 97.5th percentiles of the SDs estimated from 40 runs of 1000 simulated value assignment events. This simulation procedure augments the method applicable to simple value assignment processes described in GUM (4).
iwc and production calibrator uncertainties
The entire uncertainty evaluation process is flow-charted in Fig. 2
. Items 1.1 through 1.7 refer to the actual IWC and production calibrator formulation and value-assignment processes previously described. During the preliminary assessment of the value-assignment events for the IWC and the production calibrator, it was determined that instrument-to-instrument variation, as well as intersample covariation of the errors, were significant and thus must be accounted for in the Monte Carlo simulations. Items 2.3 and 2.6 in Fig. 2
refer to the extraction of covariance structure and variance components of the instrument signals from the actual value-assignment data. Here instrument signal averages obtained with the CRM470 dilutions, IWC, and the production calibrator samples were calculated. Also, the instrument-to-instrument and within-instrument variance components of instrument signals were estimated with variance components analysis (13) along with the covariance matrix of instrument signals for various calibrator levels. To impose the above covariation structure on the simulated instrument responses for various calibrator levels, the Cholesky decomposition of the (P x P) covariance matrix is multiplied with the (P x N) matrix of random numbers with independent gaussian distribution with zero means and unit variances, then divided by the diagonal of the same covariance matrix (11). Above, P is the number of calibrator levels and N is the number of instruments. Items 3.3 to 3.9 (Fig. 2
) refer to the Monte Carlo simulation of the value-assignment events. The value-assignment calculations, using the simulated instrument signals, are performed as they were in the actual calibrator value assignments. For the IWC value-assignment calculations the natural log of the signal data for the IWC dilutions is regressed with the reciprocal dilution factors. The IWC set point estimate is based on the dilution required to match the CRM470 signal. For the PC value assignment, a concentration vs signal standard curve is created from the IWC data by curve fitting. By inputting the production calibrator signal data into this standard curve, the calibrator concentrations are calculated.
View larger version (17K):
[in this window]
[in a new window]
|
Figure 2. Summary diagram of the uncertainty assessment process.
The main areas depicted are the actual standard and calibrator formulation and value assignments, the simulation parameter extraction process, and the Monte Carlo simulation process.
|
|
The gravimetric error contributions during formulation of the IWC and value assignment were much less than 0.01%, a value that was insignificant compared to the 3.72% total error of the assigned values. Therefore we ignored the contribution of errors of gravimetric dilution. The uncertainty of the CRM470 standard, which was included in the simulations, was the major source of IWC uncertainty because with the described process the error of transferring the reference value to the IWC is very small. The uncertainty of IWC was included in the simulations of the value assignment of the production calibrators. Simulation of 1000 value assignment events was repeated 40 times. The pooled SDs and the nonparametric 95% confidence intervals obtained from these simulation results characterize the uncertainties. The 95% confidence intervals of the uncertainties were set as the nonparametrically estimated 2.5th and 97.5th percentiles of the 40 groups of 1000 simulated assigned values for each level of the calibrator.
This method enables estimation of the calibrator uncertainties; however, because of the multilevel nonlinear calibration, it is not obvious how these uncertainties propagate to patient result uncertainty. To evaluate these effects, recoveries of patient sample concentrations were simulated. In this simulation the calibrator-assigned values varied according to the previously determined uncertainties. Then, using target responses for various concentrations, patient results were calculated with various calibrations using those variable assigned values. Calculated patient result SDs showed the effect of the calibrator uncertainty. This method allows for establishing requirements to the calibrator uncertainty.
|
Results
|
|---|
The IWC value-assignment results are shown in Table 1
. During the value-assignment evaluations, multiple dilutions of the highest level IWC were made such that their signals were near the signal of CRM470, and a linear fit was made to find the dilution necessary to obtain the same signal as the reference material (Fig. 1
). Once the dilution was determined, the assigned value of the highest-level calibrator was calculated by dividing the assigned value of the reference material by the dilution factor used to match the highest-level calibrator. Table 1
shows the 3 assigned values calculated for the 3 different instruments, and the average of those values as the final assigned value for the highest level IWC. For comparison, the results using the stochastic approximation method (10) are also included. Our approach, because it uses all the data from many dilutions to estimate the dilution of the level 8 IWC, might provide for smaller uncertainty of the assigned values than does stochastic approximation (SD of 7.7 vs 11.2 mg/L), in which the signal data from dilutions that do not match those of the reference material are discarded. Here, however, the difference is not statistically significant based on F-test with 2 degrees of freedom. The assigned values of the lower levels of the IWC were calculated using the dilution ratios of the highest-level calibrators.
Simulation of the IWC value assignment event was repeated; 1000 value assignments were simulated 40 times yielding 40 sets of 1000 assigned values. A histogram of 1 of the sets of 1000 assigned values is shown in Fig. 3
. The pooled SD of these assigned-value sets, SD = 26.9 mg/L, is the estimate of the assigned-value uncertainty, Uiwc, for the highest-level IWC. The uncertainties, based on dilution ratio, for all 8 levels of the IWC are summarized in Table 2
.
View larger version (18K):
[in this window]
[in a new window]
|
Figure 3. Histogram of 1000 simulated assigned values for level 8 of the Internal Working Calibrator.
The standard reference deviation of this data is an estimate of uncertainty of the IWC assigned value. The lower IWC level uncertainties are determined by multiplying the high level uncertainty by the dilution ratio.
|
|
For the production calibrator value assignment, measurements of the IWC were taken to create a standard curve, from which the assigned values of the production calibrators were calculated (See Fig. 2 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue4). The pooled SD of the 40 sets of 1000 simulated value- assignment events is the uncertainty. By using this approach, the production calibrator uncertainty includes the IWC and CRM470 components and the error of value transfer from IWC. The final results of the value assignment, as well as the uncertainty analysis, are summarized in Table 2
. The relative uncertainties over all levels of the production calibrator are 3.75%.
Because the uncertainty component associated with the production calibrator value assignment was rather small, we assessed the effect of reducing the amount of testing. Rather than making 144 measurements (4 instruments, 36 replicates per sample), we made 48 measurements (4 instruments, 12 replicates per sample) were made. The results of those simulations are summarized in Table 3
. Here the production calibrator value assignment was isolated from the previous events such that the error contribution from this event alone may be considered. Lastly, the effect of calibrator uncertainty on patient result uncertainty is provided in Table 3
.
|
Discussion
|
|---|
Our results demonstrate that that the process of transferring PAB values from the reference material to the production calibrators contributes little to the total error. The assigned-value uncertainty of CRM470 is 243 (9.0) mg/L, with a relative error of 3.70%. Through the value transfer process from this reference material to the IWC, the highest-level IWC had an assigned value of 722 (26.9) mg/L with a relative uncertainty of 3.72%. The component of uncertainty of the IWC assigned value, contributed by the process of transferring the value from reference material, is negligible. So, the described value- assignment process allows for a precise transfer of values from a single-level international standard CRM470 to a multiple level IWC. Unlike the stochastic approximation approach (10), which relies on determining the assigned values by an iterative process involving many tests and calculations, the described approach is simpler and provides for at least as good calibrator uncertainty. Multiple dilutions of the highest-level IWC were made such that the signals were near that of CRM470 when assayed. The relationship between the dilutions and the signals allows for the determination of the highest-level IWC assigned value, while gravimetric dilutions introduce no significant errors to lower levels of the IWC.
The final relative total uncertainties of the PAB production calibrators were all 3.75%, insignificantly higher than the relative uncertainty of the reference material. The relative uncertainties of the assigned values of all levels of the production calibrator, excluding the uncertainty of CRM470, were all <0.8%. So, the major part of the uncertainty of the BCI PAB production calibrators is comprised of the uncertainty of the reference material, because the value-transfer process adds very little to the total uncertainty. The exceedingly small contribution to the total error of the IWC and production calibrator value- transfer steps, although counterintuitive, is partially of the result of taking the square root of the sum of squared individual contributions per equation (2). In addition, the relatively large number of measurements reduces the uncertainty. In this instance the reduction is 10-fold, the square root of one hundred. As shown in Table 3
, reducing the amount of testing of the production calibrator by 33% reduces the cost of the calibrator but has little effect on the total calibrator uncertainty or on the assay precision.
The question of clinical usefulness can be addressed in relation to patient result uncertainties due to calibrator uncertainties (Table 3
). To be clinically useful, the errors of the patient results caused by the calibrator uncertainties must be small relative to the PAB reference range of 180–380 mg/L (14). At the lower and upper limits of the reference range, uncertainties of the patient results caused by the uncertainties of the calibrator are 1.6 and 2.8 mg/L, respectively, with 96 measurements per sample. These values as percentages of the width of the reference range are 0.9% and 1.6%, respectively, small relative to the reference range. Therefore, the risk of patient misdiagnosis due to calibrator uncertainty is small.
|
Acknowledgments
|
|---|
The contributions to this work by Phebe Shaw, Phil Martz, Art Kessner, and Steve Alter were invaluable.
|
Footnotes
|
|---|
1 Nonstandard abbreviations: ISO, International Organization of Standardization; PAB, prealbumin; IWC, internal working calibrator; PC, production calibrator; VA, value assignment; GUM, Guide to the Expression of Uncertainty in Measurement. 
|
References
|
|---|
- Beck FK, Rosenthal TC. Prealbumin: A Marker for Nutritional Evaluation. Am Fam Physician 2002;65(8):1575-1578.[Web of Science][Medline]
[Order article via Infotrieve]
- "Council Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices", Official Journal of the European Union L331 (December 12, 1998)..
- . International Organization for Standardization. In vitro diagnostic medical devices-measurement of quantities in biological samples-metrological traceability of values assigned to calibrators and control materials (17511) 2003:23 ISO Geneva. .
- . International Organization for Standardization. Guide to the Expression of Uncertainty in Measurement 1995:101 ISO Geneva. .
- Krouwer J. Critique of the Guide to the Expression of Uncertainty in Measurement Method of Estimating and Reporting Uncertainty in Diagnostic Assays. Clin Chem 2003;49(11):1818-1821.[Abstract/Free Full Text]
- Kristiansen J. The Guide to Expression of Uncertainty in Measurement Approach for Estimating Uncertainty: An Appraisal. Clin Chem 2003;49(11):1822-1829.[Abstract/Free Full Text]
- Whicher JT, Ritchie RF, Johnson AM, Baudner S, Bienvenu J, Blirup-Jensen S, et al. New International Reference Preparation for Proteins in Human Serum (RPPHS). Clin Chem 1994;40(6):934-938.[Abstract/Free Full Text]
- Johnson AM, Sampson EJ, Blirup-Jensen S, Svendsen PJ. Recommendations for the Selection and Use of Protocols for Assignment of Values to Reference Materials. Eur J Clin Chem Clin Biochem 1996;34:279-285.[Web of Science][Medline]
[Order article via Infotrieve]
- Vance DE, Weinstein DB, Steinberg D. Isolation and Analysis of Lipoproteins Secreted by Rat Liver Hepatocytes. Biochim Biophys Acta 1984;792:39-47.[Medline]
[Order article via Infotrieve]
- Schlain B. A Stochastic Approximation Method for Assigning Values to Calibrators. Clin Chem 1998;44(4):839-848.[Abstract/Free Full Text]
- Rubenstein RY. Simulation and the Monte Carlo Method 1981 John Wiley New York. .
- Venables WN, Ripley BD. Statistics and Computing. Modern Applied Statistics with S-Plus 1994 Springer-Verlag New York. .
- Corbeil RR, Searle SR. Restricted maximum likelihood (REML) estimation of variance components in the mixed model. Technometrics 1976;18:31-38.[CrossRef][Web of Science]
- Synchron CX Chemistry Information Manual. Beckman Coulter reorder no. 249595..
Top
Abstract
Introduction
