|
|
||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
General Clinical Chemistry |
1 Laboratory of Clinical Biochemistry, Psychiatric University Hospital, Risskov, Denmark.
2 Division of Biostatistics, HFZ-542, U.S. Food and Drug Administration, Rockville, MD.
aAddress correspondence to this author at: Laboratory of Clinical Biochemistry, Psychiatric University Hospital, Skovagervej 2, DK-8240 Risskov, Denmark. Fax 45-77893549; e-mail linnet{at}post7.tele.dk.
| Abstract |
|---|
|
|
|---|
) and II (ß) errors into account. The suggested procedure, however, supposes gaussian distributions of both blank and sample measurements and a linear calibration curve. In clinical chemistry, asymmetric, nongaussian blank distributions are common, and the calibration curve may be nonlinear. We present a partly nonparametric procedure that takes these aspects into account. Methods: Using theoretical distribution models and simulation studies, we developed a LoD estimation procedure suitable for the field of clinical chemistry that is partly based on nonparametric statistics.
Results: For sample size n, the nonparametrically determined 95th percentile of the blank measurements {obtained as the value of the [n(95/100) + 0.5]th ordered observation} defines the limit for results significantly exceeding zero [limit of blank (LoB)]. The LoD is the lowest value that is likely to yield a result exceeding the LoB. LoD is estimated as: LoB + cß x SDS, where SDS is the analytical SD of a sample with a low concentration; cß = z1 - ß/[1 - 1/(4 x f)]; z1 - ß is the standard normal deviate; and f is the number of degrees of freedom for estimation of SDS. cß is approximately equal to 1.65 for a type II error of 5%. Approaches and needed tabular values for calculation of confidence limits are presented as well as sample size. Worked examples are given to illustrate estimation and verification of the limit of detection. Simulation results are used to document performance.
Conclusion: The proposed procedure appears useful for application in the field of clinical chemistry and promotes a standardized approach for estimating LoDs of clinical chemistry assays.
| Introduction |
|---|
|
|
|---|
In a statistical sense, not only should the type I error (the significance test) be taken into account, but also the so-called type II error, i.e., the error of not detecting the presence of analyte. Recently, the International Organization for Standardization (ISO) published a set of guidelines concerning determination of the LoD that encompasses both type I and II errors related to detection (1)(2)(3)(4). This set of guidelines assumes (in line with most publications on determining the LoD) that distributions of values are gaussian, and accordingly, parametric statistical approaches are applied (5)(6). Actually, in the field of clinical chemistry, the distribution of blank measurements is often truncated at zero and thus is asymmetric and nongaussian. This is so because the raw analytical signal is seldom provided. Instead, the user is presented with direct read-out or black-box systems providing concentration results that are nonnegative. Furthermore, the calibration function is often hidden in the apparatus; therefore, approaches based on specific forms of calibration functions (linear, four-parameter logistic, or other types of curves) often can not be applied (2)(5)(7). With this in mind, we present here a nonparametric approach for estimation of the LoD that is suitable for the field of clinical chemistry. The procedure has the advantage of being generally applicable without relying on special distributional assumptions. The form of the calibration function is not relevant for the suggested procedure. The procedure is outlined, and we present tables with critical values, worked examples for estimation and verification of the LoD, and a performance study based on simulations.
| Overview of the LoD Concept |
|---|
|
|
|---|
). On the other hand, we observe that some of the measurements for a sample with a low amount of analyte fall below this limit. Defining the alternative hypothesis as the case with analyte present implies that the alternative hypothesis is erroneously rejected in case of measurements of the low sample falling below the 95th percentile of blank measurements. Thus, we commit an error, the so-called type II error (ß). We observe here that hypothesis testing is one-sided.
|
Recently, the ISO recommended a definition of the minimum LoD in relation to stated levels of type I and II errors (1)(2)(3)(4). The default level for these errors was set to 5%, i.e.,
= ß = 5%. An
value of 5% corresponds to using the 95th percentile of the distribution of blank values as the limit for declaring a measured value significantly higher than the blank. Given a gaussian distribution of blank values (Fig. 1A
), this limit corresponds to:
![]() |
B are the mean and SD of the blank measurements, respectively.
For the situation in Fig. 1B
with an asymmetric distribution of blank values, the 95th percentile is not estimated correctly by a parametric approach. Using simulations, we estimated the error for a sample size of 25. If the true mean is 0 so that one-half of the blank values are negative and are given the value 0, the average estimated SD corresponds to 57% of the SD of the unmodified gaussian distribution. The average estimated mean is 0.40, and the average estimated 95th percentile corresponds to 1.37, i.e., 83% of the anticipated value of 1.65. For a true mean equal to 0.5 SD, the average estimated parametric 95th percentile amounts to 91% of the true value of 2.15.
The most straightforward procedure to estimate the 95th percentile of an asymmetric distribution is to apply a nonparametric principle based on the ordered values (8). Let nB be the number of measurements of the blank sample. When we rank nB values according to size, the 95th percentile may be estimated as the value of the [nB(95/100) + 0.5] ordered observation (see the Appendix that accompanies the online version of this article at http://www.clinchem.org/content/vol50/issue4/). In case of a non-integer value, interpolation is carried out between neighboring values (see the example later in the text). The limiting percentile (Perc) of the blank distribution, which cuts off the percentage
in the upper tail of the distribution, will in what follows be called the LoB, i.e.:
![]() |
To address the type II error level, one has to consider the minimum sample concentration that provides measured concentration values exceeding the LoB with a specified probability. If the type II error level, ß, is set to 5%, 95% of the measurements should exceed the LoB. Fig. 2
illustrate two cases: one with a true sample concentration equal to the LoB, and another with a true sample concentration at a value so that the 5th percentile of the distribution of sample measurements equals the LoB. In the first case, 50% of the sample measurements are below the LoB with the other 50% of values exceeding the LoB. Only the latter 50% will be declared as significantly exceeding the blank value. In Fig. 2B
, on the other hand, 95% of the measurements exceed the LoB and are declared significantly higher than the blank value. Thus, only 5% of the measurements are erroneously declared not significantly different from the blank, which is the type II error. According to the ISO definition, the true analyte concentration of this sample is the minimum detection limit, or the LoD. Usually, the sample distribution is gaussian, and in this case the 5th percentile of the distribution can be derived from the mean and SD as:
![]() |
S are the mean and SD of the sample measurements, respectively. Overall, we have:
![]() |
|
If the sample distribution is not gaussian, the 5th percentile of the sample distribution can be estimated nonparametrically in the same way as the LoB. However, parametric estimation is more efficient and should be used when possible (see the online Appendix).
estimation of LOB and LOD
The outline given above was based on theoretical (population) distributions. In practice, one has to deal with real sample distributions. Thus, LoB must be estimated from repeated (nB) measurements of one or several blank samples, and the SD of sample measurements from repeated measurements (nS) of sample(s) with a relevant concentration. Thus, an estimate (indicated with the subscript EST) of the LoD is then obtained as:
![]() |
/E(SDS). z1 - ß is the (1 - ß) percentile of the standard gaussian distribution, and E(SDS) is the mean value of SDS. If the number of repeated measurements (nS) is not too small, the mean value of SDS can be approximated by
x [1 - 1/(4 x f)] (for details, see the section on "Estimation of SDS" in the online Appendix), and LoDEST = LoBEST + z1 - ß/[1 - 1/(4 x f)] x SDS is an unbiased estimate of LoD = LoB + z1 - ß x
S.
The uncertainty of the estimates is considered next. With regard to LoB, the standard error of estimates for theoretical distributions can be derived (see the online Appendix). With actual data, the user may obtain the 95% confidence interval (CI) limits from the ranked blank values according to Table 1
(9). The uncertainty of the LoD estimate is composed of a component originating from the uncertainty of the LoB estimate and a part from the sample measurements. As detailed in the online Appendix, an approximate 95% CI of the LoD can be derived by combining the 80% CI for the LoB and SDS, supposing nB = nS. The 80% CI limits for the LoB can be derived from the ranked values by use of Table 1
, and multiplication factors that provide the 80% CI limits for SDS are presented in Table 2
. The suggested procedure can be applied down to a sample size of nB = nS = 50. An alternative to the described procedure that also may apply at lower sample sizes is the bootstrap principle (10).
|
|
In relation to estimation of the LoD, a problem may be that
S often is nonconstant because it frequently increases with sample concentration. However, over the limited range of low concentration values that are of interest in the present context,
S may be approximately constant, and the outlined procedure is then straightforward (see the example later in the text). Otherwise, a more complicated approach can be undertaken in which it is assumed that the sample SD is a function of the concentration (2).
verifying a claimed LOD
In addition to estimation of the LoD, one may also be interested in primarily verifying a claimed LoD, e.g., a laboratory may want to assure that an assay fulfills the claim of a given LoD stated by a manufacturer. In that case, it is necessary to distinguish between a full and a partial verification procedure. A full procedure consists of estimating the LoB, performing repeated measurements of sample(s) with a concentration equal to the claimed LoD, and estimating the proportion of results exceeding the LoB. If the recorded proportion is in agreement with the expected 95%, then the data support the claim of the LoD. A partial procedure consists of applying the LoB stated by the manufacturer and recording the proportion of results exceeding the LoB for sample(s) with a concentration equal to the claimed LoD. Shown in Table 3
, for sample sizes of 201000, are the lower bounds for the recorded proportion that are in accordance with
= ß = 5% for the full and partial procedures. The interval is slightly wider for the full verification procedure because, in this case, the LoB is a random variable (see the online Appendix). Notice that the bounds for partial verification of the LoD can also be applied as bounds for verification of a claimed LoB. If the proportion of blank measurements below a claimed LoB is equal to or higher than the proportion listed in Table 3
, the claimed LoB is supported. An example of verification is shown in the Examples section.
|
characteristics of blank and sample
The blank sample(s) should be as similar as possible to the natural patient samples, e.g., for a drug assay a suitable blank sample would be a serum or plasma sample free of drug, and not just a buffer solution. To assure that the measurements are representative, compilation of measurements on several blank samples is preferable. Thus, instead of repeated measurements of only one particular serum sample, a set of 510 or more blank serum samples would be preferable because matrix differences exist from sample to sample. With regard to endogenous compounds, it may be difficult to obtain blank samples. In some situations, e.g., for tumor markers, samples from nondiseased individuals may be appropriate. For hormones, blank samples may in some cases be provided from diseased individuals or individuals with suppressed hormone concentrations attributable to pharmacologic treatments, assuming that these samples have characteristics otherwise similar to the routine test samples. Otherwise, blank samples might be samples stripped of the component, e.g., by precipitation by an antibody, by enzymatic degradation, or by adsorption to charcoal. However, such treatments may also remove potential interfering compounds and thus give a too optimistic picture of assay performance.
With regard to the sample(s) with low analyte concentrations, it may be preferable to add analyte, e.g., a drug, to a set of serum samples from different patients rather than to just one serum sample or a serum pool. For endogenous compounds, ideally a set of patient samples with concentrations in the low range might be used. A pooled SDS estimate can then be derived from repeated measurements of the set of samples, e.g., 510 measurements of each of 510 samples (see the example presented later). Measurements on different days should be carried out, so that SDS reflects the total analytical variation.
For the estimates of LoB and LoD to be meaningful, the measurements by the method in question should be unbiased. When a reference method for the analyte exists, comparison of measurements in the low range with this method should be undertaken. Alternatively, measurements of samples to which analyte has been added might be used to demonstrate that realistic measurements are obtained. This prevents a spuriously low LoD from being reported because if the assay provides values that are lower than the true analyte concentration, other factors being equal, a too-low LoD is going to be estimated.
sample size considerations
The optimum ratio between the number of blank and sample measurements in the estimation procedure is related to the uncertainties of the estimated LoB and SDS of the sample measurements. Nonparametric estimation of LoB is roughly half as effective as a parametric estimation procedure (see the online Appendix). The uncertainties of percentile or SD estimates will also tend to be proportional to the dispersion of the distributions. Thus, nonparametric estimation of the LoB would suggest that the number of blank measurements should exceed that of the sample measurements. However, the dispersion of sample measurements is likely to exceed that of the blank measurements. Overall, an approximately equal number of blank and sample measurements is likely to be near optimal in many cases (see the online Appendix). Concerning the precision of estimates of LoB and LoD in relation to the number of measurements, see the Simulation section.
reporting of results
In a laboratory, the LoB may be used to determine how patients results will be reported: i.e., as substance detected or not detected. It should be kept clear what the exact meanings of "detected" and "not detected" are. Not detected, i.e., a result below the LoB, means that the true concentration is likely to be less than the LoD. "Likely" refers to the type II error level (ß), which often is set to 5%. Thus, a result less than the LoB should be reported as "<LoD" and not as "<LoB" or "zero". A result exceeding the LoB, i.e., detected, means that the true concentration is likely to exceed zero, and the reporting could be ">zero" or "detected". Likely refers here to the type I error (
), which often is set to 5%.
A modification of the above-mentioned principles for reporting results might be considered in cases in which results are to be used in scientific studies. Here, unbiased results for the groups of individuals being investigated are obtained by reporting the concentrations actually measured irrespective of whether the values are below or above the LoB. Otherwise, biased results for groups may be obtained.
The relative uncertainty of measurements at or just exceeding the LoD is often large, and usually a quantitative result is not reported. The lower limit for reporting quantitative results (LoQ) relates to the relative imprecision (CV) considered acceptable by the laboratory. From a precision profile for the assay, the LoQ may be determined, e.g., corresponding to a CV of 10% or 20% (5). The possible bias of the method at this level might also be taken into consideration, so that an upper limit of the total error determines the LoQ. The LoQ will constitute the lower limit of the reportable range for quantitative results for the assay. The LoQ is not considered further here.
| Examples |
|---|
|
|
|---|
= ß = 5% were used. The blank measurements consisted of four measurements on each of five sera from patients not taking the drug, and the low-sample measurements were based on four measurements on each of five other sera from patients not taking the drug, to which 10 nmol/L of the drug had been added (Fig. 3
![]() |
|
This is a point estimate, and the true LoD may be somewhat lower or higher. The 95% CI for the LoD can not be derived as described above because of the small sample sizes used here. The usual steady-state serum concentrations for patients in treatment with the drug extend from 50 to 350 nmol/L. Accordingly, a LoD of 11.6 nmol/L is acceptable for clinical use of the assay.
example of verifying a claimed LOD of an assay
A given analytical procedure is claimed by the manufacturer to have a LoD of 45 U/L with
= ß = 5%. The user decides to carry out a full verification procedure on the basis of 25 blank measurements (5 measurements of 5 blank samples over 5 days) and 25 measurements of samples to which 45 U/L of the analyte had been added (5 measurements of 5 samples over 5 days; Fig. 4
). Visual inspection reveals that the distribution of blank values is asymmetric (nine 0 values), and accordingly the LoB is estimated nonparametrically. The blank values are ranked according to size, and the 95th percentile corresponds to the 24.25th ordered observation (25 x 0.95 + 0.5). Linear interpolation between the 24th and 25th observations yields 19.17 U/L [18.01 + 0.25 x (22.65 - 18.01)].
|
The proportion of sample measurements that exceed the LoB is 96% (24 of 25). From Table 3
we see that 96% is above the lower bound for agreement (87%; full verification procedure). Thus, the observed proportion is in accordance with the expected one of 95% according to the claim, and the present evaluation does not contradict this claim.
| Evaluation of the Performance of the LoD Estimation/Verification Procedures by Simulation |
|---|
|
|
|---|
The blank distribution follows a standard gaussian distribution (µB = 0 and
B = 1) with negative values assigned the value zero. The sample distribution is gaussian with µS = 4.1125 and
S = 1.5. Under this model, the population LoB is 1.645, and the LoD is 4.1125 [1.645 + 1.5 x 1.645 (
= ß = 5%)]. It is assumed that
S is constant in a region around the LoD. Thus, the estimation procedure consists in estimating the LoB and adding cß SDS.
The performance of the suggested LoD estimation procedure can be evaluated as a function of the sample size by drawing pseudo-random numbers from these distributions. The bias and precision of LoB and LoD estimates are recorded (Table 4
).
|
Observe that the nonparametric LoB estimate has considerable uncertainty. At a sample size of 25, the SE amounts to 38% of
B in the model, decreasing to
20% of
B at a sample size of 100. The SE of the LoD exceeds that of the LoB for a given sample size, illustrating that both the uncertainty with regard to the LoB estimation and that associated with SDS estimation contribute to the overall uncertainty. Thus, if a manufacturer wishes to assure a reasonably precise estimate of the LoD, a fairly large sample size is needed. At least 100 blank measurements and 100 measurements of samples with low amounts of analyte should be considered. Given an automated assay, this may not constitute a problem, but for manual assays, e.g., a hormone assay, a large number of repetitions may represent a considerable task. When estimating the LoD, one should ideally keep the conditions near routine operating conditions, e.g., the measurements should be dispersed over several days and not restricted to a single run.
performance of the verification procedure
The performance of the full and partial verification procedures can be studied by simulations. Two aspects are of interest here. One is the ability to verify a claimed LoD that is equal to or higher than the true LoD, and the other is to discover a claimed LoD that is too low. Concerning the first point, the use of CIs in the verification procedures defines the probability of verifying a true LoD. A one-sided 95% CI assures 95% probability of verifying a claimed LoD equal to or higher than the true LoD, according to the either partial or the full verification procedure. We studied the other problem, discovering a claimed LoD as being too low, by simulations.
The ability (power) of detecting a claimed LoD as being too low depends on the sample size and the ratio between the dispersions in the sample and the blank,
S/
B. We chose to study the detection of a claimed LoD that was 25% lower than the true LoD according to the full procedure (Table 5
). For sample sizes ranging from 20 to 70, the power of detection ranged from 24% to 85%. The highest range of power (5685%) was observed for
S/
B = 1. Given a slightly higher ratio,
S/
B = 1.5, the range of power decreased to 4578%. For the highest ratio studied,
S/
B = 5, the range of power was lowest, 2451%.
|
evaluation of the coverage of the suggested procedure for estimation of the ci of LOD
The suggested procedure for estimation of the 95% CI for a LoD relies on some approximations outlined in the online Appendix. To study the performance of the procedure, we evaluated the coverage by simulations (10 000 runs for each parameter combination). The same model situations as considered above were investigated. The coverage indicates the frequency with which the true value of the LoD for a given model is included in the estimated 95% CI. Ideally, the frequency should be 95%. As may be observed from Table 6
, the actual coverage ranged from 87% to 96%. The best results were obtained for low
S/
B values. Thus, the estimated limits were approximately correct. An alternative to the suggested procedure is to apply the bootstrap approach, which, however, requires computerized calculations.
|
| Discussion |
|---|
|
|
|---|
Although one part of the problem is the general concept of the LoD, another part involves the statistical techniques applied. Somewhat surprisingly, all of the publications cited here, except Brown et al. (7), suppose gaussian distributions of both blank and sample measurements. The background may be that most authors deal with the signal response (y) rather than the concentration scale (x). Having determined the statistical limits on the response scale, the authors may subsequently transform values to the concentration scale on the basis of the calibration function. Accordingly, procedures linked to linear or nonlinear calibration functions have been described, the latter mainly in connection with immunoassays (2)(5)(7)(14)(15). When one is working on a response scale, symmetric distributions of blank responses may occur frequently, and a fully parametric approach is appropriate. In clinical chemistry, however, the instrument response is usually hidden, and one has access only to the final concentration output. Given a symmetric distribution of blank responses, transformation to concentration values may give both positive and apparently negative concentration values. The latter are usually converted to zero or a small positive number by the instrument, which leads to asymmetric distribution of blank concentration values. As mentioned earlier, this is the background for the procedure suggested here. To apply the proposed procedure, it is necessary to have concentration values down to zero available. If the instrument software truncates results at a higher value, it is not possible to carry out a meaningful assessment of the LoD.
The present procedure has the advantage of being able to deal with asymmetric blank distributions. In addition, the procedure conforms to currently accepted concepts for establishing the LoD, as outlined in the ISO guidelines, primarily in regard to both type I and II errors. Furthermore, by focusing on concentration or amount of analyte directly, the procedure becomes general and does not depend on a particular type of calibration function. Variations in terminology that exist in this field can create confusion. Several reports, including the ISO guidelines, use the term critical limit (xc) for the significance limit of blank measurements. Because "critical" in laboratory medicine often has another meaning, i.e., an alarm value that should be reported immediately, we prefer another term, the LoB. With regard to LoD, many variations exist, e.g., minimum detectable level (MDL) and lower limit of detection (LLoD); these variations are of minor significance as long as the precise definition is clear and the probability levels are indicated. A drawback of the present procedure is that it gives a less precise estimation of the LoB values than may be obtained by fully parametric procedures. It is possible to increase the precision of the nonparametrically estimated percentile slightly by applying modern computer-based statistical methods, e.g., a resampling principle (bootstrap method) (10)(16), weighted percentile estimation (17), or smoothing techniques (18). The bootstrap principle is the simplest and most generally applicable procedure of those mentioned and has the advantage of also providing confidence limits for the percentile estimate. The gain in precision corresponds to saving 1015% of the observations compared with the simple nonparametric procedure (16). The bootstrap method also represents an alternative procedure for estimation of the CI of the LoD.
The LoD of a method should not be confused with the so-called sensitivity. According to ISO, the analytical sensitivity relates to the slope of the calibration function (19). The steeper the slope, the more sensitive the assay is to slight changes in the amount of analyte. This definition of sensitivity expresses an entirely different concept than the LoD. The distinction between the LoD and the LoQ should also be kept clear (5). A laboratory may choose to report results at or only slightly above the LoD as being detected and restrict the reporting of quantitative results to values at or exceeding the LoQ (20). The LoQ indicates the lowest value at which the analytical procedure fulfills claimed imprecision or total error specifications and will often be a more important characteristic for a quantitative analytical procedure than the LoD. The LoD may be of particular interest in relation to detection of drugs of abuse and in relation to tumor markers.
It is important that measurements of blanks and samples be carried out in a comparable way. If only a buffer solution is used as a blank, a falsely low LoB may be obtained because the dispersion of measurements often is larger in the real matrix (21). Concerning sample(s) with low concentrations, estimation of the dispersion (SDS) from several samples is an advantage to assure that average performance is assessed. Finally, total assay variation should be reflected by measurements performed on different days.
Manufacturers may compete with regard to producing assays with the lowest possible LoD. Thus it is important to have a standardized procedure with expressed levels of type I and II errors for determining the capability of detection. Use of this procedure could allow a fair comparison of competing assays.
| Acknowledgments |
|---|
| Footnotes |
|---|
1 Nonstandard abbreviations: LoD, limit of detection; LoQ, limit of quantification; LoB, limit of blank; ISO, International Organization for Standardization; and CI, confidence interval. ![]()
| References |
|---|
|
|
|---|
The following articles in journals at HighWire Press have cited this article:
![]() |
H. Keshishian, T. Addona, M. Burgess, D. R. Mani, X. Shi, E. Kuhn, M. S. Sabatine, R. E. Gerszten, and S. A. Carr Quantification of Cardiovascular Biomarkers in Patient Plasma by Targeted Mass Spectrometry and Stable Isotope Dilution Mol. Cell. Proteomics, October 1, 2009; 8(10): 2339 - 2349. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. Hassibi, H. Vikalo, J. L. Riechmann, and B. Hassibi Real-time DNA microarray analysis Nucleic Acids Res., August 31, 2009; (2009) gkp675v1. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Moxness, S. Tatarewicz, D. Weeraratne, N. Murakami, D. Wullner, D. Mytych, V. Jawa, E. Koren, and S. J. Swanson Immunogenicity Testing by Electrochemiluminescent Detection for Antibodies Directed against Therapeutic Human Monoclonal Antibodies Clin. Chem., October 1, 2005; 51(10): 1983 - 1985. [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |