Clinical Chemistry 52: 880-885, 2006.
First published March 2, 2006; 10.1373/clinchem.2005.063107
(Clinical Chemistry. 2006;52:880-885.)
© 2006 American Association for Clinical Chemistry, Inc.
|
General Clinical Chemistry |
Effect of the Reporting-Interval Size on Critical Difference Estimation: Beyond "2.77"
Graham Ross Dallas Jones
1 Department of Chemical Pathology, St. Vincent’s Hospital, Sydney, and Faculty of Medicine, University of New South Wales, Sydney, Australia.
Address for correspondence: Department of Chemical Pathology, St. Vincent’s Hospital, Victoria St., Darlinghurst, NSW 2010, Australia. Fax 61-2-8382-2489; e-mail gjones{at}stvincents.com.au.
 |
Abstract
|
|---|
Background: The reporting interval is the bin size used to report numerical pathology results and must be determined for every analyte. The influence of the size of the reporting interval on the critical difference (CD) between two results from the same patient has not been addressed previously.
Methods: The effect of changing the reporting-interval size (RIS) on CDs was modeled by use of a spreadsheet application. The findings were applied to data on CDs with analytical precision values from our laboratory.
Results: As the RIS increases relative to the combined analytical and within-person biological variation, there is an approximately linear increase in the CD from the value determined by use of published techniques. The revised estimate is as follows: CD = 21/2 x z x (SDa2 + SDi2)1/2 + 1.5 x RIS, where CD, SD, and RIS are all in the same units. This effect is seen for any probability associated with the critical difference and for both uni- and bidirectional changes.
Conclusions: The choice of reporting interval should be made in the light of assay requirements. Where there is a clinical need for detection of small changes in analyte concentration, the reporting interval should be kept small relative to the combined variation attributable to assay precision and within-person biological variation.
|
Introduction
|
|---|
The reporting interval, also known as the reporting-increment or grouping interval, is the interval into which continuous laboratory results are "binned" for reporting purposes. The reporting-interval size (RIS)1
is the difference between the upper and lower borders of the interval. Each interval is commonly referred to by the numerical value of the center of the interval, and this central value may then be used for interpretation or mathematical manipulation. The term "rounding" is also used to describe this process because the final result may be numerically different from the raw data. The reporting interval is commonly set to the nearest decile, e.g., 0.1, 1, 10, or 100; therefore, the concept is also commonly described as the number of significant figures used in reporting results.
Sequential laboratory results from the same patient are often reviewed to assess possible changes in the clinical status of the patient. In this situation it is important to be sure that a difference in results is statistically significant and is therefore unlikely to be attributable to chance. Fraser et al. (1)(2) have addressed this question based on the combination of analytical precision, expressed as the CV of the analysis (CVa), and the average within-subject biological variation (CVi), from which a critical difference (CD) can be calculated for a predetermined probability. For example, the difference in results, moving either up or down, that is more than 95% likely to indicate a true change in the patient (i.e., is <5% likely to be attributable to random variation), is described as:
where CD95, CVa, and CVi are expressed as percentages. The factor 2.77 is equal to 
2 times the z statistic for the difference, which is 1.96 for this example. The z scores for various degrees of certainty have been described for differences in either direction (two-tailed) and in a single direction (one-tailed) (1). For the purposes of this report, differences between results that have a certain probability of being statistically significantly different are described as indicating that probability of indicating a true change in the patient.
The choice of RIS is known to influence the measured precision of an assay (3) and the efficacy of quality-control protocols (4). However, the influence of the choice of reporting interval on the CD has not been addressed previously.
|
Materials and Methods
|
|---|
The change in a patient’s results from one reporting interval to another encompasses a range of possible differences in the underlying data. The smallest difference that may be represented by such a change is the difference between the closest extremities of the intervals into which the two results reported, and the largest difference is that between the furthest extremities. The probabilities that the differences represented by these extremes of separation represent true changes in the patient were modeled for different values of RIS, by use of Excel 97 (Microsoft Corporation). An equation based on the NORMDIST function in Excel was used to determine the probability that the difference between two values was a true change in the patient in one or either direction.2
For different values of RIS, the range of probabilities of a true change associated with each step in reporting intervals was calculated. The total result variation, given as the total SD (SDtot) and combining the analytical (SDa) and within-subject biological variation (SDi), was defined as:
The SD rather than the CV was used for this study so that the analytical and biological variation and the RIS could be reported in the same units. These data were used to determine the range of probabilities of a true change in the patient that may be encompassed by a change to the reporting interval that includes the theoretical difference of 2.77 times the SDtot. The study also identified the first reporting interval that represented probabilities of a true change that were all greater than a determined probability with this analysis performed for one- and two-tailed probabilities of a true change at the 80%, 90%, 95%, and 99% confidence levels. The lines of best fit for the relationships between CD and RIS were validated by Passing–Bablok regression (Analyze-it).
To assess the application of the theoretical developments, precision data were obtained from routine analysis of quality-control material (Bio-Rad) on a Modular <P> analyzer (Roche Diagnostics Australia) in use in our laboratory. Within-person biological variation data were taken from the Biological Variation database on the Westgard Website (5). These data were used to evaluate the values of RIS relative to SDtot encountered in a routine laboratory. As an example of the effect of choice of RIS over a range of analyte concentrations, data for within-instrument assay precision for serum creatinine were taken from the RCPA-AACB Quality Assurance Program. The data represent the median instrument CVs for serum creatinine for 8 analyte concentrations from 2 cycles of data (4 data points per instrument per concentration) in 2004 for 326 enrolled instruments reporting to the program to the nearest 1 µmol/L. For the purposes of this model, a constant CVi of 4.3% was used at all concentrations of serum creatinine (5). The effect of using reporting intervals of 1 µmol/L or 0.1 mg/dL (8.84 µmol/L) were modeled as examples.
Formal ethics committee approval was not sought because no patient-related information was included in the work.
|
Results
|
|---|
Sequential results from the same patient may appear within the same reporting interval or in separate intervals. The two results are represented on a pathology report by the values of the center of each of the two intervals, and calculations based on these results are performed with these central values. The range of probabilities of a difference between reporting intervals representing a true change in the patient was derived from the spreadsheet application, and example results are shown in Fig. 1
. If the CD is calculated using current theory (1)(2), a reporting interval can be identified that includes this calculated value. This interval is marked with an asterisk for each example in Fig. 1
, and it can be seen that as the RIS increases, the probabilities for a true change represented by this interval also increase. It can also be seen from Fig. 1
that the difference in results that can be reliably identified as being >95% likely to indicate a true change in the patient increases with the RIS, as shown by the arrows in Fig. 1
, which move to the right as the RIS increases.
View larger version (19K):
[in this window]
[in a new window]
|
Figure 1. Illustration of the range of probabilities of differences in results representing a true change for different sizes of reporting interval.
The width of the boxes indicates the RIS, and the lower and upper edges of each box indicate the minimum and maximum probabilities of a true change for the reporting interval indicated. The box farthest to the left in each graph represents data remaining within the same reporting interval, the next box represents a change to the adjacent interval, and boxes farther to the right indicate changes to the sequentially more distant reporting intervals. The changes are expressed as multiples of the SDtot. (A), RIS = 0.1 x SDtot; (B), RIS = 0.5 x SDtot; (C), RIS = 2.0 x SDtot. * indicates the reporting interval that includes the change of 2.77 x SDtot. In each panel, the arrow indicates the first reporting interval for which all probabilities for that interval representing a true change are >95% (probability at the 95% confidence level is represented by the dashed line).
|
|
If the CD95 is calculated with use of the parameter 2.77, then, as discussed above, the range of probabilities of a true change that are associated with the reporting interval that includes the calculated difference increase as the RIS increases. This is shown in more detail in Fig. 2
, which displays the maximum and minimum probabilities associated with a difference of 2.77 x SDtot. As can be seen, once the RIS is more than 1.8 times the SDtot, the change in intervals may also represent no difference at all. Thus, the use of current estimates of the probability of a true change may become less specific as the RIS increases.
View larger version (14K):
[in this window]
[in a new window]
|
Figure 2. Relationship between RIS and the range of probabilities of a true change in a patient for a change in results from an initial reporting interval to the reporting interval that includes a change of 2.77 x SDtot.
The curves marked by the filled circles represent the maximum and minimum probabilities. The dashed horizontal line indicates the probability at the 95% confidence level.
|
|
If the aim is to be certain at a predetermined confidence level that a difference in results is not attributable to random variation, it is necessary to identify the first reporting interval that represents probabilities of a true change that are entirely greater than that value. This relationship is plotted in Fig. 3
, which shows the number of reporting intervals by the results must change for 95% certainty as well as the CD associated with that change. As can be seen, the theoretical CD95 of 2.77 x SDtot is achieved only with an RIS that is negligible compared with the value of SDtot. The lack of smoothness of the line for CD in the graph is attributable to quantum changes in the number of reporting intervals required for a significant change and the fact that some intervals provide probabilities higher than 95%. In Fig. 3
, a line with a slope of 1.5 x SDtot and an intercept of 2.77 has been drawn on the data, showing a good correlation. Lines with slopes ± 33% of this line of best fit (slopes of 1.0 and 2.0) have also been drawn (dashed lines in Fig. 3
), giving boundaries for all data points in the relationship. Passing–Bablok analysis was used to determine the slopes and intercepts for data sets developed for one- and two-tailed statistics for commonly used probabilities. This analysis was performed with data up to a value for RIS/SDtot of 1.0 because the data showed greatest linearity up to that point (Table 1
). As can be seen in Fig. 3
, the statistical lines of best fit passes through the expected value when the reporting interval is of zero size, within the error of the estimate, and also that the slope of 1.5 is within the error of the estimate of the slopes for all probabilities. Review of the data also showed that no individual data points were outside the lines drawn with the expected intercepts and slopes of 1.0 and 2.0.
View larger version (21K):
[in this window]
[in a new window]
|
Figure 3. Relationship between RIS and difference in results, represented by the number of reporting intervals that are required to detect a true change with >95% probability ( ) and the CD associated with the change ( ).
The dashed line is described by the equation: y = 2.77 + 1.5*SDtot; the dotted lines have slopes of 1.0 and 2.0.
|
|
View this table:
[in this window]
[in a new window]
|
Table 1. Slopes and intercepts for lines of best fit, z-scores, and predicted coefficients for calculating CD (z-value x 2) determined for data sets of CD compared with RIS at various probabilities for both one- and two-tailed statistics.
|
|
To assess the possible implications of these data in a routine automated chemical pathology laboratory, a range of data is presented in Table 2
. The first line of each reporting interval and the units shown for each analyte are those in use in our laboratory, and the subsequent lines (in mg/dL) are for comparative purposes. As can be seen, at the analyte concentrations shown the RIS in use in our laboratory vary between 0.03 and 1.74 times the SDtot. This equates to CD95 values between 2.8 and 5.2 times the SDtot. The use of mass units gives values for RIS between 0.06 and 3.1, giving CD95 values between 2.9 and 5.1 times the SDtot.
View this table:
[in this window]
[in a new window]
|
Table 2. Estimates of RIS/SDtot and the CD obtained with data from an example laboratory for commonly used SI and non-SI units.
|
|
The effect of RIS on the CD varies according to the analyte concentration, as shown for the example of serum creatinine in Fig. 4
. With a smaller value of RIS, the number of reporting intervals associated with a significant difference is higher at all creatinine concentrations but the numerical value associated with the difference is less. On the basis of the creatinine data given as an example, use of a reporting interval of 0.1 mg/dL (8.84 µmol/L) increases the CD95 by an average of 11 µmol/L compared with the CD95 obtained with a reporting interval of 1 µmol/L. The mean increase in CD99 for this change is 
13 µmol/L (data not shown). Because these changes are similar over the range of creatinine concentrations tested, the percentage effect on the CD is markedly higher at low serum creatinine concentrations.
|
Discussion
|
|---|
The effect of the choice of reporting interval has been shown to influence the CD at all levels of probability. At the 95% level, the CD has previously been described as 2.77 times the combined analytical and biological CV (1)(2); however, it can be seen that this is true only when the RIS is small compared with the total result variation. An expanded general equation to include the effect of RIS is as follows:
where CD, SDa, SDi, and RIS are all in the same units. This equation is an approximation for the zig-zag relationship seen in Fig. 3
but is correct within the extremes defined by values calculated by use of 1.0 and 2.0 in place of the coefficient 1.5 in the equation. The data in Table 2
demonstrate that the reporting intervals in routine use in our laboratory and elsewhere produce a wide range RIS/SDtot giving CD95 values as high as 5 times the SDtot compared with the theoretical value of 2.77. These data are based on quality-control material with concentrations close to the reference interval, but higher values for RIS/SDtot may be expected at lower analyte concentrations and the reverse at higher analyte concentrations. It has also been demonstrated that as the RIS increases, the use of previous versions of the CD calculation to determine the required difference in results leads to progressive loss of sensitivity for true changes in a patient.
This effect of analyte concentration on the effective CD is shown in Fig. 4
for serum creatinine with reporting intervals of 1 µmol/L or 0.1 mg/dL (8.84 µmol/L). At a creatinine concentration of 60 µmol/L, the effect is quite marked, with the CD increasing from 21% to 42% if the larger reporting interval is used. As can be seen, as the creatinine concentration increases, the effect becomes smaller in percentage terms. These data are used as an example only, and laboratories must determine the effect of their chosen RIS on data produced by their laboratory. It is important to note that the estimates of CD described here are only as accurate as the inputs used in the calculations. Thus, deviations from the calculated CD may be caused by poor estimates of analytical or biological variation, application of biological variation from a healthy population to an unhealthy population without supporting evidence, or other effects such as preanalytical variation or assay interferences.
From these findings it is possible to add some thoughts to the discussion on choice of reporting intervals for laboratory tests. This may be seen as a balance between masking potentially useful information (with larger reporting intervals) and reporting unhelpful information, i.e., indicating a precision that does not exist (with smaller reporting intervals). Other less "scientific" factors include practical issues such as the capabilities of the laboratory computer system and simplicity of results for ease of memory and reduction of transcription errors. Previous discussions on the appropriate RIS for routine reporting of pathology results have focused on avoiding the reporting of apparently unhelpful information based on analytical precision only (6) or on a combination of analytical and biological variation (7). Another approach is based on reporting up to and including the first uncertain figure (8). Applying the principles indicated in these reports to data from our laboratory would produce coefficients for CD95 estimation of 4.8 for serum creatinine (6), 4.4 for serum magnesium(6), 4.6 for serum calcium (6)(7), and 4.5 for serum amylase (6)(7). These examples demonstrate that information loss may occur when one attempts to avoid reporting apparently misleading precision. This situation may become even more marked if more than one sample is used before and after an intervention because the SDtot of the grouped data will be even smaller than the values estimated here from singleton collections and measurements.
It is unlikely that there is a single acceptable degree of change in CD suitable for all assays that would allow a universal recommendation about choice of reporting intervals; however, for assays in which small changes in results may be of clinical significance for monitoring purposes, the effect of RIS on the CD should be taken into consideration. Additionally, the uncritical implementation of CD calculations using the coefficients of 2.77 for CD95 and 3.65 for CD99 may generate misleading interpretations.
|
Footnotes
|
|---|
1 Nonstandard abbreviations: RIS, reporting-interval size; CVa, CV of the analysis; CVi, within-subject CV; CD, critical difference; SDtot, total standard deviation (combination of analytical and within-subject biological variation); SDa, standard deviation of the analysis; and SDi, within-subject standard deviation. 
2 The two-tailed probability of a result being separated from another by an amount X was determined with the Excel NORMDIST worksheet function, using the following equation: Probability = 2*(NORMDIST(X,0,SDtot*1.414,True)-0.5), where X is the difference being evaluated; SDtot is the combined SD for SDa and SDi for the analyte concentration under consideration, which is multiplied by 1.414 to allow for variation in both measurements; and True indicates a cumulative probability. The NORMDIST function is the probability function for a gaussian distribution from negative infinity to X. To change this to the two-tailed probability (–X to +X), 0.5 is subtracted from this function, and the result is multiplied by 2. For a one-tailed probability, the following equation was used: NORMDIST(X,0,SDtot*1.414,True.
|
References
|
|---|
- Fraser CG, Hyltoft Petersen P, Lytken Larsen M. Setting goals for random analytical error in specific clinical monitoring situations. Clin Chem 1990;36:1625-1629.[Abstract/Free Full Text]
- Fraser GC. Biological Variation: From Principles to Practice 2001:67-90 AACC Press Washington. .
- Williams GW, Schork MA. Some statistical implications of rounding in the College of American Pathologists Survey. Am J Clin Pathol 1976;66:162-166.[ISI][Medline]
[Order article via Infotrieve]
- Groth T, Falk H, Westgard JO. An interactive computer simulation program for the design of statistical control procedures in clinical chemistry. Comput Programs Biomed 1981;13:73-86.[CrossRef][ISI][Medline]
[Order article via Infotrieve]
- Westgard JO. Biological variation database. Westgard QC Website. http://www.westgard.com/essay88.htm (accessed March 31, 2005)..
- Badrick T, Wilson SR, Dimeski G, Hickman P. Objective determination of appropriate reporting intervals. Ann Clin Biochem 2004;41:385-390.[CrossRef][ISI][Medline]
[Order article via Infotrieve]
- Hawkins RCW, Johnson RN. The significance of significant figures. [Letter]Clin Chem 1990;36:824.[Free Full Text]
- Garcia-Santamarina S, Suarez-Vega I, Fuentes-Arderiu X. Significant decimals and rounding. [Letter]Clin Chem Lab Med 2004;42:1071-1072.[CrossRef]
Top
Abstract
Introduction
) or 0.1 mg/dL (8.8 µmol/L; 
). (B), CD expressed in µmol/L (
and •) and as a percentage (