| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Technical Briefs |
1 Department of Chemical Pathology, St. Vincent’s Hospital, Darlinghurst, NSW, 2010, Australia; fax 61-2-8382-2489, e-mail gjones{at}stvincents.com.au
The multirules system for optimizing assay error detection originally developed by James Westgard and others is now widely used in clinical pathology laboratories (1). The power of various combinations of rules to detect changes in assay performance can be assessed by use of power function charts (2), some of which are freely available on the Westgard QC website (3). In this report I reassess the power of error detection of rules that require data from more than one quality-control (QC) run.
Variables that can be adjusted in setting a QC protocol, and modeled in power function charts, include the choice of rules, the number of QC samples in a run (n), and the number of runs over which data are to be assessed (R). In the development of power function charts, other variables must also be clearly defined so that the charts represent the actual practice in the laboratory.
Westgard’s rules can be divided into within-run rules, where the data for decision-making are available from within one QC run, and cross-run rules, where data from more than one QC run are required (R >1). Using approved terminology (4), examples of within-run rules for n = 2 include 13s and 22s, and examples of cross-run rules for n = 2 include 41s and 10x. Within-run QC rules are clearly preferable to cross-run rules because they will allow detection of changes in assay performance as soon as possible after the change. Cross-run rules are commonly added to within-run rules with the aim of gaining additional power of error detection to meet quality specifications.
In this report I use a spreadsheet application to develop power function charts with the aim of assessing the power of error detection of combinations of rules when cross-run rules are included. Specifically, I review the assumptions made in generating currently available power function charts (3) and reassess the power of error detection for these combinations of rules.
Power function charts were produced with an application developed in Microsoft Excel (Microsoft Corporation). QC results were simulated with the Excel random number generator function with a gaussian distribution. Changes in assay bias were modeled by adding various constants to the output of the random number generator. QC rules and combinations of rules were evaluated by the frequency with which the rules were triggered for each change in bias. For n = 2, 13s, 22s, 41s, and 10x and for n = 4, 13s, 22s, 41s, and 8x were evaluated for their ability to detect changes in assay bias. For n = 2, the 22s, 41s, and 10x rules were all applied across both sets of QC data. For n = 4, 22s was applied only to the same QC material and 41s and 8x across the two materials, the same criteria as used on the Westgard website. A 12s "warning" rule was not used in the models. Data representing 1000 QC runs were used to evaluate each set of rules. I did not investigate the effect of increases in imprecision because the cross-run rules under consideration are designed to detect changes in bias rather than changes in precision. Similarly, I also did not include the R4s rule in the evaluation because only changes in bias were considered. Further information on the spreadsheet model can be found in the Data Supplement that accompanies the online version of this Technical Brief at http://www.clinchem.org/content/vol50/issue4/. No approval was sought from the institutional ethics committee because no patient samples or data derived from patient materials were involved in the study.
The spreadsheet application was used initially to reproduce the power function chart for multirules with n = 2 (Fig. 1A
) as shown on the Westgard website. The legend shows the number of QC runs (R) for each of the sets of rules as appears on the website diagram. When I was choosing the parameters required to replicate the original data, it became clear that the meaning of R is somewhat complex when applied to multirules that include one or more cross-run rules. For example, a 13s/22s/41s combination of rules is listed as R = 2, when this value of R applies only to the 41s rule because the within-run rules are applied only to the latest set of QC data (R = 1). Similarly, when the 10x rule is added, the data line indicated with R = 5 represents different data collections over 1, 2, and 5 QC runs amalgamated in the same line. An important finding was that the available charts for cross-run rules do not consider the probability that a change in assay performance may have been detected by other rules during the period of data accumulation.
|
I believe that the version of run definitions implied in current power function charts does not reflect actual practice. A QC run can either detect a change in assay performance that has occurred since the previous QC run or pick up a change that has persisted for longer but has not previously been identified. Cross-run rules have the ability to act only in the latter manner; however, their power of error detection should be limited to changes that have not been detected by other rules that have been run during the period of data accumulation. Fig. 1B
shows the effect of adjusting the model to take this limitation into account. Specifically, the 41s rule was triggered only if a shift had been present for two QC runs and the within-run rules did not detect the shift during the previous QC run. Similarly, the 10x rule was triggered only if a shift had been present for five QC runs and the shift had not been identified by the within-run rules or the 41s rule in the previous four runs. It can be seen that the power of error detection for the cross-run rules changes markedly with this alteration to the model, showing reduced probability of error detection for all changes in bias between 0.5 and 4.0 times the SD of the assay. A full description of this power function chart would be the probability of detecting a new bias with the within-run rules together with the probability of identifying a previously undetected preexisting bias with the cross-run rules.
One method of summarizing the data in power function charts is the shift that can be detected with 90% certainty. This parameter is a marker of the change in assay bias that the QC protocol can reliably detect. As seen in Fig. 1A
, the change in bias in an assay that Westgard multirules (13s/22s/41s/10x; n = 2; R = 5) can apparently detect with 90% certainty is 
2.0 times the SD of the assay. When the model is adjusted so that previously detected shifts are removed from the data, the assay bias that can be detected with the same degree of certainty is increased from 2.0 to 3.3 times the assay SD, no different from the change that can be detected by the within-run rules in use. Fig. 1C
shows the additional error detection provided individually by the 41s rule beyond that provided by the within-run rules, and 10x beyond that of the shorter-run rules. It can be seen that the additional error detection of these cross-run rules peaks and then decreases. This decrease is caused by the increased chances that a large change in assay bias will be detected by within-run rules before the application of the cross-run rules.
Data were also modeled for n = 4, and the error detection for full multirules was lower than described on the Westgard site for all changes in bias from 1.0 to 3.0 times the SD when previously detected shifts were removed from the 8x rule (data not shown). The change in bias that can be detected with 90% certainty decreases from 1.8 times the SD, as shown on the Westgard website, to 2.5 times the SD, the same change in bias that can be detected with the use of within-run rules only.
The above data suggest that the true power of error detection of cross-run rules is markedly less than has previously been claimed. It can be seen that the action of the cross-run rules conforms with previous descriptions as warning rules for detecting smaller shifts in assay bias than are easily detectable by common within-run rules. However, these rules cannot detect these minor shifts with any certainty. For example, the 41s rule will detect assay changes a maximum of 25% of the occasions an otherwise unidentified change occurs. However, the present study was limited to evaluation of changes in bias at a single point in time. It is likely that cross-run rules may be more useful in detecting a gradual change in bias over time because such changes may be less likely to be detected previously by within-run rules.
It is important that laboratory scientists be aware of the performance characteristics of the QC protocols in use in their laboratories. Reliance on within-run rules is preferable to rules requiring data from multiple runs because errors can be detected sooner after they have occurred. However, if cross-run rules are to be used they are of value only if they add to the error detection generated by the rules that have already evaluated the data that are required for them to fire. By excluding previously detected changes in assay bias, I have demonstrated that the cross-run rules 10x and 41s with n = 2 and 8x with n = 4 have a lower probability of error detection than is shown in the data on the Westgard website and do not add to error detection at the 90% confidence level derived by use of within-run rules.
References
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
), 13s (R = 1). (B), the same rules as in A except that cross-run rules add to error detection only on those occasions when a change in bias has not been detected in the QC runs required for application of the cross-run rules. For cross-run rules, error detection is shown for the first run in which each rule has sufficient data to operate. (· · · · ·), 13s/22s/41s/10x; (- - -), 13s/22s/41s; (- · - · -), 13s/22s (R = 1); (