| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Technical Briefs |
1
Dept. of Pathol. and Lab. Med., Univ. of Wisconsin Med. School, Madison, WI 53792,
2
Westgard Quality Corp., Ogunquit, ME 03907;
a address for correspondence: fax 608-263-0910,
Efforts to totally automate laboratory testing processes must address the issue of how to assure the quality of the final test result. A recent survey by Tetrault and Steindel (1) reported that laboratories are using the same quality-control (QC) procedures today that they used 10 years ago. Actually, more than half of the laboratories indicated they are still using control limits set as the mean ± 2s (the 12s rule), a practice that dates to the 1950s and '60s (2)(3), when statistical QC was first used with manual methods and the first generation of automated Technicon AutoAnalyzer systems. Other laboratories indicated they are using variations of the multirule type of QC procedure introduced in the early 1980s (4).
Tetrault and Steindel (1) recommend that "the best set of control rules will vary from method to method and cannot be determined through simple algorithms or formulas. The laboratorian has to balance true error-detection capabilities against the probabilities of falsely rejecting a good run." The information needed about the error-detection and false-rejection characteristics of stable QC procedures became available in the clinical chemistry literature almost 20 years ago (5)(6)(7)(8) and was extended by Cembrowski et al. (9)(10)(11)(12) to consider patient data algorithms. This information was incorporated into laboratory QC texts (13)(14) and clinical chemistry texts (15)(16) in the 1990s and continues to be expanded and improved by Parvin (17)(18)(19)(20) and by Smith and Kroft (21).
Guidelines and strategies for selecting QC procedures were described in 1986 (13), and some detailed applications have been published to demonstrate the selection and design of QC procedures (22)(23)(24). QC selection grids were introduced in 1990 (25) to provide a simple table "look up" approach for selecting QC procedures on the basis of the size of systematic error that must be detected and on an assay's expected stability or frequency of errors. More quantitative quality-planning models were introduced in 1991 (26)(27), and a new planning tool—the OPSpecs chart1 —was developed (28)(29) to show the relation between allowable precision and accuracy as well as what QC is necessary to assure detection of critical-sized errors that would otherwise cause method performance to exceed a defined analytical or clinical quality requirement. For applications involving analytical quality requirements and commonly used QC procedures, a compilation of OPSpecs charts is available to support manual applications for the full range of CLIA proficiency testing criteria for acceptability (30). For applications involving clinical quality requirements and a wider variety of QC procedures, a Windows1 -based PC program (QC Validator1 , Version 1.1; Westgard Quality Corp., Ogunquit, ME) was developed to prepare power function graphs, critical error graphs, and OPSpecs charts (31).
Quality-planning models describe the mathematical relationships between
a quality requirement and the factors that can cause variation in the
final test result. Some of these factors are analytical, such as the
stable imprecision (smeas, %) and inaccuracy of the method
and the sensitivity of the QC procedure to detect unstable random and
systematic errors (
SEcrit). Others are preanalytical,
such as the within-subject biological variation
(swsub), which describes the changes in concentration
about the subject's true homeostatic set point. We have now developed
an automated process to select statistical control rules and numbers of
control measurements (N) that will assure the quality required by
clinical decision interval (Dint) criteria or analytical
total error (TEa) criteria. The quality-planning models
used in the automatic process are essentially the same as those
described earlier (26)(27), except that the
effect of replicate measurements on method performance and QC design
are more completely and directly considered by entering the number of
replicate samples analyzed. Here, we describe the automatic process and
demonstrate that it provides results comparable with those in earlier
studies documented in this journal.
The computer program used, QC Validator Version 2.0, runs under
MicroSoft Windows 3.1 or Windows 95. It requires an IBM or
IBM-compatible computer with a 386 or higher processor, 1 MB hard disk
space, 2 MB RAM memory, a VGA interface and monitor, a printer
supported by Windows, and TrueType fonts. The program includes an
installation utility, a detailed HELP facility, and a manual that
provides tutorials, a reference guide to program function and
operation, and a review of the technical background for the program.
The computer program allows the user to enter information about the
characteristics of a method's performance (e.g., smeas,
inaccuracy, expected frequency of errors, swsub) and
the quality required (clinically important change or Dint,
TEa). The user then initiates automatic selection on the
basis of the number of control materials to be analyzed (i.e., 1, 2, or
3), and the program constructs a chart of operating specifications
(OPSpecs chart) that displays the selected control rules and N. The
automatic QC selection process is based on user-editable criteria for
the types of control rules that can be implemented by the laboratory,
the total numbers of control measurements that are practical, the
maximum percentage of false rejections that can be tolerated, and the
minimum percentage of error detection that is acceptable for detection
of medically important SEcrit or random errors.
The table of candidate QC procedures contains the power curves and
OPSpecs calculation parameters for 94 different QC procedures,
including: constant-limit single rules 12s,
12.5s, 13s, and 13.5s, with N
values from 1 to 8; multirules such as
13s/22s/R4s/41s/10
,
with N values of 2 and 4 applied over runs from 1 to 5; multirules such
as 13s/2 of
32s/31s/12, with
N values of 3 and 6 applied over runs from 1 to 4; multirules such as
13s/22s/R4s/41s/8,
with N values of 8 applied over 1 run; variable-limit single rules such
as 10.05 and 10.01, with N values from 2 to 8;
mean and range rules such as
0.05/R0.05,
0.01/R0.01, and
0.002/R0.002, with N values
from 2 to 8; and extended limit rules such as 14s,
15s, and 16s, with N values from 1 to 8. The
program includes an editor utility that allows users to add power
curves and OPSpecs calculation parameters for other rules and N values.
OPSpecs charts were prepared for the examples illustrated earlier
by spreadsheet calculations of allowable imprecision and allowable
inaccuracy for both the clinical model (26) and the
analytical model (27). Fig. 1
(top) shows the OPSpecs chart for a cholesterol example, for
which Dint = 20% and swsub = 6.5%. The
x-intercepts for maximum allowable imprecision range from
2.4% to 3.5%, which agrees with the range of 2.4% to 3.5% observed
previously (Fig. 2a in (26)). For individual QC
procedures, the largest difference was observed for the 13s
rule with N = 4, for which the current intercept is 2.8% (vs the
previous estimate of 2.95%). Performing duplicate cholesterol tests
and using the average of the two measured concentrations for diagnostic
classification provides a maximum allowable imprecision of 3.3% to
4.7%, compared with the earlier estimate of 3.2% to 4.9% (Fig. 2b in
(26).
|
For a cholesterol example in which TEa is 10%, Fig. 1
(bottom) shows the estimated allowable imprecision
(x-intercepts) to be 1.9% to 2.6%, the same as the 1.9%
to 2.6% documented earlier (Fig. 4 in (27)).
In the same example, when error detection is optimized for random error
rather than systematic error, the allowable imprecision is estimated as
1.2% to 2.3%, very similar to earlier estimates of 1.2% to 2.4%
(Fig. 6 in (27)).
The differences between the program output and the earlier spreadsheet calculations are the result of the different calculation parameters for the sizes of systematic error that can be detected with the specified probability. These parameters were estimated as the x-values that correspond to the 0.90 intersection with the power curves for the different control rules and N values, which in the spreadsheet calculation were constructed on a point-to-point basis and then later fitted with smoothed curves to improve the estimates.
The automatic QC selection process was tested by comparing the QC
procedures selected by the program that uses the OPSpecs methodology
with those selected in an earlier study in which critical error graphs
were used manually to select control rules and N values for 18
different analytes on a multitest chemistry analyzer (23).
In the earlier study, the design strategy fixed N at 2 per run,
considered only single-rule QC procedures, and varied the control rules
to match the error detection capability to the test method performance.
Table 1
shows the test, total error requirement, observed imprecision,
calculated 
SEcrit, and the QC procedure selected
manually from critical error graphs or selected automatically from
OPSpecs charts. For the automatic selection process, the frequency of
errors parameter was set to <2%, to permit use of 50% Analytical
Quality Assurance (AQA) charts in those situations where 90% AQA could
not be achieved.
|
For 14 tests, both the manual and automatic approaches selected the 13.5s rule with N = 2; for another test (albumin), both approaches selected the 12.5s rule with N = 2. For two tests (chloride and total CO2), the 12.5s rule with N = 2 had been selected manually, whereas the automatic selection process identified a multirule procedure with N = 4. In these latter two cases, if the automatic QC selection criteria were changed to restrict N to 2, then the automated process would select a multirule procedure with N = 2; if the selection logic for 50% AQA charts was changed to prefer single rules over multirules, then a 12.5s rule with N = 2 would be selected. That is, the automatic QC process can be modified to implement the same preferences used earlier in the process of manual selection from critical error graphs. For one test (calcium), method performance relative to the quality desired was so marginal that the assay required a special effort to make duplicate measurements, which was necessary to improve method performance and process control. The effect of making duplicate measurements could be considered directly with the automatic selection process, which then identified a 12.5s rule with N = 4 as necessary to provide 90% detection of medically important errors.
Therefore, although the QC designs and outcomes of the automatic and manual selections were similar, one thing was not: The time required to assess the QC designs for these 18 tests and to document the selection with printouts of OPSpecs charts was ~30 min with the automatic QC selection process, whereas the earlier study took several months.
In the future, one can envision an automatic QC selection process integrated into laboratory instrument software, on-line QC monitors, data management workstations, laboratory information systems, and automated laboratory process control software, thereby permitting laboratory scientists to focus on defining the quality needed for a test, rather than worrying about the control rules and numbers of control measurements that should be used. When coupled with on-line data acquisition and appropriate software to estimate method performance characteristics, the information needed for QC design will itself be automatically available. When linked to software for on-line QC monitoring, the selected rules can be implemented automatically. Thus, an integrated system for analytical quality management is possible when an automatic QC selection process is linked with method performance data and on-line QC monitoring, allowing the management of analytical quality with an appropriate QC design based on current information on method performance.
Such an integrated quality-management system provides the opportunity for a dynamic QC system that can automatically adjust to changes in method performance, e.g., changing to more-sensitive control rules and higher N values if performance deteriorates and relaxing the rules and reducing N if performance improves. Changes in method stability can be factored into the dynamic QC process and can also be used to adjust run length, as suggested by recent design work on the application of "average of normals" patient data algorithms to measure process stability and determine when to requalify the testing process (32). Thus, technology for the next-generation system for analytical quality management can be envisioned now that automatic QC selection is a reality.
Acknowledgments
Westgard Quality Corp. supported the development of this software. Robert Kennedy performed the coding, and Sten Westgard developed the HELP facility and program documentation. Additional information about the program is available at www.westgard.com.
Footnotes
1 QC Validator and OPSpecs are registered trademarks of Westgard Quality Corp. Windows is a registered trademark of Microsoft Corp. 
References
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  S. P. Caudill, S. J. Smith, G. R. Cooper, G. L. Myers, J. O. Westgard, and D. A. Wiebe Adequacy of NCEP Recommendations for Total Cholesterol, Triglycerides, HDLC, and LDLC Measurements Two of the authors of the above- mentioned article reply: Clin. Chem., May 1, 1998; 44(5): 1063 - 1066. [Full Text] [PDF]
|
|
|
|
|
|
P. C. Fallest-Strobl, E. Olafsdottir, D. A. Wiebe, and J. O. Westgard Comparison of NCEP performance specifications for triglycerides, HDL-, and LDL-cholesterol with operating specifications based on NCEP clinical and analytical goals Clin. Chem., November 1, 1997; 43(11): 2164 - 2168. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
