Clinical Chemistry 53: 575-580, 2007.
First published February 22, 2007; 10.1373/clinchem.2006.083311
(Clinical Chemistry. 2007;53:575-580.)
© 2007 American Association for Clinical Chemistry, Inc.
|
Current Issues in Laboratory Medicine |
Evaluation of the Performance of Randomized versus Fixed Time Schedules for Quality Control Procedures
Curtis A. Parvin1 and
Sanford Robbins, III2,a
1 Department of Pathology and Immunology, Washington University School of Medicine, Saint Louis, MO.
2 Department of Pathology, Anne Arundel Medical Center, 2001 Medical Parkway, Annapolis, MD.
aAddress correspondence to this author at: Department of Pathology, Anne Arundel Medical Center, 2001 Medical Pkwy., Annapolis, MD 21401. Fax 443-481-4207; e-mail srobbins{at}aahs.org.
 |
Abstract
|
|---|
Background: Although minimum regulatory standards exist for determining QC testing frequency, decisions regarding when and how to run QC samples are not standardized. Most QC testing strategies test control samples at fixed time intervals, often placing the samples in the same position on an instrument during subsequent QC events and leaving large gaps of time when control samples are never run, yet patient samples are being tested.
Methods: Mathematical derivations and computer simulation were used to determine the expected waiting time between an out-of-control condition and the next scheduled QC test for various QC testing strategies that use fixed or random intervals between QC tests.
Results: Scheduling QC tests at fixed intervals yields an average time between the occurrence of an out-of-control error condition and the next scheduled QC test that is equal to half of the fixed time interval. This performance was the best among the QC scheduling strategies investigated. Near-optimal performance, however, was achieved by randomly selecting time intervals between QC events centered on the desired expected interval length, a method that provides variation in QC testing times throughout the day.
Conclusions: If the goal is to vary QC testing times throughout the day while maintaining the shortest expected length of time between error conditions and the next scheduled QC test, then a near-optimal QC scheduling strategy combines randomly selected time intervals centered on the desired length of time between QC events with fixed time intervals.
|
Introduction
|
|---|
Traditionally, performance evaluation of QC strategies has focused on the power of QC rules to detect out-of-control error conditions when QC testing is performed. Relatively little has been written regarding the effect of the frequency of QC testing on QC performance, although CLIA regulations set minimum standards for QC frequency (1). Specifically, QC testing is required to be performed during "each day of testing" and must follow the manufacturer’s instructions for control testing if they exceed these requirements.
Until equivalent QC testing has been scientifically proven to supplant the need for QC or unless a control sample is run with every patient sample, traditional QC strategies will leave intervals of time during which patients are being tested, but controls are not running. Of specific concern is a tendency for most laboratories to run controls at relatively fixed intervals that are the same each day. The net effect of this strategy is to allow certain periods of time in the analytical testing process when patient testing is performed, yet QC samples are never run.
Workflow in the laboratory can be broadly separated into batch mode processes and continuous mode processes. Most of the theory of how to design a good QC strategy has been based on batch mode processes, for which the batch is often referred to as an analytical run. For batch mode processes, the 2 main design parameters that determine the performance of a QC procedure are the number of QC samples in the analytical run and the QC rules that are applied to the control sample results.
For continuous mode processes, a 3rd important design parameter that must be specified is when to schedule QC testing. During the operation of a continuous mode process, events will occur that are known to pose an increased risk of system malfunction, such as reagent change or system maintenance. QC testing is generally scheduled immediately after these events. During the operation of a continuous mode process, however, there is also the chance that an out-of-control error condition may occur unpredictably, at any point in time, and adversely affect subsequent patient samples until the error condition is detected and corrected.
We investigated different strategies for specifying when QC testing is performed in situations in which there is no expectation of an increased risk of system malfunction at any particular time during the day. All strategies are required to have the same rate of QC testing (the same average interval length between QC events). The length of time that a process is out-of-control after an error condition occurs can be divided into 2 segments: the interval from the onset of the error condition until the next scheduled QC testing time, and the interval from the 1st QC testing time after the error condition occurs until the error condition is detected. The length of the 1st time segment depends solely on the QC scheduling strategy. The length of the 2nd time segment is dependent on the type of out-of-control error condition, the power of the QC rules to detect the error, and the average length of time between QC events. Therefore, the primary outcome measure of interest in this study was the effect of QC scheduling on the expected length of time between the occurrence of an out-of-control error condition and the next scheduled QC testing time.
A 2nd outcome measure of interest was the length of time to the next scheduled QC event as a function of the time of day. Ideally, the expected length of time to the next scheduled QC event (or since the last scheduled QC event) should be independent of the time of day that a patient sample is tested. The goal is to identify strategies that have near optimal performance with respect to the lag time between the occurrence of an out-of-control error condition and the next scheduled QC testing time while also allowing QC testing to be performed at varying times throughout the day.
For some laboratory batch mode processes, QC position within the batch may play a role similar to that of QC scheduling over time in a continuous workflow system. Examples might include batch testing performed with samples on a multiple-well plate or on a wheel that holds multiple samples in numbered positions. We focus on continuous workflow testing systems and do not further discuss batch mode testing.
|
Materials and Methods
|
|---|
The different approaches to QC testing schedules that we evaluated can be categorized into 4 strategies: strategy 1, QC events scheduled at fixed time intervals; strategy 2, QC events randomly scheduled within fixed time intervals; strategy 3, QC events scheduled at random intervals; and strategy 4, 1 QC event scheduled at a random interval, followed by a series of N QC events scheduled at fixed intervals. For purposes of this study, a QC event is 1 or more assayed QC samples with subsequent QC rule evaluation.
The length of fixed intervals between QC events is denoted I. For QC events randomly placed within fixed intervals (strategy 2), the time of the QC event measured from the beginning of the interval is denoted 
I, where is a random variable with probability distribution g(), 0 
1. The time to the next QC event for randomly scheduled QC intervals (strategies 3 and 4) is defined as I(1 + 
) from the current QC event, where is a random variable with mean = 0 and probability distribution g(), 
–1.
A convenient probability distribution to use for the random scheduling of QC events is the beta distribution, denoted Beta(a,b). The beta distribution is defined over the range (0,1). When the 2 parameters are equal, the distribution Beta(a,a) is symmetric, with mean = 
and variance = 1/[4(2a + 1)]. Beta(1,1) is equivalent to a Uniform(0,1) distribution. As the parameter a increases, the variance of the distribution decreases and the shape of the distribution looks more like a bell-shaped curve. Fig. 1
shows examples of the Beta(a,a) distribution for various values of the parameter a.
Random scheduling for strategy 2 is demonstrated by use of values randomly selected from Beta(1,1) and Beta(3,3) distributions. Random scheduling for strategies 3 and 4 is demonstrated with values that are computed as = w(2x – 1), where w is a fixed constant between 0 and 1 and x is Beta(1,1) or Beta(3,3). Thus, will have a mean = 0 and will be randomly distributed between ±w.
The primary performance measure of interest is the expected length of time between the occurrence of an out-of-control error condition and the next scheduled QC event. It is assumed that the probability of occurrence of an out-of-control error condition can be modeled by an exponential distribution with the mean time between out-of-control error conditions large compared with the interval of time between QC events. The probability that an out-of-control error condition occurs in a particular interval between 2 QC events is proportional to the length of the interval. In addition, the position within the particular interval where the error condition occurs will be approximately uniformly distributed over the interval (2). These assumptions are used to mathematically derive and confirm by computer simulation estimates for the average length of time between the random occurrence of an out-of-control condition and the next scheduled QC event.
Computer simulations set the average interval between QC events at 8 h for all of the evaluated scheduling strategies. The probability of an out-of-control error condition is modeled by use of an exponential distribution with a mean time between occurrences of out-of-control error conditions set at 30 days. For each simulated trial, the time of occurrence of the out-of-control error condition is randomly generated and QC event times are scheduled until a QC event time follows the time of occurrence of the error condition. The average difference between the time of the out-of-control error condition and the time of the subsequent QC event is computed from 100 000 simulated trials.
The secondary outcome measure is the average length of time to the next scheduled QC event as a function of time of day. For a given QC scheduling strategy, QC event times were scheduled over a 365-day period. The length of time to the next QC event was calculated at different time points throughout the day (12:30 AM to 11:30 PM in 1-h increments) for each of the 365 days, and the average length of time was computed.
|
Results
|
|---|
The expected times from the occurrence of an error condition to the next scheduled QC event for each of the 4 basic types of QC testing schedules were derived mathematically. The details of the derivations are given in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue4 . (References 7–11 are cited in the online Data Supplement.) Letting TQ represent the time from the occurrence of an out-of-control error condition until the next scheduled QC event, the expected times for the 4 general strategies were as follows:
 | (1) |
 | (2) |
 | (3) |
 | (4) |
Fixed interval QC scheduling (strategy 1) gives the minimum average length of time between the occurrence of an out-of-control error condition and the next scheduled QC event, which is equal to half of the fixed interval length. The increase in the expected time between an error condition and the next scheduled QC event for the randomly scheduled QC strategies depends on the variance of the probability distribution used for the random scheduling. A scheduling strategy that randomly selects the time intervals between QC events (strategy 3) has a shorter expected time between the occurrence of an out-of-control error condition and the next scheduled QC event than a scheduling strategy that uses the same probability distribution to randomly place QC events within fixed time intervals (strategy 2). The scheduling strategy that selects a random interval for the next QC event and then follows it with a series of QC events at fixed intervals (strategy 4) has an expected time between the occurrence of an out-of-control error condition and the next scheduled QC event that decreases as the number of fixed interval QC events following each random interval increases. Table 1
gives values for E(TQ) for a number of different QC schedules covering the 4 strategies. In all cases the average interval between QC events is 8 h.
View this table:
[in this window]
[in a new window]
|
Table 1. Expected time from the occurrence of an error condition to the next scheduled QC event for different QC scheduling strategies.
|
|
Assuming that a Beta(a,a) distribution is used to generate the random time intervals for strategy 4, then the expected time between the occurrence of an out-of-control error condition and the next scheduled QC event is given by the following equation:
 | (5) |
This strategy allows for 3 design parameters that can be chosen to balance the increase in E(TQ) and the degree of variability in the time of day that QC events are scheduled: the parameter for the Beta distribution (a), the half width of the random interval (w), and the number of fixed QC intervals following each random interval (N). In general, increasing the Beta parameter, a, decreasing the interval half width, w, or increasing the number of fixed intervals, N, will decrease E(TQ) and also decrease the rate at which QC event times disperse throughout the day.
Fig. 2
gives examples of a fixed interval schedule and 2 different strategy 4 scheduling schemes that were designed so that E(TQ) was 1% greater than E(TQ) for a fixed interval strategy. In panel B, a = 2.625, w = 2 h, and N = 0. In panel C, a = 1.583, w = 4 h, and N = 5. Fig. 3
shows the expected length of time to the next scheduled QC event as a function of time of day for the 3 different QC schedules. The expected wait time to the next scheduled QC event is highly dependent on the time of day for the fixed interval schedule, but the 2 random schedules have expected wait times that are nearly independent of the time of day.
View larger version (27K):
[in this window]
[in a new window]
|
Figure 2. Examples of 3 different QC scheduling strategies.
(A), 8-h fixed intervals. (B), Intervals randomly selected using a Beta(2.625,2.625) distribution scaled to range between 6 and 10 h. (C), An interval randomly selected using a Beta(1.583,1.583) distribution scaled to range between 4 and 12 h, followed by 5 intervals of 8 h. Examples (B) and (C) are 2 different QC scheduling strategies that provide QC testing at different times throughout the day while maintaining an expected time between the occurrence of an out-of-control error condition and the next scheduled QC event that is only 1% greater than the fixed interval QC scheduling strategy.
|
|
View larger version (21K):
[in this window]
[in a new window]
|
Figure 3. The average time to the next scheduled QC event as a function of the time of day (24-h clock) for the 3 QC scheduling strategies shown in Fig. 2
.
Corresponding scheduling strategies in Figs. 2
and 3 share the same plotting symbols.
|
|
|
Discussion
|
|---|
The original goal of this study was to devise a QC testing strategy that scheduled QC events at different times throughout the day and at the same time minimized the average length of time between an out-of-control condition and the next scheduled QC event. QC samples are generally run at the beginning of each shift or at some other time that is convenient to laboratory operation. A way to overcome these restrictions is to use sampling methods that ensure that the entire system is being tested with QC. Patient samples may be run at any time of the day, so it seems logical that control testing should also be performed at any time that patients might be tested. Ideally, the length of time from the testing of a patient sample until QC testing is performed should not depend on what time of day the patient sample was tested. This idea appears to be surprisingly novel for QC practices in a clinical laboratory.
Only a few papers have appeared in the laboratory medicine literature that address the issue of when to schedule QC events. Neubauer et al., using a combination of expert opinions, computer simulations of customary cost models, and review of literature, looked at the optimal frequency of QC testing for automatic multichannel analyzers (3). They concluded that the optimal number of samples between controls should be between 30 and 100. Steindel and Tetrault used a College of American Pathologists Q-Probes study to assess QC practices in hospital laboratories (4). Their study found wide variability in QC frequency, QC rule use for data analysis in determining out-of-control events, and response to out-of-control events, highlighting the lack of standardization in the laboratory community with regard to QC procedures. Concluding that a need exists for new approaches to QC on modern automated analyzers, Steindel and Tetrault recommended the use of patient results to supplement the infrequent analyses of control materials and emphasized the need for simplification of QC systems (4). Parvin and Gronowski discussed the arbitrary definition of an analytical run in the modern clinical laboratory (5). They defined performance measures that accounted for different definitions of an analytical run within the overall QC strategy and used statistical analysis to evaluate the performance of different analytical run definitions for detecting when the process is out of control. Their analyses were based on the assumption that an out-of-control condition can occur with equal probability anywhere within a run and will remain present until detected. NCCLS guidelines discuss the concept of an "analytical run" as being defined by the time or number of analyses for which the measurement process is stable (6). The document does not define or endorse any specific QC strategy for an individual device. It describes the concepts of Manufacturer’s Recommended Run Length and User’s Defined Run Length, stating that there currently are no well-accepted methods for establishing run lengths in a more scientific manner. Furthermore, the document makes no specific recommendations regarding the location of control samples within a run, but it does discuss defining a User’s Defined Run Length as either a batch of samples or a time interval. Several alternative strategies for placing control samples within a run are discussed, but no specific strategy is endorsed.
This study was intended to address concerns regarding limitations in traditional QC scheduling practices. Specifically, there is a tendency for most laboratories to run QC at fixed points in time that do not vary from day to day. This type of QC testing strategy leaves large gaps of time during which patient samples are run but QC testing is never performed. Traditional QC testing strategies therefore are at least theoretically in violation of the spirit of the CLIA regulations (1). CLIA regulations explicitly state "Test control materials in the same manner as patient specimens." Furthermore, the regulation states that the control procedures must "Monitor over time the accuracy and precision of test performance that may be influenced by changes in test system performance and environmental conditions, and variance in operator performances."
Out-of control situations are relatively infrequent events. It is not possible to accumulate sufficient real-time QC data for comparing the efficacy of different testing strategies. We used both statistical modeling and computer simulation to compare a variety of different QC scheduling strategies: with testing at fixed time intervals, random time intervals, and a combination of both. Our findings demonstrate that fixed time intervals between QC events will detect persistent out-of-control situations sooner on average than other QC scheduling strategies. If a laboratory is interested only in minimizing the expected time to detect an error condition, then a fixed interval QC scheduling strategy should be used. However, when other issues are also of concern, such as assuring that the expected length of time until QC testing is performed is independent of the time of day that a sample is tested, then this work shows how a random QC scheduling strategy can accomplish this. It should be noted that employing a fixed time interval between QC events that is not evenly divisible into a 24-h day will also accomplish the goal of varying the time of day of QC events, but in a nonrandom way.
It might appear that a QC scheduling strategy that randomly places QC events within fixed time intervals (strategy 2) and a QC scheduling strategy that randomly selects the length of time to the next QC event (strategy 3) are effectively equivalent. Although both of these strategies accomplish the goal of randomly distributing QC events throughout the day, the performance of the 2 approaches is clearly different with respect to the expected length of time between the occurrence of an out-of-control error condition and the next scheduled QC event.
To be workable, any strategy that involves changing the time of day when QC is scheduled will require some type of automated system to remind a technologist when to perform QC. Such a system should be easily programmable into existing laboratory instrumentation or laboratory information systems if the laboratory community adopts this strategy as a standard practice.
Finally, this work demonstrates the valuable role that statistical thinking and theory can play in laboratory QC design and evaluation. The insight gained from the theoretical derivations of the expected lengths of time from the occurrence of an out-of-control error condition to the next scheduled QC event for the various QC scheduling strategies ultimately led to the discovery of a design strategy that can balance the 2 goals of minimizing the expected time from the occurrence of an out-of-control error condition to the next scheduled QC event and enabling the scheduling of QC events throughout the day.
|
References
|
|---|
- . Department of Health and Human Services. Centers for Medicare and Medicaid Services Centers for Disease Control and Prevention. 42 CFR Part 493 Medicare, Medicaid, and CLIA Programs; Laboratory Requirements Relating to Quality Systems and Certain Personnel Qualifications; Final Rule. Fed Regist 2003;68:3653-3657.
- Reynolds MR, Amin RW, Arnold JC, Nachlas JA.
charts with variable sampling intervals. Technometrics 1988;30:181-192.[CrossRef][ISI] - Neubauer A, Wolter C, Falkner C, Neumeier D. Optimizing frequency and number of controls for automatic multichannel analyzers. Clin Chem 1998;44:1014-1023.[Abstract/Free Full Text]
- Steindel SJ, Tetrault G. Quality control practices for calcium, cholesterol, digoxin, and hemoglobin: a College of American Pathologists Q-probes study in 505 hospital laboratories. Arch Pathol Lab Med 1998;122:401-408.[ISI][Medline]
[Order article via Infotrieve]
- Parvin CA, Gronowski AM. Effect of analytical run length on quality-control (QC) performance and the QC planning process. Clin Chem 1997;43:2149-2154.[Abstract/Free Full Text]
- . NCCLS. Statistical Quality Control for Quantitative Measurements: Principles and Definitions; Approved Guideline—Second Edition 1999 NCCLS Wayne, PA. .
- Lorenzen TJ, Vance LC. The economic design of control charts: a unified approach. Technometrics 1986;28:3-10.[Medline]
[Order article via Infotrieve]
- Rahim MA. Determination of optimal design parameters of joint and R charts. J Qual Technol 1989;21:21-70.
- Yu FJ, Jiang LH. Optimization of design parameters for
control charts with multiple assignable causes. J Appl Stat 2006;33:279-290.[CrossRef][ISI] - Evans M, Hastings N, Peacock B. Statistical Distributions, 3rd ed 2000:77-81 John Wiley New York. .
- Costa AFB. charts with variable parameters. J Qual Technol 1999;31:408-416.
Top
Abstract
Introduction
