Clinical Chemistry Siemens Point of Care - Urinalysis
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
 QUICK SEARCH:   [advanced]


     


Clinical Chemistry 47: 322-330, 2001;
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (8)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Parsons, P. J.
Right arrow Articles by Matte, T. D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Parsons, P. J.
Right arrow Articles by Matte, T. D.
Related Collections
Right arrow Laboratory Management
Right arrow Drug Monitoring and Toxicology
(Clinical Chemistry. 2001;47:322-330.)
© 2001 American Association for Clinical Chemistry, Inc.


Articles

Evaluation of Blood Lead Proficiency Testing: Comparison of Open and Blind Paradigms

Patrick J. Parsons1,a, Andrew A. Reilly1, Debra Esernio-Jenssen2,1, Lloyd N. Werk3, Howard C. Mofenson4, Noel V. Stanton5 and Thomas D. Matte6,2

1 Wadsworth Center, New York State Department of Health, PO Box 509, Albany, NY 12201-0509.

2 Division of General Pediatrics, North Shore University Hospital, Great Neck, NY 11021.

3 Division of General Academic Pediatrics, Nemours Children’s Clinic, Arnold Palmer Hospital for Children and Women, Orlando, FL 32806.

4 Long Island Poison Control Center, Winthrop University Hospital, 259 First St., Mineola, NY 11501.

5 Wisconsin State Laboratory of Hygiene, 2601 Agriculture Dr., PO Box 7996, Madison, WI 53707-7996.

6 Department of Environmental and Community Medicine, University of Medicine and Dentistry of New Jersey, 675 Hoes Ln., Piscataway, NJ 08854-5635.
a Author for correspondence. Fax 518-473-7586; e-mail patrick.parsons{at}wadsworth.org.


   Abstract
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 
Background: Most proficiency testing (PT) programs operate with an open design in which clearly identified performance samples are distributed directly to participating laboratories on a shipping schedule announced in advance. In this study, we examine the effectiveness of assessing clinical laboratory performance for blood lead with an open PT by comparing its results with a double-blinded testing protocol.

Methods: Aliquots from up to 72 blood lead performance pools from the New York State Department of Health and the Wisconsin State Laboratory of Hygiene were disguised as routine patient specimens and submitted in two phases to up to 42 certified clinical laboratories for blood lead analysis. These 42 laboratories also received aliquots of the same performance samples for blood lead analysis directly from the "open" PT program provider.

Results: Data reported under blind and open strategies were scored against acceptable target ranges using the Clinical Laboratory Improvement Amendments of 1988 (CLIA ’88) criteria established for blood lead, i.e., ± 0.19 µmol/L (± 4 µg/dL) or ± 10%, whichever is greater. Performance differences between the strategies were also assessed. We found that 17.7% of all blind PT results were classified as unacceptable compared with only 4.5% of open PT results (P <0.001). In phase 1, 13 of 22 laboratories (60%) exhibited a statistically significant difference (P <0.05) between their blind and open PT performances, although for 6 laboratories the poorer blind performance may not necessarily have led to unsuccessful PT participation under CLIA ’88 criteria. Seven (32%) laboratories had unsuccessful aggregate performance (<80%) under blind testing while maintaining successful performance in open testing. Of these seven, two had gross discrepancies motivating further investigation.

Conclusions: The data suggest that although ~60% of clinical laboratories make special efforts to improve analytical performance on open PT samples relative to performance achieved for routine patient specimens, in most cases the differences are clinically insignificant and would not likely affect cumulative PT performance. Occasional use of blind PT may deter the inclination to treat performance samples more carefully.


   Introduction
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 
In 1991, the CDC issued a revised statement on preventing lead poisoning in young children (1), in which the blood lead (BPb)3 concentration deemed harmful to children was lowered from 1.21 to 0.48 µmol/L (25 to 10 µg/dL).4 At this lower BPb threshold, the erythrocyte protoporphyrin test, which is a biomarker of lead exposure, becomes unsuitable for lead screening purposes because of its poor diagnostic sensitivity and specificity (2); therefore, a direct BPb test is recommended for both screening and diagnostic purposes. The CDC recently updated their guidance and advice on childhood lead poisoning (3) and reiterated that screening should be carried out using a direct BPb test. The impact of the CDC’s 1991 statement on analytical laboratories was substantial because it required transition from a relatively simple test for erythrocyte protoporphyrin, typically performed on a point-of-care screening device (hematofluorometer), to a much more complex test that has traditionally been the domain of certified clinical laboratories.

In the US, clinical laboratories are required to participate satisfactorily in a proficiency testing (PT) program for BPb approved under the Clinical Laboratory Improvement Amendments of 1988 (CLIA ’88) (4). The acceptable range for results reported to BPb PT schemes accredited under CLIA ’88 is ± 0.19 µmol/L (± 4 µg/dL) or ± 10% of the established target value (whichever is greater). Before 1992, the acceptable range for BPb was less stringent, ± 0.29 µmol/L (± 6 µg/dL) or ± 15%. This range is still used by the US Department of Labor, Occupational Safety and Health Administration to evaluate the performance of occupational health laboratories providing BPb monitoring services. The more demanding PT performance required under CLIA ’88 is based on a consensus of what modern analytical methods for BPb are capable of producing in terms of accuracy and precision, and are intended to support national and local public health policy toward childhood lead poisoning. Under CLIA ’88, a certified BPb laboratory must report satisfactory results for at least four of five (80%) challenges per test event. Failure to maintain a minimum score of 80% for at least two of three consecutive test events is considered unsuccessful PT performance and can lead to sanctions that may include revocation of the laboratory’s certification to test patient samples. Although under CLIA ’88 the range for acceptable BPb results reported to approved PT schemes is tighter than required previously, it is still more lenient than external quality assessment schemes operated in Canada (Le Centre de Toxicologie du Québec, Programme de Comparisons Interlaboratoires, Laboratorie de Toxicologie-CHUL, 2705 Boul. Laurier, Sainte-Foy, Quebec, Canada G1V-4G2) and in the United Kingdom [Trace Elements External Quality Assessment Scheme (TEQAS), Centre for Clinical Science, School of Biological Sciences, University of Surrey, Guildford, Surrey, United Kingdom GU2 5XH], where good laboratory performance at a BPb concentration of 0.48 µmol/L (10 µg/dL) is expected to be ± 0.05 µmol/L (± 1 µg/dL), and acceptable performance is considered ± 0.15 µmol/L (± 3 µg/dL).

BPb laboratories have improved analytical performance over the last two decades by upgrading their analytical instrumentation and adopting more rigorous internal quality-control practices (5). The role of external PT within the laboratory’s overall quality assurance strategy is generally regarded as useful, but it can be a cause for concern because of the regulatory consequences of unsuccessful performance (6). A valid criticism of current PT program arrangements is that performance samples are not truly blinded; they are clearly identified as PT samples, and shipment schedules are provided to participating laboratories well in advance. Thus, laboratory technical staff are fully cognizant of, and are prepared to analyze, PT performance samples. There may be a tendency to treat PT samples with more care or to analyze them using more experienced personnel, more repeated testing, or with different analytical protocols than routine patient specimens receive.

Under CLIA ’88, the clinical laboratory director and the individuals performing the test are required to sign an attestation statement on the PT program report form that PT samples are tested in the same manner as patient samples (4). Moreover, it is the policy of some state PT programs that repeated testing or analysis of PT samples is not permitted unless the laboratory performs the same repetitive testing or analysis on similar non-PT samples. There has been criticism of some PT programs for providing performance samples/challenges that do not truly resemble patient specimens, for example, when a reconstitution step is required (7). Among the criticisms aimed at current BPb PT program designs is that they test only the analytical component of the process, leaving preanalytical variables untested (8).

In this study, we explore some of these criticisms by examining the effectiveness of the current open PT strategy. We describe a double-blinded assessment of clinical laboratory performance for BPb in which open PT samples, deliberately disguised as routine patient samples, were submitted to up to 42 clinical laboratories certified for this analysis. Thus, the testing laboratory was blinded not only to the BPb concentration, but also to the performance nature of the sample. We report the results of this large-scale assessment and its implications for the traditional open approach to external PT schemes.


   Materials and Methods
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 
sources of BPb PT MATERIALS
BPb PT materials were obtained from two well-established PT programs in the US operated by (a) the New York State Department of Health (NYSDH), Albany, NY, and (b) the Wisconsin State Laboratory of Hygiene (WSLH), Madison, WI.

NYSDH BPb PT program.
The NYSDH BPb PT program was established in the mid-1970s and is currently administered by the NYSDH’s Wadsworth Center (5). In 1993, the NYSDH program obtained CLIA ’88-approved PT program provider status. Under federal criteria for approved status, the program must provide five challenges (samples) three times per year. PT sample materials are based on whole blood derived from lead-dosed goats so that lead is "physiologically bound" within the red blood cells. Sample reconstitution is not required because whole blood is taken from the animal’s jugular vein, transferred into blood collection bags, and immediately preserved in dipotassium EDTA. All blood collection procedures are covered under an active protocol approved by the appropriate Institutional Animal Care and Use Committee. Approximately 400–500 mL of blood per animal are collected, and 3-mL aliquots are dispensed into additive-free evacuated glass tubes. PT materials are shipped in styrofoam boxes to participating laboratories as liquid whole-blood samples without refrigeration. This practice ensures comparability with shipment of routine patient specimens and avoids any potential matrix effects that can occur when inorganic lead is added into a blood matrix. However, the PT samples are clearly identified as such and are easily recognizable in the laboratory because the additive-free tubes have red stoppers.

WSLH BPb PT program.
The WSLH BPb PT program was established with federal grant funding in 1988 after cessation of the CDC PT program in 1987 (9). The program was modified in 1995 to provide for three special events in which the number of challenges was increased from three to five to satisfy federal requirements for approved status under CLIA ’88. In the WSLH program, lead-dosed cows are used as the source of PT samples. Routine practice at the WSLH is to withdraw an 800-mL blood pool from each animal on each of 4 subsequent days following a lead dose. Blood is collected into a blood bag containing disodium EDTA preservative, and 3-mL samples are transferred into additive-free (red-stoppered) tubes. These tubes are stored frozen for subsequent shipment to participating laboratories. Freezing causes the red blood cells to lyse, and when thawed, the WSLH PT samples appear very dark rather than bright red in color. In this form, WSLH PT samples are difficult to disguise as patient samples and would likely be rejected as a specimen for analysis, or may be identifiable as a PT sample regardless of the tube identification. To avoid this problem and to maintain blindness for the WSLH samples, aliquots of material were transferred into prewashed, lavender-stoppered tubes for distribution as blind samples. Providing WSLH PT materials prepared as intact red blood cells, rather than as lysed material, would not be expected to affect analytical methods designed to measure lead in whole blood.

The issue of a "species effect", i.e., animal vs human blood, is occasionally raised to explain cases of PT failure. However, there is no direct evidence that modern BPb methods suffer from a species effect. A major advantage in using animal blood is that it avoids potential health hazards associated with distributing human blood of unknown etiology. Animal blood is the most common sample matrix for BPb performance assessment among PT schemes in the US and overseas (10).

Taken together, the NYSDH and WSLH PT programs provide for as many as 57 BPb challenges per year. Target values for BPb in both programs are calculated only after all participant results have been reported and are based on a 90% (or better) consensus of at least 10 referee laboratories using well-established techniques. Both PT programs are also used by the Occupational Safety and Health Administration for accrediting occupational BPb testing laboratories.

study design
This 3-year observational study consisted of two phases. To achieve sufficient statistical power from a large number of PT challenges over the study period, phase 1 included 22 certified clinical laboratories participating in both the NYSDH and WSLH PT programs for BPb. The second phase was limited to 24 certified clinical laboratories participating in a single event from the NYSDH PT program.

In phase 1, clinical laboratories were selected for a blinded assessment of laboratory performance in the following manner: A list of laboratories that participate in the NYSDH BPb PT program was matched with one from the WSLH BPb PT program. A total of 41 laboratories were identified as participants in both PT programs. Each laboratory was contacted by one of us (D. J., L. W., H. M., or T. M.) and asked if they would accept specimens for BPb analysis submitted by a physician. Nineteen laboratories (46%) indicated that they were unable to provide laboratory services to individual physicians. The remaining 22 laboratories formed the basis of the phase 1 study population. They included 14 independent, 6 hospital-affiliated, and 2 public health laboratories.

Blinding was accomplished by preparing duplicate sets of PT samples in standard lavender-stoppered (EDTA) blood collection tubes labeled with fictitious patient names. Phase 1 blind PT samples were prepared and distributed as follows: Aliquots (2 mL) of the same PT whole blood were dispensed into lavender-stoppered tubes to ensure that they closely resembled routine venous specimens from patients. The lavender-stoppered tubes were prewashed with doubly deionized water to remove endogenous disodium EDTA anticoagulant because the PT material was already preserved with either dipotassium EDTA (NY) or disodium EDTA (WI) during the collection process. Collection tubes were air-dried in a dust-free cabinet set aside to dry acid-washed laboratory ware that is also used for trace element analyses. Without a prewash step, the endogenous EDTA in the tubes would have doubled the concentration recommended for anodic stripping voltammetry (ASV) techniques, and this would have adversely affected ASV method performance. One laboratory requested that blood specimens be submitted as dried blood spots using their special lead-free filter paper collection kit. To accommodate these specimen requirements, its PT samples were transferred to filter paper sheets, carefully dried in a dust-free cabinet, and submitted for BPb analysis.

Double-blinded PT samples were submitted with laboratory-specific requisition slips by one of four physicians to each respective participant laboratory. Local and state childhood lead-poisoning program personnel were notified of the study in advance and were provided with the fictitious patient names and addresses to prevent any unnecessary public-health concerns or follow-up actions, including the incorporation of fictitious data into public-health database registries. Fictitious patient test results were reported by each testing laboratory to the requesting physician, who forwarded the BPb test results to the study coordinators for comparison with (a) the matching open test results reported directly to the PT program provider by the certified laboratory and (b) the established PT target value for that sample.

In phase 2, the selection criteria were modified to target a different clinical laboratory population. We limited the selection to laboratories that analyze capillary whole-blood (micro) specimens for BPb, and because we confined phase 2 to a single test event in the NYSDH PT program, we concentrated on identifying laboratories that participate in the NYSDH PT program but not in the WSLH program. Of ~113 potential selectees, we contacted 40 phase 2 laboratories and asked if they would accept capillary blood specimens for BPb analysis from a physician’s office. A total of 24 laboratories (60%) responded positively. These included 1 public health laboratory, 14 independent, and 9 hospital-affiliated laboratories. PT samples from a single NYSDH test event were transferred into prewashed, lavender-stoppered, plastic Microtainer tubes (Becton Dickinson) and, disguised as routine screening specimens, were submitted to the participant laboratories. As above, the tubes were prewashed to remove disodium EDTA anticoagulant.

statistical analysis
We examined the discrepancies between blind and open PT results in two ways. A discrete assessment was made by grading each PT challenge as either acceptable or unacceptable based on the fixed criteria allowed under CLIA ’88, i.e., ± 0.19 µmol/L (± 4 µg/dL) or ± 10% around the established target value (whichever was greater). We then compared the frequency of acceptable results reported for the blind and open protocols. In this study, it was not possible to follow each laboratory’s consecutive performance under CLIA ’88 criteria because the number of test events provided by the WSLH program are more frequent than required under the CLIA ’88 model. Additionally, most of the challenges provided by the WSLH program are not graded for CLIA ’88 purposes. Pooling the data from two different programs also presented logistical difficulties with grading for certification purposes. Therefore, to provide an indication of cumulative performance comparable to that required under CLIA ’88 for successful performance, we used an aggregate score of 80% acceptable responses to define successful performance for blind and open PT.

A continuous assessment of the data is also desirable because the use of fixed acceptability criteria leads to a loss of information when laboratory performance is simply dichotomized as either acceptable or unacceptable. For the continuous assessment, we chose to compare root mean squared errors (RMSE), defined as the square roots of the average differences between reported BPb values and the program target values, [log (reported result) - log (target value)]2. A comparison is then possible in terms of an open PT reliability index, which is defined as a ratio of RMSE (blind) to RMSE (open), so that values greater than 1 indicate worse performance when analyzing blind PT samples (Table 1 ). Unlike the discrete assessment, which is based on fixed PT criteria, the continuous assessment may be influenced by outliers because they inflate the RMSE value.


View this table:
[in this window]
[in a new window]
 
Table 1. Continuous assessment of BPb laboratory performance.


   Results
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 
In phase 1 of this study, 848 blind PT samples were distributed to 22 certified BPb laboratories participating in up to 4 NYSDH events (20 PT samples) and up to 16 WSLH events (48 samples). Participating laboratories received open PT samples directly from the two PT programs providers at approximately the same time that the blind samples were distributed. Preliminary results have been reported in an abstract (11). In phase 2, 120 blind PT samples were distributed to 24 laboratories participating in a single NYSDH event (5 PT samples). However, only 23 laboratories reported data. Laboratories participating in phase 2 received matching PT samples directly from the NYSDH program at approximately the same time that the blind samples were distributed. In phase 1, more than one-half (64%) used electrothermal atomization atomic absorption spectrometry (ETAAS) to measure BPb; in phase 2 less than one-half (43%) used ETAAS. The remainder used ASV: one used extraction with flame atomic absorption spectrometry and another used filter paper collection with Delves-cup atomic absorption spectrometry. Although generally ETAAS is more widely used than ASV in both PT programs, the reversal in ETAAS and ASV between phase 1 and phase 2 simply reflects the different laboratory population sampled.

When individual test results were graded against CLIA ’88 criteria, we found that 17.7% of all blind PT results were classified as unacceptable compared with 4.6% of the open PT results. This difference is statistically significant (P <0.001, {chi}2 test for symmetry on 1 degree of freedom). Laboratory-specific performances are shown in Fig. 1 , where blind PT performance shown in the top panel can be directly compared with open PT performance shown in the bottom panel. Each column shows the percentage of BPb test results that were graded as satisfactory. An 80% aggregate performance threshold is shown as a solid line in both panels. This approximates the minimum cumulative performance allowed under CLIA ’88 for certified clinical laboratories performing the analysis for BPb. Each gray column denotes a laboratory-specific data set for phase 1 with each laboratory identified by an arbitrary number (1–22). Not all laboratories participated in the same number of PT events; therefore, the number of individual challenges or samples tested varied from 5 to 67 during phase 1, as shown in the top panel of Fig. 1 . Because of the matched blind-open design of the study, the number of open samples was often the same (9 laboratories) or within ± 5 (12 laboratories) of the number of blind samples. However, laboratory 7 had 9 fewer and laboratory 10 had 32 fewer open samples. The reduced number of open samples for laboratory 10 accounts for its lack of significance. Additionally, the performances of within-blind and -open categories were tested for equality. No significant differences were detected among the open performances. However, when blinded, laboratories 17, 16, and 22 (as indicated by asterisks in Fig. 1 ) had significantly more unsatisfactory results, and laboratory 19 (as indicated with the pentagon in Fig. 1 ) had significantly more satisfactory results than the combined results of the other phase 1 laboratories. The pooled data from all 22 laboratories in phase 1 are shown as a hatched column (column P1 in Fig. 1 ).



View larger version (75K):
[in this window]
[in a new window]
 
Figure 1. Laboratory-specific (gray columns) and pooled (hatched columns) PT data showing percentage of satisfactory results scored under CLIA ’88 criteria for BPb challenges distributed under blind (top panel) and open (bottom panel) strategies.

The total number of blind PT challenges evaluated as either laboratory-specific or combined into pooled data sets is shown above the columns in the top panel. Individual laboratories (phase 1) are identified with arbitrary numbers below the columns in the bottom panel; pooled data sets are identified as P1 (phase 1), P2 (phase 2), and P1+P2 (phase 1 and phase 2). The middle panel shows percentage of difference between blind and open PT scores with P values indicating increasing statistical significance from left to right. Three laboratories identified with an asterisk had significantly more unsatisfactory results than the combined results of all other phase 1 laboratories, whereas only one laboratory, identified with a pentagon, had significantly more satisfactory results (see text).

During the course of phase 1, two laboratories ceased to accept patient specimens because of unsuccessful PT performance, which necessitated their removal from the blind testing PT study. In phase 2, we elected to pool all data from 23 laboratories that returned blind results because the number of challenges in phase 2 was only five for each laboratory. The phase 2 data were pooled into two groups (shown in Fig. 1Up as two hatched columns): one group (n = 20) that did not participate in phase 1 (column P2) and another group (n = 3) that participated in both phase 1 and phase 2 (column P1+P2).

Individual differences between blind and open PT performance are shown in the middle panel of Fig. 1Up . Participating laboratories and pooled data sets are categorically sorted to show increasing discrepancy from left to right. The statistical significance (P) of the difference between blind and open PT performance is given in the middle panel with increasingly poorer performance in blind PT testing compared with open PT shown from left to right. The middle panel also shows that, collectively, both phase 1 (column P1) and phase 2 (column P2) laboratories scored significantly worse on blind challenges than on open PT challenges. However, they differed significantly from each other only on their open performances (P = 0.0047). Thirteen of the 22 phase 1 laboratories (60%) exhibited a statistically significant difference between their blind and open PT performance (P <0.05). Seven (32%) failed to obtain an aggregate performance score of 80% on blind PT samples but maintained a cumulative performance >=80% for open PT samples. Within this latter group, two laboratories with the poorest reliability indices (Table 1Up ) exhibited a large negative bias under blind testing; therefore, further investigations were warranted.

In Fig. 2 , difference plots for blind and open PT results are shown for the best 3 (top panel) and worst 3 (bottom panel) laboratories selected from the 22 phase 1 participants. The worst three (identified with an asterisk in Fig. 1Up ) had significantly more unsatisfactory results than the combined results of all other phase 1 laboratories. Each plot shows the deviation between a reported result and the program target value as a function of increasing BPb target value concentrations in µg/dL. Matching blind and open result pairs are shown with a line joining the two data points. To allow judgment of the clinical acceptability of individual deviations from the established target values, the acceptable range for PT samples under CLIA ’88 is shown as solid lines.



View larger version (37K):
[in this window]
[in a new window]
 
Figure 2. Deviations from established target values for open ({circ}) and blind (•) PT results for 6 of the 44 participant laboratories.

Target values for the PT samples are shown on the x-axes, whereas differences between the target value and test result reported by the laboratories are plotted on the y-axes. The top and bottom solid lines represent the acceptable range for BPb PT results under CLIA ’88. Paired blind and open laboratory test results for the same PT sample are indicated with vertical lines joining the values. Unpaired values are shown with the corresponding result missing (see text). Top panels show data from the three best-performing laboratories, respectively (laboratories 19, 20, and 4). Bottom panels show data from the three worst performing laboratories, respectively (laboratories 17, 16, and 22). The latter laboratories, especially laboratory 22, have significant excesses of blind values outside the upper limit under CLIA and show greater variability in blind vs open differences.


   Discussion
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 
Laboratories included in this study were not chosen randomly; therefore, the possibility of selection bias may have influenced the results. For example, we were unable to include many of the public-health laboratories enrolled in the PT programs because they usually accept only specimens from a single source and typically would not accept specimens for analysis submitted by individual physicians. This is not true of all public-health laboratories, and two did accept blind PT specimens for analysis under phase 1. This number was too small to make any statistical comparisons between them and other types of clinical laboratories. Another example of potential selection bias in phase 1 was the limitation of study participants to laboratories participating in both the NYSDH and WSLH PT programs for BPb. Because the latter is a voluntary program that is offered without charge to participants, it is possible that phase 1 was limited to a group of laboratories with a higher standard of good laboratory practices. In phase 2, therefore, we concentrated on laboratories that are required to participate in the NYSDH program for state regulatory purposes but did not participate in the voluntary program from the WSLH. Interestingly, the fraction of blind PT results graded as unsatisfactory was 23% for the phase 1 laboratories and 17% for phase 2, but the difference between the two groups was not highly significant (P = 0.0973). Although all 22 phase 2 laboratories scored 100% (5 of 5) for open PT samples, only 15 (68%) achieved 100% for matched blind PT samples. The remaining seven scored 80% for the blind PT samples, indicating that one of the blind results was outside the acceptable range. One possible explanation for the observed discrepancies is that laboratory protocols for repeating the test may depend on whether a venous or capillary blood specimen is submitted for BPb analysis. For example, an increased venous BPb result might warrant a more rigorous retesting protocol than capillary results because the latter are used only for screening purposes, or the volume of capillary blood specimens may be insufficient to permit retesting. In the blind component of phase 2, 300 µL of blood was submitted for analysis, more than sufficient for retesting. However, it is prudent to retest all capillary BPb results >0.48 µmol/L (>10 µg/dL) whenever possible to eliminate bench contamination errors and to avoid reporting a false-positive result. Nonetheless, although laboratory performance was better for open PT samples, all phase 2 laboratories would have achieved a passing grade (>=80%) for this test event based on their blind PT sample scores.

The possibility exists that some laboratories may have recognized the blind samples as PT materials. Most PT program designs challenge laboratories across the range of BPb concentrations expected in clinical practice, but increased BPb concentrations are much less prevalent today than they were two decades ago. Therefore, some clinical laboratories may encounter increased BPb concentrations only occasionally, but PT challenges almost always contain such samples. Although the five blinded PT challenges from the NYSDH PT events were separated into groups of two and three and distributed over a 2-week time frame, one phase 1 participant and one phase 2 participant suspected that PT samples were being referred for BPb analysis by another testing laboratory, a practice that is not permitted under the NYSDH program regulations. The phase 1 laboratory rejected the blind samples for analysis and informed the study physician who had submitted them. The phase 2 laboratory reported its suspicions directly to the PT program provider, who then revealed the blind nature of the study. Disclosure of the blind nature of the study to these two laboratories necessitated their elimination from further blind shipments. Thus, a valid criticism of the current study is that we cannot be sure that these two laboratories did not disclose their suspicions to other laboratories, nor can we be absolutely certain that other laboratories did not recognize the blind specimens as PT samples. Such recognition would be expected to diminish the differences between blind and open PT. These difficulties illustrate some of the reasons that blind PT schemes are not conducted routinely, and if implemented in future, such schemes will need to be carefully designed.

In one case, a laboratory (laboratory 21) reported blind PT results with a large negative bias compared with 100% acceptable open PT results. Because of the potential public-health implications, the laboratory was contacted, and a practice of treating PT samples differently was identified. In a second case, another laboratory (laboratory 22) also reported blind PT results with a large negative bias compared with 100% acceptable open PT results in both the NYSDH and WSLH programs. The laboratory was contacted, and again a practice of treating PT samples differently was identified. Both laboratories were dropped from the study because the blinded nature had been disclosed.

Two previous studies examined BPb laboratory performance under conditions where the nature of the performance samples was blinded to the testing facility (12)(13). Sargent et al. (12) submitted nine well-characterized BPb samples disguised as clinical specimens to 14 private and 4 state laboratories. BPb concentrations ranged from 0.43 to 1.59 µmol/L (9–32.9 µg/dL) with six samples below 0.43 µmol/L. These authors found a wide variation in the validity of BPb measurements reported among 18 CLIA-certified laboratories. Although most laboratories performed within the CLIA criteria used by PT programs in the US, clinically important disparities were evident, with one laboratory repeatedly making false-negative misclassifications. They also reported that up to 42% of test results for one sample with a BPb concentration of 0.43 µmol/L (9.0 µg/dL) were misclassified as false positive. Sargent et al. (12) warn clinicians to be wary of laboratory reports that do not bear out their clinical suspicions, and they advise caution in interpreting BPb results that are within ± 0.14 or 0.19 µmol/L (± 3 or 4 µg/dL) of a threshold value. On the basis of the results from our study, we would certainly concur with their advice.

In another study, Jobanputra et al. (13) examined the performance of eight BPb laboratories by submitting specimens from seven human subjects and three bovine reference materials to eight laboratories. The blood specimens were not identified as performance samples; thus, laboratories were blinded to the true objective for analysis. True BPb concentrations in test specimens were established using inductively coupled plasma mass spectrometry with isotope dilution. The BPb concentrations in all of human specimens were <0.48 µmol/L (10 µg/dL). The BPb concentrations in the three bovine samples ranged from 0.26 to 0.79 µmol/L (5.4–16.4 µg/dL). All eight laboratories reported results within 0.14 µmol/L (3 µg/dL), and there was strong reproducibility within and among laboratories, suggesting adequate accuracy and precision in this group. Only one laboratory exhibited unsatisfactory performance. The limitations of the study include its small sample size and the clustering of BPb concentrations around the 0.48 µmol/L (10 µg/dL) CDC threshold. Because clinical decisions and public-health follow-up actions are based on BPb concentration thresholds of 0.97, 2.12, and 3.38 µmol/L (20, 44, and 70 µg/dL), it is important that BPb laboratory performance also be assessed at these concentrations.

Assessment of PT programs in other areas of clinical laboratory medicine has been reported, but few studies have used blinded strategies. Richardson et al. (14) examined improvements in the quality of diagnostic microbiology over a 20-year period in Ontario through a peer-group proficiency assessment program. They compared results from an open PT program with a retrospective examination of routine work on patient specimens. They found that poorly performing laboratories lacked effective quality control, used nonstandard methods, and failed to follow in-house standard operating procedure manuals. Black and co-workers (15)(16) used a blind split-sample study design to examine the performance of bacteriology laboratories. They reported 94% correct identifications of mucoid Escherichia coli under an open PT design but only 47% under blinded conditions. Significant variations were also found for other microorganisms, suggesting that performance in open PT schemes has little to do with performance on real patient specimens. In a recent retrospective blind study design, Reilly et al. (17) compared laboratory performance in an open PT program for mycology with blind performance assessed against previously tested and reported patient specimens. They reported differential misclassification rates between open and blind PT performances for several organisms.

The design of current open PT schemes is largely driven by logistics and cost and, although not a perfect predictor of true laboratory performance, they are useful educational tools. To our knowledge, this is the first account in which PT samples have been distributed in parallel under both blind and open strategies to clinical laboratories certified under CLIA ’88. The study provides convincing evidence that, at least for a small, nonrandom selection of clinical laboratories that provide BPb testing, some effort is made to improve analytical performance in a manner that is absent for routine patient specimens. This may be possible because PT sample shipment dates are known well in advance, enabling laboratories to schedule instrument calibration and maintenance activities to coincide with arrival of PT samples. In addition, the amount of PT sample provided usually is sufficient to permit multiple testing on different instruments to provide a more robust estimate for reporting.

In conclusion, in our study of 22 laboratories in phase 1, 13 (60%) exhibited a statistically significant difference (P <0.05) between their blind and open PT performances, although for 6 laboratories their poorer blind performance may not necessarily have led to unsuccessful PT participation under CLIA ’88 criteria. Seven (32%) exhibited a highly significant improvement (P <=0.001) in aggregate performance under open testing that could be considered equivalent to successful (>=80%) PT performance under CLIA ’88 but which could not be confirmed under blind testing. Of these seven, two had gross discrepancies, motivating further investigation. The data suggest that although ~60% of clinical laboratories make special efforts to improve analytical performance on open PT samples relative to performance achieved for routine patient specimens, in most cases the differences are clinically insignificant and would not likely change PT accreditation. Occasional use of blind PT may deter the inclination to treat performance samples more carefully.


   Acknowledgments
 
This project was supported under a cooperative agreement from the CDC through the Association of Schools of Public Health (ASPH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC or the ASPH. Special thanks go to Jill Dingle, Aaron Pulaski, and Yan Yan Zong (Wadsworth Center’s Lead Poisoning Laboratory); Stacey Salem (North Shore University Hospital); and Andrea Donatelli (Winthrop University Hospital). Additional thanks go to John Qualia and Teresa Wilson for assisting with database management and computer program development.


   Footnotes
 
Departments of 2 Environmental Health and Toxicology and 3 Biometry and Statistics, School of Public Health, The University at Albany, Albany, NY 12237.

1 Current address: Division of General Pediatrics, Schneider Children’s Hospital, Suite 108, New Hyde Park, NY 11042.

2 Current address: New York Academy of Medicine, Center for Urban Epidemiologic Studies, 1216 Fifth Ave., New York, NY 10029-5293.

3 Nonstandard abbreviations: BPb, blood lead; PT, proficiency testing; CLIA ’88, Clinical Laboratory Improvement Amendments of 1988; NYSDH, New York State Department of Health; WSLH, Wisconsin State Laboratory of Hygiene; ASV, anodic stripping voltammetry; RMSE, root mean squared error; and ETAAS, electrothermal atomization atomic absorption spectrometry.

4 In the US, BPb concentrations are widely reported as µg/dL. To convert µg/dL into SI units (µmol/L), multiply by 0.04826. To convert µg/L into µmol/L, multiply by 0.004862.


   References
Top
Abstract
Introduction
Materials and Methods
Results
Discussion
References
 

  1. Centers for Disease Control. Preventing lead poisoning in young children: a statement by the Centers for Disease Control. US Government Printing Office Document 1996-533-410. Atlanta, GA: US Department of Health and Human Services, Public Health Service, 1991..
  2. Parsons PJ, Reilly AA, Hussain A. Observational study of erythrocyte protoporphyrin screening test for detecting low lead exposure in children: Impact of lowering the blood lead action threshold. Clin Chem 1991;37:216-225.[Abstract/Free Full Text]
  3. Centers for Disease Control and Prevention. Screening young children for lead poisoning: guidance for state and local public health officials. US Government Printing Office Document. Atlanta, GA: US Department of Health and Human Services, Public Health Service, 1997..
  4. . US Department of Health and Human Services. Medicare, Medicaid, and CLIA programs; regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:7002-7186.[Medline] [Order article via Infotrieve]
  5. Parsons PJ. Monitoring human exposure to lead: an assessment of current laboratory performance for the determination of blood lead. Environ Res 1992;57:149-162.[Medline] [Order article via Infotrieve]
  6. Sainato D. Proficiency testing: friend or foe?. Clin Lab News 1998;24:1-5.
  7. Weber JP. Quality in environmental toxicology measurements. Ther Drug Monit 1996;18:477-483.[Web of Science][Medline] [Order article via Infotrieve]
  8. Moyer TP, Nixon DN, Ash KO. Filter paper lead testing [Editorial] Clin Chem 1999;45:2055–6..
  9. Stanton NV. Blood lead proficiency testing: overview of the federally sponsored program in the US. J Int Fed Clin Chem 1993;5:158-161.[Medline] [Order article via Infotrieve]
  10. National Committee for Clinical Laboratory Standards. Analytical procedures for the determination of lead in blood and urine; proposed guideline. NCCLS Document C40-P. Wayne, PA: NCCLS, 1998:1–74.
  11. Reilly AA, Parsons PJ, Shahangian S, Werk LN, Esernio-Jenssen D, Salem S, et al. Effectiveness of overt proficiency testing as a regulatory tool for assessing laboratory performance for blood lead [Abstract]. In: Martin SM, Halloran SP, eds. Proceedings of the XVI International Congress of Clinical Chemistry, London: Association of Clinical Biochemists, 1998:525..
  12. Sargent JD, Johnson L, Roda S. Disparities in clinical laboratory performance for blood lead analysis. Arch Pediatr Adolesc Med 1996;150:609-614.[Abstract/Free Full Text]
  13. Jobanputra NK, Jones R, Buckler G, Cody RP, Gochfeld M, Matte TM, et al. Accuracy and reproducibility of blood lead testing in commercial laboratories. Arch Pediatr Adolesc Med 1998;152:548-553.[Abstract/Free Full Text]
  14. Richardson H, Wood D, Whitby J, Lannigan R, Fleming C. Quality improvement of diagnostic microbiology through a peer-group proficiency assessment program. A 20-year experience in Ontario. The Microbiology Committee. Arch Pathol Lab Med 1996;120:445-455.
  15. Black WA, Dorse SE, Whitby JL. A regional quality control program in microbiology. I. Administrative aspects. Am J Clin Pathol 1976;66:401-406.[Web of Science][Medline] [Order article via Infotrieve]
  16. Black WA, Dorse SE. A regional quality control program in microbiology. II. Advantages of simulated clinical specimens. Am J Clin Pathol 1976;66:407-415.[Web of Science][Medline] [Order article via Infotrieve]
  17. Reilly AA, Salkin IF, McGinnis MR, Gromadzki S, Pasarell L, Kemna M, et al. Evaluation of mycology laboratory proficiency testing. J Clin Microbiol 1999;37:2297-2305.[Abstract/Free Full Text]



The following articles in journals at HighWire Press have cited this article:


Home page
Am J Clin PatholHome page
S. Shahangian and S. R. Snyder
Laboratory Medicine Quality Indicators: A Review of the Literature
Am J Clin Pathol, March 1, 2009; 131(3): 418 - 431.
[Abstract] [Full Text] [PDF]


Home page
Clin. Chem.Home page
J. H. Shelton, C. A. Santa Ana, D. R. Thompson, M. Emmett, and J. S. Fordtran
Factitious Diarrhea Induced by Stimulant Laxatives: Accuracy of Diagnosis by a Clinical Reference Laboratory Using Thin Layer Chromatography
Clin. Chem., January 1, 2007; 53(1): 85 - 90.
[Abstract] [Full Text] [PDF]


This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (8)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Parsons, P. J.
Right arrow Articles by Matte, T. D.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Parsons, P. J.
Right arrow Articles by Matte, T. D.
Related Collections
Right arrow Laboratory Management
Right arrow Drug Monitoring and Toxicology


HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS