| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1 Centre for Clinical Science and Measurement, School of Biomedical and Life Sciences, University of Surrey, Guilford GU2 7XH, United Kingdom.
2 Institute of Occupational, Social and Environmental Medicine, University of Erlangen-Nuernberg, 91054 Erlangen, Germany.
3 Unit of Epidemiology, Scientific Institute of Public Health, B-1050 Brussels, Belgium.
4 National Institute of Occupational Health, 2100 Copenhagen, Denmark.
5 Higiene Industrial, Centro de Seguridad y Salud en el Trabajo, Gobierno de Cantabria, 39012 Santander, Spain.
6 Laboratorio di Biochimica Clinica, Istituto Superiore di Sanità, 00161 Rome, Italy.
7 Laboratoire de Toxicologie, UFR de Pharmacie, Université de Nantes, 44035 Nantes, France.
8 Association Européenne des Metaux, 1150 Brussels, Belgium.
9 Biomonitoring Laboratory, Topeliuksenkatu 41aA, Finnish Institute of Occupational Health, FIN-02500 Helsinki, Finland.
10 MCA Laboratory, Queen Beatrix Hospital, 7101 BN Winterswijk, The Netherlands.
aAddress correspondence to this author at: Centre for Clinical Science and Measurement, School of Biomedical and Life Sciences, University of Surrey, Guildford, Surrey GU2 7XH, United Kingdom. Fax 44-1483-689979; e-mail A.Taylor{at}surrey.ac.uk.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Methods: Real results for blood lead and serum aluminum assays, reported by participants in Italian and United Kingdom EQASs, were evaluated according to individual scheme scoring criteria. The same results were then used to produce z scores using scheme-based between-laboratory SDs as the estimate of variability to determine whether simple performance-derived quality specifications produced better agreement among schemes.
Results: The schemes gave conflicting assessments of participants’ performance, and participants judged to be successful by one scheme could be defined as performing inadequately by another. An approach proposed by Kenny et al. (Scand J Clin Lab Invest 1999;59:585), which uses clinical inputs to set targets for analytical imprecision, bias, and total error allowable, was then used to elaborate quality specifications.
Conclusions: We suggest that the CLIA '88 recommendations for blood lead (± 40 µg/L or ± 10% of the target concentration, whichever is the greater) could be used as a quality specification, although a revision to ± 30 µg/L or ± 10% is recommended. For serum aluminum, a suitable quality specification of ± 5 µg/L or ± 20% of the target concentration, whichever is the greater, is suggested. These specifications may be used to compare laboratory performance across schemes.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Even within the same country, multiple criteria for the judgment of laboratory performance may exist. The criteria set for blood lead in the US by the Occupational Safety and Health Administration for occupational monitoring purposes are ± 60 µg/L or ± 15%, whichever is the greater. Clinical laboratories, however, must comply with the performance criteria for blood lead set under CLIA '88, i.e., ± 40 µg/L or ± 10%, whichever is the greater. These more restrictive criteria, enacted in 1992, were designed to support a major change in public health practice in the US in 1991, i.e., the lowering of the pediatric blood lead threshold from 250 µg/L to 100 µg/L.
In 1999, nine European organizers of EQASs for monitoring assays relevant to occupational and environmental health began formal collaboration as an EU Thematic Network. One of the main objectives of the Network was to harmonize the goals of individual schemes with respect to setting common standards for laboratory performance.
Performance of a participant on a single test item within an EQAS may be evaluated by reference to specific scheme-devised targets, to a z score, or to En numbers:
 |
 |
The objectives of this study were to look at European EQASs for blood lead and serum aluminum determinations to:
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
We used results reported by participants in the Italian and United Kingdom schemes for blood lead on control samples, where target concentrations were at or close to 100, 400, and 700 µg/L, to compare the different methods used by scheme organizers to evaluate laboratory performance for blood lead analysis. The same approach was applied to results reported by participants for serum aluminum concentrations of 100 and 150 µg/L. Of the nine schemes participating in the Network, only six monitor performance for serum aluminum.
The results reported by participants were assessed according to (a) the "performance limits" used by each scheme (7) or (b) z scores calculated using the typical between-laboratory SDs at the appropriate concentrations after outliers were excluded according to individual scheme rules, as the s value (7). The performance limits vary among schemes. In Denmark, limits are defined as a z score of ± 3. At concentrations 
100 µg/L, the s value is taken as 0.1 times the target concentration. Thus, the effective limits are at ± 30% of the target concentration (lead in blood only). In Belgium, France, Italy, and the United Kingdom, limits are set by the organizers to take into account clinical needs and analytical performance among experienced laboratories. In Germany, limits are 3 times the CV associated with the results of a group of reference laboratories. In Spain, deviations of ± 60 µg/L at target concentrations <400 µg/L and ± 15% at higher concentrations are considered acceptable (lead in blood only). The EQAS in The Netherlands provides results only and does not offer any judgments as to proficiency.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
. It is evident that there are large variations among the schemes with differences up to two- and threefold at 700 and 400 µg/L, respectively, the key concentrations for suspension of workers from occupational lead exposure. However, the differences at 100 µg/L, the key pediatric threshold that is defined as harmful, are much smaller, almost in agreement at ± 30 µg/L.
|
The typical SDs and CVs determined from the results reported by participants in the European EQASs are also shown in Table 1
. Although most laboratories participating in these European schemes use similar equipment and methodologies, it is evident that the differences in performance among participants in the schemes are quite marked: the CVs range from 7% to 21.2% at 100 µg/L, from 7% to 16.0% at 400 µg/L, and from to 9.8% to 14.3% at 700 µg/L. It has been shown that variations in performance reflect the role and responsibilities of participants and that specialist laboratories reduce bias and imprecision compared with those where the trace element activity is a minor component of the work (8). It is likely that the differences in SDs evident in Table 1
will reflect the nature of the participant laboratories in individual schemes and that this in turn will reflect the approach to occupational and environmental monitoring in the particular countries. The European Network "agreed limits" refer to the decisions taken at the Second Network Meeting (held in Rome during November 2000) as being indicative of minimum acceptable analytical performance. They were developed taking into account the performances achieved by participating laboratories and were proposed as a first attempt to demonstrate how harmonization among schemes could be obtained. These limits (Table 1
) are effectively ± 20% of the target concentrations, and equivalent ranges were applied to the five control samples used in this study to identify results indicating less than acceptable performance.
The upper section of Table 2
summarizes blood lead results obtained in different exercises carried out in the Italian and United Kingdom schemes and used here to compare the scoring systems. The target concentrations are the consensus medians after the removal of outliers (i.e., values outside the range "mean ± 3 SD" of all reported results). These outliers are, however, included in the assessments of performance.
|
The five data sets were examined to determine how many individual results would be reported as unacceptable according to the criteria used by each scheme organizer, as shown in Table 1
. The number of unacceptable results is given in lower part of Table 2
. There is little agreement among the schemes and little overlap with the European Network agreed limits. Table 2
also illustrates the varied performance associated with the laboratories of different schemes. The percentage of "poor results" for samples with approximately the same concentrations is much higher in the Italian compared with the United Kingdom scheme, which could be a consequence of the greater number of specialized laboratories that participate in the United Kingdom scheme (8). If the European Network agreed limits were applied, it can be seen that there would be a concentration-related effect, with a greater percentage of unacceptable results at 
100 µg/L compared with higher concentrations.
The variation in identifying poor performers caused by the different scheme acceptance limits is illustrated in Fig. 1
. Individual results reported for sample B (target lead concentration, 412.5 µg/L) are shown with the acceptance limits for each scheme superimposed. The huge differences among the concentrations at which the results begin to fall outside the limits associated with the schemes can be clearly seen. Similar trends were obtained for the results of the other specimens.
|
For each scheme, z scores based on the between-laboratories SDs given in Table 1
were determined for the results observed for each sample. The scheme-related z scores for sample B are plotted in Fig. 2
, and again the variations among the schemes are obvious and the number of "unacceptable" results is quite different from one scheme to another (Table 3
).
|
|
aluminum in serum or plasma
Data for assessing laboratories measuring aluminum were treated as for blood lead. A description of the procedure and the results may be seen in the data supplement to this article at Clinical Chemistry Online (http://www.clinchem.org/content/vol48/issue11/).
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
and 2
). These differences follow from the more exacting acceptance limits used by some organizers compared with others and are unacceptable when trying to ensure that the results of biological monitoring programs within the EU are comparable. Thus, harmonization of performance evaluation is needed to eliminate any equivocation. One problem in setting common quality specifications is that the target value is established in different ways in different schemes, and the uncertainty of the target value may differ between schemes. In the z score system, it is possible only to formulate common quality specifications in terms of s, if it is assumed that the uncertainty of the target value is either negligible or the same in all schemes. An alternative is to evaluate performance based on the approach used in the field of calibration by use of the En numbers, where the uncertainty of the target value (Utarget) is taken into consideration when evaluating performance. Hence, if analytical quality specifications were set in terms of a "laboratory uncertainty" (Ulab), then the En number will be a valid indicator of the fulfillment of the quality specifications. However, uncertainty of target values is not currently determined in most schemes.
Harmonization could be achieved if all schemes reported a z score as recommended by the International Organization for Standardization (4), but if the s values in the z score formula were simply derived from the typical between-laboratories SDs observed in each scheme at the target concentration, this also would give dissimilar scores. As seen in Table 1
, the results reported in some schemes are more closely grouped around the target concentration than in other schemes. Thus, z scores based on analytical data will give dissimilar outcomes unless there is agreement to use the same s values in the z score formula. When there is agreement to use the same s values, the z score will no longer represent the "consensus analytical performance" for certain schemes but will include a degree of compromise. For individual schemes, this compromise will reflect either a relaxation or an enhancement of analytical standards depending on whether the s values have been increased or decreased.
The analytical limits agreed on at the Second Network meeting in Rome bear little relationship to the performance targets in use by the majority of scheme organizers, who tend to recommend much smaller allowable deviation about the target concentration (Table 1
). The Network limits were proposed with consideration to the minimum overall performance of all participants. It may be argued, therefore, that the EQAS performance limits are unrealistic for many of the participants to achieve and that they do not represent the "analytical situation" in real laboratories. Alternatively, it may be suggested that limits, which simply reflect the broad spectrum of performance, neither eliminate poor laboratories nor reward those who are competent. It has been demonstrated that performance targets, which provide a realistic and achievable challenge to participants, have a positive effect on the overall standard of performance (10).
An alternative to simple considerations of interlaboratory variance in setting quality specifications has been developed by Kenny et al. (11) and may be useful to this discussion. This uses a systematic hierarchical approach that takes account of analytical and clinical targets. To develop quality specifications that are suitable for harmonization of European EQASs for measurement of lead in blood and aluminum in serum, we have applied this approach to our data and other observations.
goals based on current state of the art
As demonstrated by data from EQASs.
This information is given in Table 1
, and there is some consensus with CVs at 15%.
Other data.
Good performance is shown by results from reference laboratories. For example, the United Kingdom trace element Supraregional Assay Service Centres and the specialist laboratories used by the German and New York State EQASs achieve between-laboratory CVs that are <10% at 100 µg/L and <5% at higher concentrations (data from scheme reports presented in 2000 and 2001).
quality specifications set by
Regulatory bodies.
Many regulatory organizations state that laboratories undertaking measurements must meet the criteria for competence set by the EQAS organizer (see below). Specifications, with legislative authority, have been defined for certain applications, but these are usually established to eliminate very poorly performing laboratories rather than to stimulate improvement or with consideration to the clinical relevance of the assay. Thus, the German occupational biological monitoring program allows for limits that are three times the SD achieved by reference laboratories (equivalent to 22.2% at 100 µg/L, decreasing to 12.0% at 700 µg/L), whereas the US CLIA ’88 (12) defines acceptable performance for lead in blood as being within ± 40 µg/L or ± 10% of the target concentration, whichever is the greater, and the criteria set by the US Occupational Safety and Health Administration for occupational monitoring purposes are ± 60 µg/L or ± 15%, whichever is greater.
Organizers of EQASs.
Goals set by the organizers of the European schemes are indicated in Table 1
. As noted previously, some are based only on analytical grounds, whereas others attempt to also account for the clinical usefulness of the test (13)(14). These goals correspond to CVs of 8–30%, even at the higher concentration of 700 µg/L.
published professional recommendations
The US NCCLS and the US CDC currently recommend that the specification for internal quality-control limits should be ± 20 µg/L or ± 10%, whichever is greater (15)(16). To our knowledge, there are no other published recommendations from expert bodies, groups, or individuals for quality specifications in occupational and environmental laboratory medicine.
effect of analytical performance on clinical decisions
Data based on biological variation.
The observed values of a biomarker of exposure, such as lead in blood, depend on the individual exposure and are expected to change to reflect changes in exposure. However, if the extent of exposure can be considered constant, some information can be obtained on the biological variability for such a biomarker. A few studies involving individuals not occupationally exposed to lead have investigated seasonal variability of blood lead concentrations by comparing the values obtained in samples collected over a period of time (17)(18)(19)(20). When seasonal variability can be excluded, these series of data can be used to derive information on inter- and intraindividual variability of blood lead. Delves et al. (17) have reported on the temporal stability of blood lead concentrations of 21 healthy adults (14 men and 7 women), exposed only to environmental lead, in the UK between 1981 and 1982. For these individuals, 9–17 blood samples were collected serially over a period from 7 to 11 months. The mean (SD) blood lead concentration was 120 ± 22.4 µg/L (18.7%), and the intraindividual variation ranged from 1.4% to 9.1%, with a mean of 4.5%. In a series of reports, Fraser et al. (20) have suggested desirable targets for analytical imprecision (CVa), bias, and total error allowable (TEa) derived from biological variation and expressed by the following formulas:
 |
 |
 |
).
|
Data based on clinician opinions.
We are unaware of any published opinions relating to lead in blood performance criteria for occupational and environmental laboratory medicine.
effect of analytical performance on clinical outcomes in specific clinical settings
Relationships between the concentration of lead in blood and the onset of clinical signs and symptoms have been presented by various authors (21)(22)(23). Although there are differences among individuals, on a population basis it can be seen that the activities of the erythrocyte enzymes pyrimidine 5'-nucleotidase and 
-aminolevulinic acid dehydratase are inhibited at lead concentrations of 100 and 200 µg/L, respectively, that erythrocyte porphyrins are increased and nerve conduction velocity is decreased at 300 µg/L, that urinary -aminolevulinic acid is increased and sperm morphology and function are affected at 400 µg/L, and that hemoglobin concentrations begin to decrease at 500 µg/L. Thus, although it may be a relatively crude observation, most biochemical and health effects in adults do appear at apparently regular intervals of 100 µg/L. Therefore, for this assay a suitable quality specification for total error should at least allow differentiation between blood lead concentrations that are 100 µg/L apart; therefore, the maximum TEa should not exceed ± 50 µg/L. However, more restrictive limits may be required at the lower concentrations, which are critical for the monitoring of pediatric populations. Better performance at these lower concentrations is easily achievable by most laboratories and is desirable to reduce misclassification.
proposed standard of performance for the measurement of lead in blood
From the previous discussion, it appears that a performance target for the determination of lead in blood could be established taking into account biological variability and the effect of analytical performance on clinical outcomes. From the former criterion, an indication for a minimum performance for TEa of 12.8% was derived from the data available at a concentration of 120 µg/L. From the latter, it can be stated that if the total maximum acceptable error should be ± 50 µg/L, the target SD for a group of laboratories performing blood lead analysis should be 
16 µg/L if all of them were to achieve acceptable results. According to this, 95% of the laboratories would achieve results within ± 32 µg/L from the target value and 99.7% of them would provide results within ± 48 µg/L from the target value. Fraser (20) remarked that, for the purposes of clinical investigations, it is not necessary to achieve accuracy that is better than the TEa. On this basis, the performance target does not alter with time (unless there is a change in clinical constraints). It should be noted however that (a) there is evidence for modern analytical methods to allow better performance (24), (b) many laboratories are already able to achieve better performance (Table 2
), and (c) professional organizations have recommended more rigorous targets (for internal quality-control purposes). The CLIA '88 criteria for acceptable results, i.e., ± 40 µg/L or ± 10% of the target concentration, whichever is the greater, correspond closely to the TEa calculated above, and we suggest that the CLIA '88 targets could be used as an analytical goal, although a revision to ± 30 µg/L or ± 10%, whichever is the greater, is recommended.
proposed standard of performance for the measurement of aluminum in serum
A similar discussion applied to measurement of aluminum in serum is presented in the data supplement to this article at Clinical Chemistry Online (http://www.clinchem.org/content/vol48/issue11/). From that information we propose that for this assay, a suitable quality specification for TEa would be ± 5 µg/L or ± 20%, whichever the greater. This is equivalent to a TEa of ± 5 µg/L up to 25 µg/L and a variation of ± 20% for higher concentrations.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
1 Nonstandard abbreviations: EQAS, external quality assessment scheme; EU, European Union; and TEa, total error allowable. 
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  J. Arnaud, J.-P. Weber, C. W. Weykamp, P. J. Parsons, J. Angerer, E. Mairiaux, O. Mazarrasa, S. Valkonen, A. Menditto, M. Patriarca, et al. Quality Specifications for the Determination of Copper, Zinc, and Selenium in Human Serum or Plasma: Evaluation of an Approach Based on Biological and Analytical Variation Clin. Chem., November 1, 2008; 54(11): 1892 - 1899. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Patriarca, M. Castelli, F. Corsetti, and A. Menditto Estimate of Uncertainty of Measurement from a Single-Laboratory Validation Study: Application to the Determination of Lead in Blood Clin. Chem., August 1, 2004; 50(8): 1396 - 1405. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
