| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Molecular Diagnostics and Genetics |
1 BioAnalytical Innovation Team, LGC Ltd, Queens Road, Teddington, Middlesex. TW11 0LY, United Kingdom.
aAuthor for correspondence. Fax 44-20-8943-2767; e-mail jacquie.keer{at}lgc.co.uk.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Methods: Fifteen laboratories, including academic, clinical, and commercial organizations, were recruited into the prototype assessment exercise. A range of test samples were provided, and participants were required to extract DNA from simple matrices, perform PCR amplification, and score the samples as positive or negative by electrophoretic analysis of the amplification products. Results were requested as both gel images and a completed results table, and the performance of each laboratory was then scored on the submitted analytical results.
Results: Overall, laboratories performed the analysis successfully, with participants scoring a high proportion of the samples correctly in the two rounds of the scheme. However, not all of the laboratories were able to achieve amplification for all samples, and the performance of some laboratories was not consistent in the two rounds. In addition, several analytical problems were encountered at all stages of the process, including DNA extraction, PCR amplification, and correct recording of results.
Conclusions: The generic approach described here has enabled effective cross-sectoral benchmarking of laboratories from a variety of analytical sectors. The problems encountered by some participating laboratories highlight the need for quality control and checks at all stages of the process to ensure accuracy of results. A statistical analysis of the results (ANOVA) allowed meaningful comparison of the consistency and sensitivity achieved by laboratories, demonstrating that an effective balance was achieved between the level of data obtained from laboratories and the time expenditure required from participants.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The scheme was designed to encompass all aspects of sample analysis, from DNA extraction to target amplification and detection, including interpretation and reporting of results. The aim was to provide a range of laboratories in different sectors with the means to evaluate their performance in a widely used molecular analysis procedure by distribution of test samples and reagents and subsequent assessment of laboratory results. The ultimate purpose of the exercise was to promote good practice in nucleic acid measurements in the scientific community through independent evaluation and reporting of individual laboratory performance.
A wide range of laboratories were canvassed for their opinions on the design of the exercise, including clinical and molecular microbiology laboratories, food research institutes, contract research organizations, and a variety of academic research groups. There was no universally acceptable target analyte and matrix for use in the study; most laboratories expressed a preference to work on their usual matrices. Considering both safety issues and the level of difficulty associated with analysis of samples in complex matrices, it was decided to use a heat-killed bacterium in a suspension of phosphate-buffered saline (PBS) as the analyte for the prototype scheme.
Another issue was the number of replicate measurements required to permit detailed statistical analysis of the results from a qualitative analysis. In the final design of the scheme, a balance was reached between the amount of effort required from participants and obtaining sufficient replicates to allow assessment of the consistency and sensitivity of analysis within each laboratory. The assessment exercise was run in two rounds, with a workshop held between the two rounds, where participants were invited to meet, discuss their results, and make recommendations for modifications to the scheme for the second round.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
108, 107, and 106 colony-forming units (CFU)/mL. Six replicates per participant of each bacterial sample (1 mL) were then dispensed into prelabeled, coded, screw-capped 1.5-mL tubes. The cells were then heat-killed by incubation at 65 °C for 30 min. PBS was used as the negative control. An aliquot of the heat-killed bacteria was used for plate counting(12), and no colonies formed on agar plates over 3 days of incubation. For round 1, the samples were stored at –20 °C until shipment. However, some participants reported problems in resuspending the high-concentration frozen pellets during the DNA extraction procedure. Consequently, for round 2 the samples were not frozen, but stored at 4 °C until shipment. Each participant received six replicates of each of the following sample types: DNA-positive PCR controls, 1 mL of high-concentration bacteria (108 CFU) in suspension, 1 mL of medium-concentration (107 CFU) bacteria in suspension, 1 mL of low-concentration (106 CFU) bacteria in suspension, and negative extraction controls (buffer only).
Participants were required to extract DNA from 24 of the samples, using any in-house method appropriate for bacterial genomic DNA extraction. They were informed that the remaining six tubes contained control DNA that should simply be amplified and interpreted and not subjected to DNA extraction.
pcr reagents
All reagents for the scheme were prepared for distribution at LGC in a dedicated pre-PCR area. Deoxynucleotide triphosphates (Amersham Pharmacia) were supplied at 1.25 mmol/L. Primers directed to a region of the rfbE gene were synthesized by Sigma Genosys (forward primer, 5'-ACTTTATGACCGTTGTTTAC-3'; reverse primer, 5'-CATCTTTACTTTCCTTGTGG-3') and supplied at 10 µmol/L. One tube of Taq polymerase and 10x reaction buffer (Amersham Pharmacia) were supplied to each participant. Positive PCR controls, purified by use of a Wizard® Genomic DNA Purification Kit (Promega), were also supplied.
electrophoresis reagents
The 100-bp DNA ladder (Amersham Pharmacia) was prepared as follows: 100 µL of ladder was mixed with 100 µL of glycerol loading dye and 800 µL of 1x Tris-borate-EDTA. One milliliter of this mixture was supplied to each participant. A low-mass DNA ladder (Invitrogen) and 300 µL of glycerol gel loading dye were also provided.
preshipment testing
Before the samples and reagents were shipped, in-house tests were carried out to ensure that the reagents and targets for the PCR were performing as expected. PCR was performed with an aliquot of the test reagents to ensure that the correct amplicon (227 bp) was obtained and that there was no reagent contamination. A full extraction was also performed on a subset of samples to check that the correct concentration of bacteria had been aliquoted and that no subsequent contamination had occurred. In all cases, the expected result was obtained.
distribution
The test materials were distributed to 2 internal laboratories within LGC and 13 external laboratories: 12 in the United Kingdom and 1 in Germany. The bacterial samples were packed in a polystyrene box, and the PCR and electrophoresis reagents and PCR positive controls were placed in separate polythene bags. Reagents were accompanied by a cover letter, a method sheet, a questionnaire, an interpretation sheet, and a prepaid return envelope. For round 1, all of the reagents for UK destinations were shipped on ice via an overnight courier, whereas the sample for the German laboratory was shipped on dry ice. For round 2, the cells were not stored frozen; therefore, a small polystyrene box was prepared with dry ice to pack the PCR and electrophoresis reagents and positive controls. The cell samples were packed separately, and both test samples and reagents were then placed in a larger box containing ice packs for shipment by overnight courier. For both rounds of the scheme, the reagents for in-house analysis were packaged as described and then left at room temperature on the bench for 18 h to replicate the transport process before being unpacked and stored as directed in the instructions.
instructions to participants
In the first round, the amplification conditions were prescriptive to facilitate meaningful comparison of laboratory performance. Participants were requested to use 10% (by volume) of each DNA extract and 5 µL of the DNA samples in the PCR amplification. As a result of feedback after the first round, the amount of DNA extract to amplify was left to participants to decide in round 2. Participants were not requested to quantify the amount of DNA present in the extracted samples because the lowest-concentration bacterial samples would not have yielded measurable amounts of nucleic acid. In addition, knowledge of the amount of DNA in each sample may also have given expectations regarding the outcome of the PCR amplification, potentially influencing reporting of borderline results. The amplification conditions and the concentrations of PCR reagents to use were prescriptive in both rounds of the scheme. The amplification conditions were initial denaturation for 3 min at 94 °C, followed by 30 cycles of 94 °C for 30 s, 50 °C for 30 s, and 72 °C extension for 30 s, with a final 7-min extension at 72 °C. The specified reagent concentrations were 1x supplied PCR buffer, 0.2 mmol/L deoxynucleotide triphosphates, 0.4 µmol/L each of the forward and reverse primers, and 1.25 U of Taq polymerase; the choice of total reaction volume was left to participants, although it was constrained by total volume of reagents provided. All amplification reagents, the DNA markers, and the loading dye mixture were supplied to participants.
In round 1, the amount of amplification product to analyze on the agarose gel and the electrophoresis conditions were specified, whereas in round 2, electrophoresis conditions were not specified. In addition, participants were provided with a score sheet and asked to record whether each sample was positive or negative for the amplification reaction, together with the details of all methods used. Participants were also requested to provide details of their extraction method and laboratory set-up, including physical separation of pre- and post-PCR amplification areas and equipment used.
statistical analysis
To compare performance of participants, the scores for each laboratory were calculated (Microsoft EXCEL) as percentage correct for each analyte concentration for each round. In addition, to assess the repeatability of participants’ performance, the qualitative results were transformed into a numerical performance score for each laboratory. Specifically, all correct scores were assigned a value of 1, whereas all incorrect results were scored as 0. The resulting data set was analyzed by ANOVA (Statistica 6; Statsoft Ltd). Although ANOVA should not strictly be applied to data sets consisting of counts, this approach allowed identification of the factors that contributed to the variance observed within the experiment and also allowed those factors to be ranked according to the magnitude of their effect. The factors considered in identifying causes of variation in performance were the laboratory identity, the round of the scheme, and the concentration of analyte under investigation. In addition, the ANOVA approach enabled identification of any significant interaction effects within the experiment, including whether laboratory performance was consistent, either between rounds of the scheme or across the different analyte concentrations. A one-way ANOVA was also performed to assess any differences in laboratory performance attributable to laboratory sector or routine analyte type, comparing total overall scores for each laboratory.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Each laboratory was provided with an interpretation sheet to record their results and a questionnaire to provide information about the extraction and amplification procedures used; the responses are detailed in Table 1
. In total, 15 data sets were reported for round 1 and 13 in round 2, comprising 2 from independent internal analysts and the remainder from independent external laboratories. Thirteen data sets, including 11 from external laboratories and 2 from internal analysts, were reported in round 2. The overall results showing scores from the two rounds for each laboratory, broken down by analyte, are shown in Fig. 1
.
|
|
The overall percentage score for each analyte concentration is shown in Table 2
and gives a measure of the overall sensitivity of the analysis. The qualitative data, converted to scores, were analyzed by factorial ANOVA. Data sets of counts of samples consisted of discontinuous variables and typically had a nongaussian distribution. Use of an ANOVA approach on such nonparametric data sets risks the incorrect calculation of P values; thus, P values should be interpreted subjectively from the results. Nevertheless, the ANOVA approach is a useful tool for establishing which factors contribute significantly to the variance observed within an experiment and for identifying any significant interactions between factors. The factors considered in the present study were laboratory identity, round number, and analyte concentration. The data set was analyzed twice because two of the participants (laboratories 5 and 9) were unable to complete the second round of the analysis. ANOVA 1 (both rounds, excluding laboratories 5 and 9) identified all three factors as significantly affecting the results, in the following order: laboratory identity >> round number >> analyte concentration. The specific effect of analyte concentration is detailed in Table 2
and shown graphically in Fig. 1
. In ANOVA 2, the effects of laboratory and analyte concentration were assessed, including the results from all 15 laboratories for round 1. Both factors were significant, with laboratory again having a greater effect than analyte concentration on overall score. Thus, the laboratory performing the analysis had a much greater impact on the accuracy of the measurements than any other variable.
|
Predictably, laboratory was the major determinant affecting results because many factors, including extraction method, instrument type, and analyst experience, form part of the inherent laboratory environment and contribute to this effect. Furthermore, the majority of variation in the reported results was attributable to laboratory, analyte, or round: the error mean square values for ANOVA 1 and 2 (0.041 and 0.046, respectively) indicated that <5% of the variation was not accounted for by these three effects.
Laboratory performance was further assessed by one-way ANOVA to determine any effect attributable to either the type of laboratory or the routine analyte of the participants. No significant differences (ANOVA test assuming significance at the 5% level) in performance were found among the academic (n = 4), contract (n = 5), or public sector (n = 6) laboratories (P = 0.33). A comparison of the performance of laboratories usually working with prokaryotes (n = 9) with laboratories working with eukaryotic analytes (n = 3) or a mixture of analyte types (n = 3) revealed no significant differences (P = 0.42). Thus, those laboratories that routinely handled bacterial samples did not perform differently from laboratories using clinical or mixed sample types in the scheme.
The performance of laboratories using calibrated and noncalibrated thermal cyclers was also considered. As expected from the results of a previous study(13), the laboratories with calibrated instruments scored better results overall, although this was qualified by the observation that the noncalibrated results were based on information from only three laboratories.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
, many of the laboratories performed consistently between rounds and among analyte concentrations, scoring a high number of samples correctly overall. In addition, despite expectations that laboratories would improve in performance in the second round of the scheme, some laboratories obtained lower scores in round 2. The poor consistency between rounds observed for some participants is likely attributable in part to the different degrees of prescriptiveness in the two rounds. The majority of laboratories adopted practices more similar to their routine methods in the second round, and performance thus depended more heavily on the robustness and efficacy of in-house practices.
The majority of the observed variability within individual laboratory performance could be attributed to sensitivity issues, with the lower-concentration sample more often failing to be detected (Table 2
), and as expected the overall trend was a decrease in percentage score as the concentration of analyte decreased. However, in round 1 the high-concentration samples were detected with slightly lower frequency than the medium-concentration extracts (Table 2
). At the workshop, several of the participants reported difficulties in resuspending the high-concentration cell pellets after the initial pelleting stage in the extraction procedure, and from our own experience we recognize that this can be difficult with previously frozen cells, especially if long high-speed centrifugation protocols are used. Ineffective resuspension of the cells in the high-concentration samples may have compromised the efficiency of the lysis stage of the extraction process and, thus, of the whole extraction.
Although some of the participants failed to detect a few of the low-concentration samples, only one participant (laboratory 7) had noticeable issues with the sensitivity of PCR. In round 1, the PCR sensitivity (for laboratory 7) was comparable to that of other participants, but in round 2, many of the low-concentration samples and the positive controls were not detected (Fig. 1
). In round 2, this participant reduced the volume of the PCR to 12 µL. This small reaction volume may have been more sensitive to any slight evaporation, inaccuracies in pipetting, or presence of inhibitors in the extractions because the DNA eluate comprised >40% of the total reaction volume. Feedback on round 2 performance was positively received by the laboratory, enabling corrective action to be taken.
All but two of the participating laboratories used a commercial DNA extraction or clean-up reagent set to perform the sample extraction in round 1, and only one laboratory in round 2 used an in-house extraction method (Table 1
). The majority of reagents chosen were suitable for extraction of bacterial genomic DNA from cell suspension. Although some of the reagents used were not appropriate, either because of capacity limitations or the type of nucleic acid extracted, laboratories using these methods obtained reasonable results. Interestingly, in round 1, two laboratories used the same extraction reagents, although one of the laboratories additionally used mechanical and extra enzymatic disruption of the cells. The laboratory using the extra steps obtained good detection rates overall, whereas the laboratory using the reagents without modifications did not detect any signals in the extracted samples despite the suitability of the recommended method for both matrix and analyte. The variety of extraction methods used made quantification of the effect of specific extraction methods on performance difficult, but the effect of extraction method is likely to be a significant component of the observed interlaboratory variability.
Several common problems of PCR amplification have been well documented(14)(15), such as inhibition by components of the sample matrix or residues of the extraction reagents used and contamination of the reaction by adventitious targets. From the results of the two rounds, some examples of both problems were apparent. Complete inhibition of the PCR was observed when 10% of the extract prepared with the PrepMan reagents was used in the amplification reaction (laboratory 4). This inhibition problem was successfully overcome in the second round of the scheme because the samples were diluted before amplification (Fig. 1
), and is an example of the lowered prescriptiveness benefiting analytical performance. In round 2, laboratory 8 experienced a probable PCR inhibition because only the DNA-positive controls yielded a positive amplification result and all extracts were negative. With the lesser constraints on the PCR in round 2, laboratory 8 used 10 µL of total eluate in each 25-µL PCR, and in addition, each nominal 25-µL PCR actually contained 29.85 µL. Successful amplification of the DNA controls provided demonstrated that the increased reaction volume still supported amplification, despite containing suboptimal reagent concentrations. Thus, the higher volume of extract used in each reaction (33.5% by volume) may have caused inhibition of the amplification in the extracted samples. Advice from a leading manufacturer of commercial extraction reagents recommends that DNA extracts should comprise <20% of the overall reaction volume in a PCR to avoid potential inhibition of the reaction by high amounts of EDTA, which is present as a common component of elution buffers.
Adventitious contamination of PCR reactions is a potential problem and would be indicated by a positive signal present in the negative extraction samples. Because the negative samples were both extracted and amplified by participants, contamination of the samples could have occurred either during the preparation stages or in the PCR set-up. Several precautions can be useful in avoiding contamination during sample preparation and PCR amplification(14)(16). In round 1, three laboratories obtained false-positive results, one of which was reported as a pipetting error (laboratory 15). The second false positive may have been attributable to incorrect sample identification because many of the results reported from the PCR did not match the gel photographs of the DNA extracts of the samples also provided by the laboratory (laboratory 12). The third false positive was likely to be a true contamination event because the participant reported that the area used to extract the samples had been heavily used recently for bacterial propagation and DNA extraction. Thorough laboratory cleaning before starting the analysis overcame this problem in the second round. In addition to the "blind" negative controls included as samples for analysis, several participants included extra negative controls in their analysis, such as reagent-only controls or in-house targets that should not amplify with the PCR primers provided. All of the participants’ additional negative controls were clean, further demonstrating the low incidence of contamination observed. Contamination is much more likely to occur where a particular PCR analysis is run routinely. In this assessment, the PCR was performed only twice in each laboratory, with an interval of several months between the two rounds of the scheme. This probably explains the very low incidence of false positives observed, even in the laboratories without dedicated pre- and post-PCR areas.
In addition to problems with the practical analysis of samples, there were also some inaccuracies in interpretation of the gel results. In round 1, one of the participants reported eight of the samples wrongly on the interpretation sheet, highlighting the value of an independent check of results to ensure accuracy of the data. We found that the clarity of results provided varied considerably. Clear gel labeling is supported by newer gel documentation systems and electronic image storage and is an effective tool for ensuring accurate interpretation and reporting of data.
conclusions
This scheme has highlighted several common analytical problems encountered by a wide sector of laboratories carrying out nucleic acid-based analysis. Many of the problems encountered were general issues in molecular analysis, such as inappropriate extraction or amplification procedures, PCR inhibition or contamination, poor labeling and/or poor quality of gel photographs, and failure to record results correctly (see Table 1
in the Data Supplement that accompanies the online version of this article athttp://www.clinchem.org/content/vol50/issue9/). Adequate staff training and subsequent monitoring, experimental planning, and good housekeeping practices can overcome most of these problems.
The validity of the generic approach was supported by the comparable performance of laboratories routinely working with bacterial analytes and those used to other routine targets. The average score was 76%, and 7 of the 15 laboratories performed well consistently, with scores above average in both rounds of the scheme. No cutoff point was set below which performance was deemed unacceptable because this was subjective and could vary widely depending on the type of laboratory under consideration and the degree of measurement accuracy required. For example, clinical and forensic laboratories may be subject to more stringent assessment standards than laboratories in other sectors. Although no overall judgment was made on laboratory performance, care was taken in reporting results to laboratories to highlight possible causes for low scores and to suggest corrective action in keeping with the educational aims of the exercise.
The ability of commercial and molecular diagnostic laboratories to demonstrate the quality of their analyses is fundamental to ensuring customer confidence(17). In addition, academic laboratories are increasingly being required to demonstrate competence to receive continued funding from many UK government agencies, as described in the recently launched Joint Code of Practice for Research(18). Independent means of assessing the quality of research, such as this pilot scheme, may be valuable in fulfilling this requirement. The provision of accessible materials to facilitate independent assessment of laboratory performance could benefit laboratories that currently have no appropriate external quality assessment schemes available. Indeed, the majority of participants in the scheme endorsed the need for more widely accessible independent methods of checking analytical quality and performance. In addition to use as a performance monitor, participation in this type of scheme could be used as a learning/training tool for analysts and or laboratories that are new to molecular analysis.
|
Acknowledgments |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  S. J. Patton, A. J. Wallace, and R. Elles Benchmark for Evaluating the Quality of DNA Sequencing: Proposal from an International External Quality Assessment Scheme Clin. Chem., April 1, 2006; 52(4): 728 - 736. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. K. Hietala and B. M. Crossley Armored RNA as Virus Surrogate in a Real-Time Reverse Transcriptase PCR Assay Proficiency Panel J. Clin. Microbiol., January 1, 2006; 44(1): 67 - 70. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
, DNA controls; 
, negative samples;
, high-concentration (
, medium-concentration (
, low-concentration (