| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Laboratory Management |
1 Institute for Clinical Chemistry and 2
the Department for Statistical Analysis, University Hospital Mannheim of the University of Heidelberg, Mannheim, Germany.
3 German Society for Clinical Chemistry and Laboratory Medicine (DGKL) Reference-Institute for Bioanalytics, Bonn, Germany.
4 National Genetics Reference Laboratory (Manchester), St Mary’s Hospital, Manchester, United Kingdom.
5 Clinical Biochemistry Unit, Department of Clinical Physiopathology, University of Florence, Florence, Italy.
aAddress correspondence to this author at: Institute for Clinical Chemistry, Theodor-Kutzer-Ufer 1-3, D-68167 Mannheim, Germany. Fax 49-621-383-3819; e-mail michael.neumaier{at}ikc.ma.uni-heidelberg.de.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Methods: The participating laboratories received a 4-sample set comprising 2 DNA plasmids, a PCR product, and a finished sequencing reaction to be analyzed. Data and information from detailed questionnaires were uploaded online and evaluated by use of a scoring system for technical skills and proficiency of data interpretation.
Results: Sixty laboratories from 21 European countries registered, and 43 participants (72%) returned data and samples. Capillary electrophoresis was the predominant platform (n = 39; 91%). The median contiguous correct sequence stretch was 527 nucleotides with considerable variation in quality of both primary data and data evaluation. The association between laboratory performance and the number of sequencing assays/year was statistically significant (P <0.05). Interestingly, more than 30% of participants neither added comments to their data nor made efforts to identify the gene sequences or mutational positions.
Conclusions: Considerable variations exist even in a highly standardized methodology such as DNA sequencing. Methodologic EQAs are appropriate tools to uncover strengths and weaknesses in both technique and proficiency, and our results emphasize the need for mandatory EQAs. The results of EQUALseq should help improve the overall quality of molecular genetics findings obtained by DNA sequencing.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
DNA sequencing, regarded as the gold standard method, has high specificity and is routinely used in clinical diagnostics for mutational or SNP analysis. For example, direct sequencing analysis for the detection of mutations in the genes coding for cationic trypsinogen (PRSS1) or trypsin inhibitor (serine protease inhibitor, Kazal type I; SPINK1) in children with chronic pancreatitis (6)(7), mutations in the cystic fibrosis gene (8)(9)(10), or SNPs in the gene encoding coagulation factor VII (11)(12)(13) can have direct therapeutic consequences. In addition, the number of genetic factors investigated by sequencing techniques is growing steadily, and human genetics external quality assessment (EQA) schemes are being provided for some genetic defects in hereditary monogenic disorders e.g., by the European Molecular Genetics Quality Network (EMQN) (14)(15)(16)(17)(18). In contrast, methodologic EQA programs for quality assessment have not been systematically established for DNA sequencing. Methodologic EQAs that address single aspects of preanalytical steps, DNA preparation and amplification, and reporting were reported as early as 1998 (19), and recommendations on quality assurance of molecular methods have been published by the IFCC (20).
To investigate methodologic influences on the most common techniques in molecular diagnostics, the European Commission has funded, within the specific support action of life sciences, genomics, and biotechnology for health, a project (EQUAL) to develop molecular EQAs for qualitative analysis, quantitative PCR, and DNA sequencing, designated EQUALqual, EQUALquant, and EQUALseq, respectively.
We developed EQUALseq, focusing on different issues: (a) length and quality of DNA sequences obtained from plasmid DNA; (b) length and quality of DNA sequences obtained from a PCR product; (c) purification of a finished sequencing reaction and subsequent sequencing run; and (d) data interpretation. Here we present the results of this first pilot EQA study.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
With respect to the material shipped to the participants, we provided plasmid DNAs, a purified PCR product, and a finished sequencing reaction to be purified and analyzed (see section below).
Each laboratory received a set of 7 vials (3 containing primers, 4 with the samples), color-coded to avoid mistakes. Detailed instructions were enclosed in set 1 and were also available online on the homepage of the European Communities Confederation of Clinical Chemistry (http://www.ec-4.org/equal/10_equal_seq.htm).
sample preparation
Primers.
Primers previously quality-controlled by mass spectrometric analysis were purchased from MWG-Biotech. Appropriate aliquots were dried and labeled.
Primers provided for sequencing were as follows: sequencing primer 1 (100 pmol, dried pellet; 5'-AACAGCTATGACCATG-3') to use in the sequencing reaction with sample 1; sequencing primer 2 (100 pmol, dried pellet; 5'-CATGGTCCTGCTGGAGTTCGTG-3') to use in a sequencing reaction with sample 2; and sequencing primer 3 (100 pmol, dried pellet; 5'-GCCTCAGAGCAGGACCTTGG-3') to use in a sequencing reaction with sample 3.
Plasmid DNAs.
All plasmids used in this trial were transformed in DH5
and isolated from overnight cultures by use of the Nucleo-Bond PC 500 EF Kit (Macherey-Nagel). Quality was assessed by agarose electrophoresis, and quantity was assessed by spectrophotometry.
PCR product.
Sample 3 was generated at the coordinator’s laboratory by use of a DNA sample from a patient heterozygous for 2 coding mutations in the HFE gene (H63D and S65C) and for 1 homozygous mutation in intron 2 (IVS2 + 4) (21)(22). This PCR product was generated in a reaction containing 2 µL of Primer FOR (10 pmol/µL; 5'-GCCTCAGAGCAGGACCTTGG-3'); 2 µL of Primer REV (10 pmol/µL; 5'-CAGCTGTTTCCTTCAAGATGC-3'); 5 µL of 10x buffer; 1.6 µL of 50 mM MgCl2; 1 µL of 10 mM nucleotide mixture (Eppendorf); 0.4 µL of Platinum Taq DNA polymerase (5 U/µL; Invitrogen); 280 ng of patient DNA; and 34 µL of H2O. The amplification was carried out on a Mastercycler gradient (Eppendorf) with initial denaturation at 95 °C for 120 s, followed by 35 cycles of 95 °C for 30 s, 62 °C for 30 s, and 72 °C for 30 s, and final elongation at 72 °C for 600 s. The amplicon was purified by use of a QIAquick PCR Purification Kit (QIAGEN) to remove the primers and byproducts. The concentration of the PCR product was determined by spectrophotometry, and sample 3 was aliquoted and subsequently dried as 300-ng aliquots.
sequencing performed at the coordinator’s facility
Sequencing reactions were performed with the ABI PRISM® BigDye® Terminator v1.1 Ready Reaction Cycle Sequencing Kit (Applied Biosystems) as follows: 2 µL of Ready Reaction Mix; 1 µL of BigDye Seq. Buffer (5x); 0.25 µL of primer 1 (10 pmol/µL); 200 ng of plasmid DNA; and 4.75 µL of water. The sequencing reaction was run on a Mastercycler gradient (Eppendorf) with the following program: initial denaturation at 96 °C for 60 s, followed by 25 cycles of 96 °C for 10 s, 50 °C for 30 s, and 60 °C for 4 min. Sequencing reactions were subsequently purified by use of the DyeExTM 2.0 Spin Kit (QIAGEN) according to the manufacturer’s recommendations. A 4-µL portion of this purified probe was mixed with 16 µL of HiDi buffer and run on an ABI PRISM 310 Genetic Analyzer equipped with a 47 cm x 50 µm (i.d.) 310 capillary filled with POP6 polymer (all from Applied Biosystems). The injection time was 30 s, and the running time was 50 min.
For sample 4, no purification of the sequencing reaction product was carried out because this sample was provided to the participants for purification and sequencing only.
sample features
Sample 1.
The purpose of sample 1 was to check for the sequencing reaction performance of the participating laboratories, including purification and result reading. We provided 2 µg of a commercial cloning and sequencing vector (pCR TOPO2.1; Invitrogen) containing a 2355-bp insert. The sequencing primer was provided.
Sample 2.
Sample 2 was a mixture of 2 different DNA plasmids (recombinant and nonrecombinant) generated to provoke comments or notes/interpretation from the participants with respect to template quality. Specifically, sequencing of sample 2 with primer 2 (GFP-specific sequence in the vector backbone pEGFP-C2; BD Clontech) produced ambiguous sequence reading downstream of the EcoR1 restriction site used for cloning of the 560-bp full-length cDNA of the murine small GTPase Rab5. As in sample 1, 2 µg of the plasmid mixture was provided to the participants together with sequencing primer 2.
Sample 3.
Sample 3 was 300 ng of a purified 310-bp PCR product obtained from a patient with clinically significant heterozygous mutations in the HFE gene. A primer was sent for performing the reaction as well. Participants were asked to identify the sequence.
Sample 4.
Sample 4 was generated from sample 1 as a finished sequencing reaction (see above). After purification and electrophoresis, the participants were asked to resend the remainder of the sample to the coordinator’s laboratory for further investigation and reevaluation.
instructions given
Participants received detailed instructions for actions to be performed on all materials to be analyzed. Concerning samples 1–3, the participants were asked to perform the sequencing reaction and to provide the longest possible sequence without any mistakes (e.g., "N"). It was explicitly stated not to perform primer-walking and to use exclusively the primer provided by the coordinator. Identification of the sequence and additional comments should be uploaded to the EQUALseq web page.
After analysis, sample 4 was to be sent back to the coordinator for further investigation. The BigDye Terminator v1.1 Cycle Sequencing Kit is compatible with sequencers such as the ABI PRISM 310 or ABI PRISM 3100. Participants using other sequence analyzers were not required to run sample 4 as part of the EQUALseq program. Again, identification and additional comments were requested.
scoring of participants’ results
Sample 1.
To allow some degree of ranking, we developed a scoring system for technical and proficiency performance. Briefly, the sequencing reactions obtained at the coordinator’s laboratory were taken as the 100% performance reference, starting at nucleotide 70 downstream of the primer annealing sites. The objective of sequencing the samples was to provide the longest possible sequence without any mistakes or ambiguities (i.e., N). Manual sequence editing and the use of database support were allowed.
A sequencing stretch of 525 nucleotides from the starting point (result obtained at the coordinator’s laboratory) was taken as the 100% reference, on the basis of which the following basic and additional points were awarded to the participants:
The final score was determined, and the number of mistakes (maximum of 5 points) was subtracted. For example, participant EQSRO068 provided a sequence of 718 nucleotides terminated after the 5th false base, representing 14 basic points. Because 524 bases were read without error, 2 additional points were granted. Because no identification was given, no additional point could be added. Five points were subtracted for the 5 false or ambiguous readings, giving a final score of 11 points for participant EQSRO068.
Sample 2.
For sample 2, we awarded a maximum of 2 points. One point was given for a general comment on problems encountered with ambiguous readings, and an additional point was awarded for providing precise analysis of sample 2 based on the primary results.
Sample 3.
We awarded a maximum of 6 points for correct analysis of sample 3: We gave 1 point for identification of the HFE PCR product, 1 point for detection of the structure of the PCR product (e.g., genomic DNA or cDNA), 1 point for a general comment on the deviation from the wild-type HFE sequence, 1 point for identification of the H63D mutation, 1 point for identification of the S65C mutation, and 1 point for detection of the IVS2 + 4 variation.
Sample 4.
Not every laboratory was able to analyze this sample because some sequencing platforms were incompatible with the chemistry used to generate sample 4. Evaluation was carried out as described for sample 1. Our own DNA sequencing results for sample 4 (no mistake for 480 bases beyond the EcoRI cloning site) were again taken as the 100% reference. Subsequent to purification and analysis, the purified sample 4 was requested for reanalysis in our laboratory. This allowed assessment of the purification efficiencies and relative performances of the sequencers compared with the single-capillary ABI PRISM 310 instrument used in our laboratory.
EQUALSEQ questionnaire
A major objective was to compare instruments and technical procedures with the outcome in the sequencing program. We therefore asked the participants to complete a questionnaire before uploading their results to the EQUALseq web page (www.dgkl-rfb.de/). The data entry was divided into 3 sections: The first section was related to details of the methods used by the individual participants. The second part comprised questions about the equipment and procedures used. In addition, the participants were asked to give comments regarding the design of the EQUALseq program. The third part of the questionnaire was dedicated to data uploading.
All participants registered with the European Communities Confederation of Clinical Chemistry homepage (www.ec-4.org/equal/home.htm) and thereby received a user identification for all further actions. All data were stored in a backend database (4Th DimensionTM). After the data collection phase was closed, the results were exported into a spreadsheet-compatible format. After evaluation of the individual data by Chromas (Ver. 1.45; Technelysium Pty Ltd), MacMolly® Tetra (Ver. 3.9; Soft Gene/Mologen Holding AG), and Microsoft Excel 2000, the results were presented on the DGKL website as a general report and as individual report sheets (PDF files). Individual reports were accessible only through the respective user identifications.
statistical analysis
Statistical differences between groups were tested by the Mann–Whitney U-test with a nonparametric approach (SASTM software, Ver. 8.2; SAS Institute). P values <0.05 were considered significant.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
results for sample 1
The longest DNA sequence provided for sample 1 encompassed 894 nucleotides before termination after the 5th base error. Ten (23%) laboratories could determine 700 and more bases, using a single primer. The mean sequence stretch was 527 bases. Only 14 participants identified the insert correctly. However, we observed no association between the length of sequence and participant comments (see Fig. 1
; also see Table 1 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol52/issue4/ ) that would have indicated that the best performances were in part attributable to working with genetic databases.
|
Also shown in Table 1 in the online Data Supplement are lists of all of the results; these results demonstrate the diversity of the data as obtained from the participants in both the length of sequence strings and the quality of managing of raw data. Five laboratories had a high frequency of errors or made no effort to edit ambiguous base calls leading to premature termination of sequence scoring. As extractable from the original ABI traces, editing of the raw data would have generated valuable sequence information in 3 of these cases, thereby giving a better score. This emphasizes the need to edit sequence data. Independent of the sequence length, only 14 (33%) laboratories provided clean sequences without any false bases (e.g., N). Eight (19%) participants reported ambiguous or false base calls within the first 250 bases. The sequence stretches and false bases are shown schematically in Fig. 1
. It is apparent that most false bases/mistakes were generated at the end of the string, and no general difficulty could be determined on any sequencing platform with this sample.
results for sample 2
Sample 2 was included in EQUALseq to test the awareness of problems in raw data by providing a mixed, but partly identical, template. We monitored comments concerning the nature and quality of sample 2 as well as explanations offered by the participants regarding the nature of the ambiguous sequence. Remarkably, 9 (21%) laboratories did not register any kind of difficulties in this part of the EQA, whereas the majority of participants commented on this sample. Indeed, excellent specific comments were received in some cases, not only correctly interpreting the nature of the analytical problem but also identifying both plasmids by name. In one case of remarkable proficiency, the background sequence was subtracted, presumably by use of database information, to allow characterization of the cloned insert. In context with the data presented from sample 1, the results from sample 2 suggest that, although 
80% of laboratories will react to very obvious artifacts, single-base errors have a higher chance of going unnoticed.
results for sample 3
The PCR fragment representing sample 3 differed from the human wild-type HFE gene in 3 distinct nucleotide positions. It is noteworthy that 24 (57%) of the participants made no efforts for identification and that 27 (64%) and 26 (62%) of the participants did not comment on the heterozygous H63D or the S65C compound heterozygosity, respectively. However, the genotype coding for H63D and S65C was clearly present in 40% (n = 17) and 36% (n = 15) of the raw sequence datasets returned (Fig. 2
). Accordingly, 13 (31%) of the 42 participating laboratories received no points in this part of EQUALseq, and only 4 (10%) laboratories were awarded all 6 points. One laboratory reported incorrect nucleotides at every position of the sample 3 sequence. This systematic mistake can be explained only by reporting of the reverse complement sequence in 3' to 5' orientation. Only then will every base be wrong. Taken together, the high proportion of reports of genetic variants without interpretation were surprising because the participants had been asked categorically for comments on the nature of sample 3 (see Table 2 in the online Data Supplement).
|
results for sample 4
The template of sample 4 was identical to that of sample 1. However, in contrast to sample 1, the participants were provided with a finished sequencing reaction (done in our laboratory), which subsequently had only to be purified and run on the sequencer. To address the influence of the participants’ purification procedures, the purified samples were reanalyzed after being sent back from the laboratories.
Among 29 laboratories that were able to analyze both samples 1 and 4, 3 (10%) received the same score (see Table 1
, whereas 14 (48%) participants performed better with sample 1, indicating that these laboratories delivered higher quality with their own sequencing procedures. In contrast, 12 (41%) laboratories gained more points with sample 4 than with sample 1. This may indicate that optimization of their enzymatic PCR sequencing reactions may be possible. Specifically, increases of more than 3 points (i.e., 150 bases) were possible in 13 (45%) laboratories. In 16 (55%) laboratories, the differences between both samples were less than 3 points. Another result of this part of EQUALseq was that transport of completed sequencing reactions at ambient temperature does not generally influence the quality of the DNA sequencing results (see Table 3 in the online Data Supplement).
|
The differences in ranking for the individual laboratories, as calculated with the EQUALseq scoring system, are shown in Figs. 3
and 4
. Specifically, Fig. 3
refers to all laboratories, whereas Fig. 4
lists all participants that analyzed all 4 samples.
|
|
EQUALSEQ questionnaire
The first question in the EQUALseq questionnaire was the number of sequencing reactions performed per year by each laboratory. The largest group of participants (n = 14; 32.6%) performed 1000 to 4999 sequencing reactions per year (median, 2600). Overall, the minimum was 20, and the maximum was 40 000 sequencing reactions per year, with a median of 1600 (data not shown). The minimum template requirements differed from 1 to 1000 ng/µL of PCR product (median, 10 ng/µL) and 1 to 500 ng/µL of plasmid DNA (median, 100 ng/µL). The methods used (all nonradioactive dideoxy terminator methods) for sequencing were capillary sequencing by 39 participants (91%) and gel-based sequencing by 4 participants (9%).
To test whether the number of sequencing reactions performed per year possibly influenced the technical and analytical quality of the performance, we categorized the participants into 2 groups according to their number of sequencing assays per year and then assigned the points for samples 1 + 2 + 3 and sample 4 to the groups (Fig. 5
). Participants running 1000–50 000 assays annually (n = 24) performed significantly better on samples 1 + 2 + 3 than participants running <1000 assays per year (n = 19; P = 0.04). We also noted a trend for sample 4, but the differences did not reach statistical significance (n = 14 and n = 17, respectively; P = 0.06). These results suggest that experienced laboratories with higher throughput are better performers. The failure of these laboratories to reach statistical significance with sample 4 may be explained by the fact that they had to analyze a sample with an enzymatic sequencing step not optimized to their overall procedure.
|
Of the 43 participants, 37 (86%) are working with ABI PRISM Genetic Analyzers from Applied Biosystems Inc. Ten laboratories (23%) are working with a single-capillary system, whereas 29 (67%) are working with multiple-capillary analyzers (Table 1
). The median numbers (range) of annual sequencing assays performed on the different instruments are as follows: ABI PRISM 310 Genetic Analyzer, 500 (50–3000); ABI PRISM 377 Genetic Analyzer, 2700 (500–20 000); ABI PRISM 3100 Genetic Analyzer, 2750 (20–20 000); ABI PRISM 3700 Genetic Analyzer, 31 072 (120–40 000); ABI PRISM 3730 XL Genetic Analyzer, 30 000 (no range; used on only 1 participating laboratory); MegaBACETM 1000, 12 516 (31–25 000); ALFexpress, 700 (no range; used on only 1 participating laboratory); and CEQTM 8000 Genetic Analysis System, 3500 (200–4200). Users working with high-throughput analyzers (n = 29) performed a mean of 3500 sequencing reactions per year (range, 20–40 000), whereas low-throughput users (n = 14) performed 600 (50–20 000) assays per year (P = 0.0178). The results for the instruments used and the scores for samples 1 + 2 + 3 and sample 4 are shown in Table 1
. The overall median (ranges) numbers of points awarded were 12 (–5 to 26) for samples 1 + 2 + 3 (n = 43) and 8 (–5 to 18) for sample 4 (n = 31). Because of the small sizes of most of the groups, the participants were divided into 2 groups: those who used low-throughput analyzers, and those who used multiple-capillary analyzers. Regarding the points obtained from samples 1 + 2 + 3 and from sample 4, the medians (ranges) were as follows: 9 (–5 to 21) and 5 (0–15.9) for users of low-throughput analyzers and 15 (0–26) and 9 (–5 to 18) for users of multiple-capillary analyzers. The differences in points awarded between the groups were significant for samples 1 + 2 + 3 (n = 14 and n = 29, respectively; P = 0.01) but not significant for sample 4 (n = 11 and n = 20, respectively; P = 0.25). The results indicate that the number of sequencing assays and the quality of the enzymatic PCR sequencing established within each participant’s laboratory are important variables for predicting good sequencing performance. The national origins of the laboratories did not have an impact on quality, whereas the affiliations of the laboratories did: regional hospital laboratories seemed to perform somewhat more weakly than the university or research laboratories. In addition, the few private laboratories entered in this study performed well (see Fig 1 in the online Data Supplement).
Table 2
gives an overview of the reagent sets used for sequencing and lists the results from the descriptive statistical analysis. Because of the small sample size of some groups, nonparametric testing with the Kruskal–Wallis test was not appropriate. We therefore tested the groups with sample sizes 
3, using the Mann–Whitney U-test group by group. No statistically significant results were obtained for samples 1 + 2 + 3. Trends for better results were seen only for laboratories using the BigDye Terminator v3.1 Ready Reaction Cycle Sequencing Kit (Applied Biosystems) and the DYEnamic ET Dye Terminator Cycle Sequencing Kit (Amersham Pharmacia Biotech, GE Healthcare). According to the manufacturer’s statement, the BigDye Terminator v3.1 Ready Reaction Cycle Sequencing Kit is more suitable for longer sequence readings than is Ver. 1.1.
|
We also evaluated the methods for the removal of unincorporated dye terminators. A majority of participants (n = 17; 40%) preferred the ethanol precipitation method with generally good results, whereas the remainder used commercially available methods (Table 3
). Again, because of the small sample size of some groups, nonparametric testing using the Kruskal–Wallis test was not appropriate, and testing was carried out with the Mann–Whitney U-test group by group as described above. The following significant results were obtained for samples 1 + 2 + 3: Sephadex vs Millipore (both n = 6; P = 0.04), Sephadex vs QIAGEN DyeEx (both n = 6; P = 0.01), and Sephadex (n = 6) vs Centri SEPTM Spin columns (n = 3; P = 0.04). No statistically significant differences were obtained for sample 4. The removal of unincorporated dye terminators by Sephadex seems to provide better results.
|
Finally, the participants were asked, after the retrieval of their results from the web page, to give some comments and suggestions about the EQUALseq program on completion of the program. Thirty-six participants (84%) gave one or more comments, and 21 participants (49%) made some specific recommendations. Twenty-six participants (72%) regarded the EQUALseq program as "interesting" or "very interesting". Special comments included statements such as "interesting is the comparison with other labs" (8%), "it was the first quality control concerning DNA sequencing in our lab" (22%), "it was interesting to use plasmid-DNA" (6%), and "more than ABI PRISM systems should be tested" (6%), a statement that referred to sample 4. The performance of participants according to their affiliations is shown in Fig 1 of the online Data Supplement. No multivariate analysis was done because of the small sample size of some groups. The descriptive statistics indicate that private laboratories seem to perform better than university, research, and regional hospital laboratories.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
DNA sequencing is being routinely used in diagnostic EQA schemes for human genetics, e.g., through quality-control networks such as the European Molecular Genetics Quality Network (EMQN) (14)(15). However, except for single reports (23), dedicated methodologic EQA programs for DNA sequencing that concentrate on various aspects of analytical proficiency have not been initiated to the best of our knowledge.
The European EQUALseq program was designed to address methodologic procedures and analytical proficiency; 60 laboratories registered for the trial. Next to the generation of extended DNA sequences, the program encouraged the editing of raw data and interpretation as a matter of quality in the overall process. In addition, to evaluate both qualities within a DNA sequencing assay, we tried to devise a model scoring system to rank the participating laboratories in this methodologic DNA sequencing EQA. This scoring system judges the generation of long sequence strings together with the quality of manual editing of raw data and, to a lesser extent, identification of the samples. Medical interpretation or medical proficiency testing was not the scope of this methodologic EQA.
As one of the most striking results of EQUALseq we noticed a considerable diversity in sequencing performance, although 39 of 43 laboratories used capillary technology, of which 87% (n = 34) were using Applied Biosystems instrumentation and chemistry (Fig. 1
and Tables 2
and 3
). In part, these may be explained by the extraordinary difference in the amounts of template that the laboratories used for setting up their enzymatic reactions. Specifically, we found that the participants routinely used between 1 and 1000 ng/µL (median, 10 ng/µL) PCR product or between 1 and 500 ng/µL (median, 100 ng/µL) plasmid DNA to generate PCR products for sequencing runs. This suggests different degrees of optimization between laboratories using the same platform. Similarly, we found that different methods were used for cleaning up the reactions. Specifically, there were differences between classic ethanol or 2-propanol precipitation methods and commercial reagents for purification of sequencing reactions (e.g., QIAGEN or Millipore; see Table 3
), possibly leading to loss of signal intensity in some cases.
DNA sequencing is a complex multistep procedure being performed with different degrees of optimization in individual laboratories. To assess this issue, we provided the participants with both a DNA template to be run through the entire process of sequencing, purification, and electrophoresis (sample 1) and a prefabricated sequencing reaction (previously generated from sample 1 in our laboratory) to be analyzed on their instruments (sample 4). In this experiment, 14 laboratories performed better with their own procedure (i.e., sample 1 gave results superior to those of sample 4), whereas 12 laboratories performed better with sample 4 than with sample 1. The results obtained from samples 1 and 4 allowed us to dissect and compare the performances of single steps within the sequencing procedure within this EQA. We suggest that such information may provide help to laboratories to optimize their sequencing efforts and therefore should be included in future EQAs (see Table 3 in the online Data Supplement). Finally, routine experience appeared to be important for the laboratories in the overall performance in EQUALseq: participants with more than 1000 sequencing assays per year were significantly better performers.
In contrast to judging the technical aspects of raw data, we consider it difficult to judge analytical proficiency in DNA sequencing. However, we were surprised to find that many laboratories restricted their attention to the purely technical issues of the electrophoretic run. Although laboratories were asked to edit the raw sequence data in EQUALseq, many participants appeared not to understand this task to be part of the technical expert’s assessment. This explains why many laboratories that showed good technical performance fell short in their overall scores. We believe that improving what we tentatively would call "postanalytical proficiency" should be a major focus of future EQAs in DNA sequencing. Certainly, the shear amount of data generated by the present technology will require increased efforts, e.g., double-checking results and consulting of databases. The scoring system we have developed in the EQUALseq program was intended to be a first step toward quality assessment of the technical validation process. As can be seen in Figs. 3
and 4
, the differences among the 43 laboratories entered into this evaluation were considerable. National origin of the laboratories did not have an impact on quality, whereas the affiliations of the laboratories did: regional hospital laboratories seemed to perform somewhat more weakly than the university or research laboratories. In addition, the few private laboratories entered in this study performed well (see Fig. 1 in the online Data Supplement). We want to emphasize that this diversity is not attributable to differences in the methods used, because the technology used to complete EQUALseq was very standardized.
In conclusion, the European EQUALseq program shows that the methodologic aspects of DNA sequencing are crucial to uncover strengths and weaknesses in techniques and postanalytical proficiency. In our opinion, this program has demonstrated that there is considerable room for improvement. With respect to the increasing importance of DNA sequencing for clinical laboratory diagnostics, we propose that methodologic EQAs should be made mandatory for laboratories providing this service. The experiences obtained in EQUALseq may serve as an approximation to such future EQAs.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
2 Nonstandard abbreviations: SNP, single-nucleotide polymorphism; EQA, external quality assessment; and EU, European Union.
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  B. Afshar, N. K. Fry, W. Bellamy, A. P. Underwood, T. G. Harrison, and and Members of the European Working Group for Legi External Quality Assessment of a DNA Sequence-Based Scheme for Epidemiological Typing of Legionella pneumophila by an International Network of Laboratories J. Clin. Microbiol., October 1, 2007; 45(10): 3251 - 3256. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. Orlando, P. Verderio, R. Maatman, J. Danneberg, S. Ramsden, M. Neumaier, D. Taruscio, V. Falbo, R. Jansen, C. Casini-Raggi, et al. EQUAL-qual: A European Program for External Quality Assessment of Genomic DNA Extraction and PCR Amplification Clin. Chem., July 1, 2007; 53(7): 1349 - 1357. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Dorn-Beineke, P. Ahmad-Nejad, U. Pfeiffer, S. Ramsden, M. Pazzagli, and M. Neumaier Improvement of Technical and Analytical Performance in DNA Sequencing by External Quality Assessment-Based Molecular Training Clin. Chem., November 1, 2006; 52(11): 2072 - 2078. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. C. Ramsden, S. Daly, W.-J. Geilenkeuser, G. Duncan, F. Hermitte, E. Marubini, M. Neumaier, C. Orlando, V. Palicka, A. Paradiso, et al. EQUAL-quant: An International External Quality Assessment Scheme for Real-Time PCR Clin. Chem., August 1, 2006; 52(8): 1584 - 1591. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. Bakker Is the DNA sequence the gold standard in genetic testing? Quality of molecular genetic tests assessed. Clin. Chem., April 1, 2006; 52(4): 557 - 558. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
, nucleotide position is identical to Medline accession no. Z92910, wild-type human HFE gene; 
, correct reading of the sequence at the positions of known SNPs in HFE; 
, sequence error reported by participant or mutation not detected;
marked with an *, ambiguous "N" reading reported by participant. + and – indicate the presence or absence of comments on detection of the H63D, S65C, or IVS2 + 4 SNPs present in sample 3.
