| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Technical Briefs |
Departments of1 Medical Education, 2 Family Medicine, and3 Laboratory Medicine, University of Washington School of Medicine, Seattle WA
aaddress correspondence to this author at: Department of Family Medicine, University of Washington School of Medicine, Box 354696, Seattle, WA 98195; fax 206-543-3821, e-mail sarakim{at}u.washington.edu
More than a decade has passed since clinical laboratories have been required by CLIA to implement competency assessment for laboratory personnel. During this time, only a few studies regarding competency assessment have been published, including a large survey of the competency assessment methods favored by laboratories in the US (1). Previous work from our group at the University of Washington Department of Laboratory Medicine has focused on computer-based methods of developing, maintaining, and assessing competency of laboratory personnel [For a recent review, see Ref. (2)]. This effort started in 1992 and for the first 8 years consisted of development of tutorial or competency assessment software and software distribution on floppy or compact disks. Starting in 2000, our work has been delivered on-line (Medical Training Solutions; Seattle, WA; www.medtraining.org).
Microscopic urinalysis has been a particular focus of our research and development regarding computer-based competency assessment (3)(4)(5). In one study, we used computer-based urinalysis competency assessment for 2 years in one institution (4). We reported the overall competency of 58 technologists and highlighted areas of conceptual difficulty encountered by these technologists. In a later study, we demonstrated the feasibility of Web-based urinalysis competency assessment, presenting competency assessment performance data from 369 technologists across 131 clinical sites (5). Both studies showed how computer programs can be used to assess the performance of an individual or group of technologists over time.
The work presented here is a significant extension of our previous work regarding online competency assessment of laboratory personnel. Since July 2002, we developed a subscription-based online competency assessment system for 14 different areas of laboratory services and implemented it in the US in 
225 laboratories. The purpose of the system is to provide an objective method of competency assessment that laboratories can use to supplement, but not replace, the more traditional approaches to competency assessment, such as direct observation. We report the results from one 6-month examination period and demonstrate how participating laboratories can use the results for benchmarking and quality improvement.
The laboratory competency assessment system is administered on-line to employees of subscribing laboratories. The system was produced collaboratively by the University of Washington Department of Laboratory Medicine and Medical Training Solutions (Seattle, WA). The minimal system requirements are Microsoft Internet Explorer, Ver. 5.0, Windows Media Player 6.4, Microsoft Windows 95, a Pentium-class processor, and a 16-bit color display capable of 800 x 600 resolution. A sound card is recommended but not required, and a printer is necessary if hardcopies of examination results are desired.
At the time the results presented below were generated, the system consisted of examinations in 14 different topics of laboratory medicine and administrative features that allowed supervisors to select examinations for each employee and view examination results. The 14 examination topics were phlebotomy, specimen processing, employee safety, general chemistry, microscopic urinalysis, toxicology, general hematology, body fluids (cerebrospinal fluid and serous fluids), coagulation, general microbiology, Gram stains, hepatitis and HIV antibody testing, transfusion services, and blood donor services. There were two examinations per year for each topic. New examinations containing all new questions were made available on January 1 and July 1, and old examinations were retired when new examinations became available.
Laboratory supervisors selected the specific examinations for each employee. For example, a phlebotomist might be required to take the phlebotomy examination and the safety examination, but no other examinations. An employee in a chemistry division of the laboratory might be required to take safety, general chemistry, urinalysis, and toxicology, but no other examinations. The competency assessment system sent e-mail reminders to participating employees. The e-mail contained a link directly to the required examinations, and by using the e-mail link, employees avoided the need to log in or remember passwords.
Each examination consisted of 10 multiple-choice questions accompanied by an associated image or video. Fig. 1
shows a typical question from the safety examination. Those who took the examinations received detailed feedback after answering each question and a performance summary at the end of the examination. Laboratory supervisors could view all results and print summaries, which could be used for documentation during laboratory inspections.
|
For this study, the data from the laboratory competency assessment system were recorded on a server and then downloaded into Microsoft Excel for analysis. We report mean scores across individual exam takers along with the median and range of test scores. Where appropriate, we performed the t-test for detecting any statistical significance in mean differences between selected institutions. The University of Washington Human Subjects Review Board approved this study.
Data analysis was performed on the set of 14 examinations (1 for each of the 14 topics) beginning January 1, 2003, and ending June 30, 2003. During this period, there were 225 participating institutions. There was a large range of usage patterns among institutions, with some using the system for one or two topics and a small number of employees, and others implementing it for all 14 topics involving a large number of employees.
The mean, median, and range of examination scores are presented in Table 1A
for each topic. The scores are rank-ordered by frequency of use. The most frequently taken examination was the safety exam, with 1868 unique exam takers from 111 institutions. The results can be used as benchmarks by any laboratory and/or their employees who completed exams. One reasonable approach to quality improvement is for laboratories that are performing poorly on particular topics to set goals for future examinations using the national data in that topic.
|
Table 1B
illustrates how the competency assessment system can be used to develop a more detailed interlaboratory comparison. For this particular illustration, we compared safety exam scores of the 10 laboratories that most frequently used the exam. However, comparisons could be based on any number of parameters, such as laboratory geographic location, laboratory size, or laboratories within the same integrated health system. Among the 10 institutions listed, 2 laboratories (labeled 1-a and 1-b) were affiliated with one integrated health system, and 2 laboratories (5-a and 5-b) with a different integrated health system. Among the 10 institutions, the mean score ranged from 82.5 (laboratory 8) to 96.0 (laboratory 1-a), and the difference in mean score between the lowest and highest scoring institution was statistically significant (t-test, P <0.001). There was no significant difference in the mean test scores between laboratories 1-a and 1-b (96.0 vs 95.0; P >0.05), and no difference between laboratories 5-a and 5-b (86.0 vs 82.6; P >0.05).
To our knowledge, this is the first implementation of online competency assessment across a substantial number of institutions for multiple topics in clinical laboratory services. The success of the implementation means that we can provide benchmark data that can be used by subscribing laboratories for quality improvement projects. Two factors account for the success of this competency assessment project: (a) enforcement of CLIA regulations regarding employee competency assessment; and (b) the ability to deliver the competency assessment over the Internet. It is now routine for organizations that accredit laboratories to ask for competency assessment records for employees, which has driven demand for objective competency assessment materials such as those described here. Online delivery is advantageous because it facilitates exams taking and collecting of data from hundreds of locations.
One of the limitations of this work is that the benchmark data in Table 1
are primarily useful to institutions subscribing to the competency assessment service, and subscriptions cost $400–1000 annually depending on the size of the subscribing institution. There is no easy way around this limitation because this University-based endeavor must be self-supporting. Another limitation is that benchmark examination scores (Table 1A
) are useful for relative, but not absolute, comparisons. We do not know whether the difference in mean scores across the 14 topics is attributable to differences in the topics or differences in the examinations. Thus, in establishing goals for improving the quality of performance in a particular topic, laboratory supervisors should seek to improve the competency of their laboratories relative to the national means or the median scores.
Future improvements to the competency assessment system will focus on expanding the number of topics to be covered and the number of subscribing institutions. In the upcoming year, we will be adding two topics: patient safety and protein electrophoresis. The future direction for our competency assessment research is to perform an item analysis for each topic to determine which concepts within a topic are most problematic for laboratories. Laboratory supervisors should find these data valuable when planning their educational activities.
In summary, we have implemented an online competency assessment system that is now used as a supplement to traditional competency assessment methods by more than 200 laboratories across the US. We are hopeful that the results generated by the system will be used in quality improvement projects that increase competency in many areas of laboratory medicine.
Acknowledgments
We thank the 20 faculty and staff from the Department of Laboratory Medicine who write and review the examination questions for the web site and the staff at Medical Training Solutions for development and distribution. The University of Washington receives licensing revenue, in relation to the competency assessment web site, from Medical Training Solutions. The authors receive no personal royalties.
References
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
