Collection Systems

Findings

We present information on 4 collection systems, none of which was reported in a peer-reviewed evaluation article (Tables 3 and 4; Evidence Table 1; Appendix H). We found descriptions of 4 commercially available stand-alone systems that can be used by public health officials, fire departments, hazardous materials teams, law enforcement, and facility owners to collect environmental samples for ongoing surveillance of high-risk locations (e.g., public buildings and airports) or to monitor clean-rooms: Smart Air Sampler System (SASS) 2000 Plus™ Chem-Bio Air, BioCapture™, SpinCon^®, and Portable High-Throughput Liquid Aerosol Air Sampler System (PHTLAAS). We recognize that other collection systems exist and may be currently available; however, we present all of the systems for which we were able to find publicly available reports. Other systems, that include a collection system as part of an integrated collection-detection-identification-communication system, are presented later in this chapter.

Table

Table 4. Collection systems.

The purpose of most of these systems is to collect aerosol environmental samples for use by first responders and for monitoring workplace exposures. We found no collection systems specifically designed for clinicians to obtain a clinical sample from patients (symptomatic or asymptomatic) with suspected exposure to a biothreat agent. Instead, most identification assays can be used on microbiological samples from nasal swabs, sputum, urine, blood, and cerebrospinal fluid collected in standard culture tubes. The descriptions of the 4 collection systems all report that they are portable, and the weight and size dimensions provided seem to justify this claim. The collection efficiency of the devices ranged from 0.5 to 10 microns. (Note that the size of the causative agent of anthrax is 1 to 5 microns, of smallpox is 0.15 to 0.3 microns, of plague is 0.5 to 2 microns, and of tularemia is 0.125 to 0.7 microns. ³⁸ ) Only the SpinCon^® reports provided a flow rate, which for this device was 1000 liters per minute (L/min).^{10, 36, 39-41} None of the reports described methods for maintaining the security of the sample.

The only evaluation information on any of these systems was provided by the manufacturer of the BioCapture™ device, which has been used by fire departments in Seattle, Los Angeles, and New York, among others, and was evaluated at Dugway Proving Ground. ⁴² The collection efficiency of BioCapture™ was reported to be 50 to 80 percent relative to the All Glass Impinger standard and 60 to 125 percent relative to the Slit Sampler Standard. ⁴² (We found no additional evaluation information about these standard devices.)

Particulate Counters and Biomass Indicators

Rapid Identification Systems

Background

Traditional methods for the detection and identification of microorganisms, viruses, and/or their products lack the speed and sensitivity to be useful in the field or at the bedside. ⁴⁹ The systems likely to be of the greatest use to clinicians and public health officials for the identification of biothreat agents are those that provide a result within minutes. Table 6 describes rapid identification systems and is organized according to the type of identification technology: antibody-based methods, nucleic acid-based methods, mass spectrometry, and others. We refer interested readers to the reviews of rapid identification systems that inform the following discussion.^{10, 36}

Table

Table 6. Rapid identification systems.

Antibody-based systems

Antibody-based systems use antibodies developed to recognize specific targets on either antigens or cells of interest to detect potential pathogens.^{10, 36} An advantage of these systems is that the use of antibodies confers high specificity^{10, 36} The antigen-antibody binding can be monitored directly or indirectly. For example, sandwich assays use a second antibody, labeled with a fluorescent dye that binds to either the antigen itself or probe antibody to monitor antigen-antibody binding. ¹⁰ The detection thresholds of these methods vary between 10³ to 10⁴ microbial cells per milliliter (mL). ¹⁰ Technical problems with antibody-based sensors include nonspecific binding (which can lead to false positive results), cross reactivity, and degradation of the antibodies over time (which can lead to false negative results). ¹⁰ Despite these technical problems, antibody-based systems can be both highly sensitive and specific.^{10, 36}

In response to the recent cases of anthrax in the U.S., considerable interest has been generated in the use of handheld antibody-based detectors by first responders. The CDC recently issued a statement on its Web site stating that the analytical sensitivity of these assays is limited and that a minimum of 10,000 spores is required to generate a positive signal. ⁷⁵ Given concerns about the sensitivity and specificity of these kits, the CDC has undertaken an independent evaluation of these tests. Conclusions from this study are expected in the near future. ⁷⁵

Nucleic acid-based systems

The specificity of nucleic acid-based systems (sometimes called polymerase chain reaction- or PCR-based systems) is derived from the selective binding of nucleic acid probes to complementary nucleic acids from the pathogen of interest. ¹⁰ Probes are designed to bind specifically to a nucleic acid sequence that is unique to the pathogen or to identify a nucleic acid sequence that is common to several pathogens. ¹⁰ The sensitivity of nucleic acid-based systems for bacteria is between 1,000 and 10,000 colony forming units (CFU); ¹⁰ however, recent reports suggest that they may be capable of greater sensitivity. ⁷⁶ Because the reaction occurs within minutes, the time-consuming parts of using nucleic acid systems are the sample preparation and the time required to detect the signal.^{10, 36} Significant limitations to the use of these methods for bioterrorism include the lack of highly specific probes for all biothreat agents (although the DOE and CDC have entered a collaboration to develop them) and the use of a single probe to test a single sample for the antigen of interest at a given time. Given security concerns, the distribution of highly specific probes will likely remain under strict federal control -- first responders are not likely to have access to these probes for testing samples in the field.

Mass spectrometry

Mass spectrometry is an analytical technique in which materials under analysis are converted into gaseous ions or other characteristic fragments.^{10, 36} The fragments are separated on the basis of their mass-to-charge ratio.^{10, 36} The technique can reportedly detect concentrations of as low as 10⁶ cells. ¹⁰ When samples are tested in the field, they are likely to contain multiple constituents (contaminants), which must be separated before they can be reliably identified. This separation can be performed by a variety of techniques, including mass spectrometry.^{10, 36}

We note that some technologies are better suited to particular agents. Nucleic acid-based systems, for example, cannot detect toxins (unlike bacteria and viruses, they do not contain nucleic acids). In contrast, mass spectrometry is more effective for the detection of toxins than bacteria.

Integrated Collection and Identification Systems

Findings

We report on 10 integrated systems that could be of use to public health officials, hospital administrators, or municipal leaders for the collection, detection, identification, and reporting of a biothreat agent; none has been described in a peer-reviewed evaluation article (Tables 3 and 7; Evidence Table 1; Appendix H).

Table

Table 7. Integrated collection and identification systems.

These systems are generally intended to transmit test results electronically to decision makers at some distance from the collection and identification site(s). They have all been designed for military use but may be increasingly available to interested public health officials and national security professionals for ongoing environmental surveillance. These systems are the size of refrigerators or larger and therefore require trucks or similar vehicles for transportation. The Canadian Integrated Biochemical Agent Detection System (CIBADS II)/4WARN system, designed to collect and identify a variety of chemical and biological agents from a commercial sport utility vehicle, is radio-linked to a command and control unit. An evaluation of CIBADSII/4WARN reported in a government document ⁵⁷ and by the manufacturer ¹¹⁶ suggested that the system was operated at speeds up to 50 miles per hour "without significant degradation of performance."^{57, 116} The impact of weather patterns on performance was also determined to be low. The exception was immediately after a thunderstorm, when the number of particles in the air rose dramatically and reduced the sensitivity of the system.^{57, 116} A government report provided evaluation data on the Portal Shield Air Base/Port Biological Detection System, which integrates data from multiple sensors linked to a centralized command post computer. ⁶⁶ This computer monitors the sensors and evaluates the data to determine if a bioterrorist attack has occurred. In the event a release is detected, the computer alerts the operator. The algorithm looks for a significant increase in at least 2 sensors before it will sound an alarm, giving the system a theoretical false positive rate of 0.25 percent. The report stated that, "after having gone through over 10,000 assays, the Portal Shield system has not had any false positives." ⁶⁶ The system can reportedly detect 8 agents, although they were not further specified. ⁶⁶ None of the reports of these integrated detection systems described the methods for maintaining the security of the sample or test results about the sample; very few details were provided about specific collection or identification components.

Background

General diagnostic DSSs are designed to assist clinicians in generating a list of possible diagnoses for a given patient. For such systems to be useful in the event of a covert bioterrorist attack, they should prompt clinicians to consider biothreat agents as a potential cause of the patient's symptoms. In this way, these systems may increase the clinician's suspicion of bioterrorism, thereby increasing the probability that the clinician performs appropriate diagnostic testing. Most of these systems require that the clinician enter information about the patient's signs and symptoms. Typically, the diagnostic DSS then produces a differential diagnosis or list of possible diagnoses for the patient. These diagnoses are sometimes ranked according to the likelihood of disease. Alternatively, some DSSs provide a calculated probability score for each diagnosis, often based on a clinical prediction rule.

Evaluation criteria

We evaluated each of the reports of general diagnostic DSSs for the following information (Table 2 -- Diagnosis; Evidence Table 4): the purpose of the system, the type of information required by the DSS (e.g., a manually-entered list of signs and symptoms provided by the clinician), the type of information provided by the DSS (e.g., a list of differential diagnoses with or without associated information about the diseases of interest), diagnostic sensitivity and specificity, whether the biothreat agents and their associated illnesses are included in the knowledge base, the method of reasoning used by the inference engine, information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, and the type of hardware required.

Findings

Our search found 6 currently available general diagnostic DSSs, 3 of which have been clinically evaluated and presented in peer-reviewed reports (Tables 3 and 8; Appendix H).

The purpose of each of these systems is to provide a differential diagnosis based on patient-specific signs and symptoms. Because general diagnostic DSSs typically provide a list of several possible diagnoses, in the event of unrecognized bioterrorism-related illness, even if the system fails to rank the correct diagnosis first, but ranks it among the top few diagnoses, this may prompt a clinician to order a diagnostic test for a biothreat agent.

All of the general DSSs require manual entry of patient information by clinicians. They then use either Bayesian (probabilistic) and/or rules-based methods to compare the patient's information with their knowledge base to generate a differential diagnosis that is typically ranked in descending order of likelihood. Some of the systems provide additional information about the suspected diseases and suggest appropriate tests if clinicians choose to pursue these diagnoses. Most of the general DSSs are available for use on personal computers, although a handheld version of DiagnosisPro^® is also available. None of the reports described if it was possible to update the probability of biothreat-related illness as the epidemic progresses. No study of a general diagnostic DSS has specifically evaluated the performance of these systems for the diagnosis of biothreat-related illness.

DXplain™, Iliad, and Quick Medical Reference (QMR) were directly compared in a multi-center trial of 105 diagnostically challenging cases. ¹²⁶ The DXplain™ knowledge base contained the correct diagnosis for 96 cases (91 percent); the Iliad knowledge base contained the correct diagnosis for 80 cases (76 percent); and the QMR knowledge base contained the correct diagnosis for 77 cases (73 percent). DXplain™ correctly included the ultimate diagnosis in 72 cases (69 percent) with an average rank of 12.4, compared with 64 cases (61 percent) with an average rank of 10.4 for Iliad and 55 cases (52 percent), at an average rank of 6.6 for QMR. (The clinical significance of this difference in rank is not clear. The importance of rank depends on how this information is used. For example, if the clinician only scans the top 5 diagnoses or if the DSS only prints out the top 10 diagnoses, then the rank may well be important. If, however, the clinician reviews the entire list of possible diagnoses specifically seeking the unusual diseases that he or she had not previously considered as a means of enhancing their diagnostic capabilities, then rank is less important.) When considering only the 63 cases for which the correct diagnosis was present in all systems, Dxplain™ identified the correct diagnosis in 50 cases (79 percent) at an average rank of 11.7. Iliad was correct in 48 cases (76 percent) at an average rank of 10.2 and QMR correctly identified the final diagnosis in 45 cases (71 percent) with an average rank of 5.4.

The other evaluations of the general diagnostic DSSs differed with respect to their study designs. Some evaluated physician acceptance of the system. However, high acceptance does not necessarily mean that a clinician would use the system for routine cases (such as a patient presenting with a flu-like illness, a common early presentation of many biothreat-related illnesses). Other study designs addressed the observed phenomenon that different clinicians use a different set of diagnostic terms to describe the same patient.^127-129 These differences may result in the DSS producing differing lists of diagnoses. Therefore, some studies compared the terms input into a system by different clinicians, given the same case, and the resulting differential diagnoses.

As of the publication of this Report, the manufacturer of Iliad has stopped selling and providing technical support for that system. ¹³⁰ We have nonetheless included Iliad in this section because it continues to be available through some retailers, and clinicians continue to use this product.

Summary: General diagnostic DSSs

The role of general diagnostic DSSs in a bioterrorism response is to enhance the likelihood that clinicians consider the possibility of bioterrorism-related illness. Therefore, these systems could contribute to the detection of a previously unrecognized release of biothreat agents. However, the reports of general diagnostic DSSs have several important limitations that prevent conclusions regarding their ability to serve this role. First, none of the DSSs has been evaluated formally with respect to bioterrorism response. Second, all of these systems require laborious manual entry of patient findings, which may be a substantial barrier to use in clinical settings. Efforts to link general diagnostic DSSs to other hospital information systems, if successful, would reduce the data entry burden substantially. In addition, availability of the system on a handheld computer (as for DiagnosisPro^®) might make the system more convenient for clinicians to use. Third, available evaluations do not indicate whether disease caused by biothreat agents are included in the databases for many systems. Thus, we were not able to assess the extent to which biothreat agents are included in any of the general diagnostic DSSs knowledge bases or whether the systems are updated with new information about the clinical presentations of these diseases (except that Iliad has not been updated since 1997). Fourth, general diagnostic systems that use probabilistic information about the likelihood of disease will have inappropriately low pretest probabilities for biothreat agents in the event of a bioterrorism event. To provide a ranking of differential diagnoses, the system relies on estimates of the prevalence or probability of diseases. If a biothreat outbreak was known or strongly suspected, the pretest probability for these agents would change dramatically from the probabilities appropriate during routine clinical use. It would be helpful if the knowledge base could be updated to reflect changes in the likelihood of diseases based on local public health data (i.e., if the system were automatically updated with local incidence and prevalence information) or could be modified in the context of a known bioterrorism event.

Background

Because many biothreat agents cause pulmonary disease, chest X-rays would be a common diagnostic procedure performed on patients presenting after a bioterrorism event. Interstitial disease would be the most likely finding. In the case of inhalation anthrax, a widened mediastinum may be seen; however, this is not always present, even in some advanced cases. ¹⁵²

Radiology interpretation systems may increase the diagnostic accuracy of radiographic reports. For this Report, we limited our search to those technologies that could be used to automate the interpretation of radiologic images for the diagnosis of biothreat agents. For example, we excluded those systems that detect mammographic lesions or pulmonary nodules. In this section, we discuss 2 types of systems -- those that assist clinicians in the interpretation of radiographic images, and those that use natural language processing methods to abstract information from the reports of radiographic procedures for diagnostic purposes.

Evaluation criteria

We evaluated each of the reports of radiologic DSSs for the following information (Table 2 -- Diagnosis; Evidence Table 4): the purpose of the system, the type of information required by the DSS (e.g., the actual radiological image or the text of a radiologist's report of the image), diagnostic sensitivity and specificity, whether the biothreat agents and their associated illnesses are included in the knowledge base, whether the system uses a standard vocabulary, the method of reasoning used by the inference engine, information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, the type of hardware required, and the system's security measures.

Findings: Radiologic interpretation systems

Our search found 2 radiologic interpretation systems, 1 of which has been clinically evaluated and described in a peer-reviewed article (Tables 3 and 9; Appendix H).

The first system, described in 3 evaluation articles, scans digitized radiographs for abnormal regions to assist clinicians in the identification of pulmonary infiltrates.^153-155 These studies calculated receiver operating characteristic (ROC) curves for each of the systems under evaluation. ROC curves are a plot of the sensitivity of a diagnostic test (typically on the y-axis) against 1 minus its specificity (typically on the x-axis). Because the ideal diagnostic test is 100 percent sensitive and 100 percent specific, the area under an ideal ROC curve would be equal to 1. Minimal improvements in the area under the ROC curve were shown when computer-aided diagnosis was employed for a small set of radiographs. Since the vast majority of infiltrates will not be related to biothreat agents, it remains unclear whether this technology can be translated into improved detection of bioterrorism-related illness.

Researchers at the same institution also found that using an artificial neural network can improve the performance of radiologists in the differential diagnosis of interstitial lung disease. ¹⁵⁵ When chest radiographs were viewed in conjunction with network output, the average area under the ROC curve increased from 0.83 to 0.91. The clinical significance of such a change is not clear.

None of the reports described whether the biothreat agents and their associated illnesses are included in the knowledge base, whether the system uses a standard vocabulary, information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, the type of hardware required, or the system's security measures.

Findings: Natural language processing systems

Natural language processing techniques have been developed to automate identification of disease concepts in free text such as radiology reports. We found reports of 2 such systems, 1 of which has been clinically evaluated and described in a peer-reviewed article (Tables 3 and 10; Appendix H).

Table

Table 10. Natural language processing systems.

The purpose of these programs is to search electronic text for concepts related to pneumonia, and then either alert the clinician or incorporate this information with other data from the electronic medical record into diagnostic or management applications. Neither of the medical language processing systems that we found was specifically designed to diagnose bioterrorism-related illness. None of the reports of these systems described whether the biothreat agents and their associated illnesses were included in their knowledge bases or whether the systems used standard vocabularies, nor did they provide information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, specify the type of hardware required (minimally, each required an electronic medical record system), or describe the system's security measures.

Two studies have evaluated the ability of medical language processing systems to identify relevant concepts in radiology reports. SymText is a medical language processing system developed by the Latter Day Saints (LDS) Hospital at the University of Utah (an additional description of this system is provided in the Management section of this chapter).^{156, 157} In one study, researchers compared the ability of SymText to identify pneumonia-related concepts in 298 X-ray reports with those of 2-word search programs, a layperson, and a resident physician. SymText performed better than the word search programs and the layperson but similar to the resident physician. ¹⁵⁶ A similar system was evaluated at the Columbia-Presbyterian Medical Center in New York. ¹⁵⁷ This study compared differences in the interpretation of 200 radiology reports by groups of 2 physicians, groups of laypersons, and the natural language processor. The differences between the interpretations of the natural language processor and the physicians were similar to the differences among physicians. ¹⁵⁷ This suggests that the natural language processor's ability to identify these concepts was similar to that of the physicians.

Researchers at LDS Hospital have integrated SymText into a real-time DSS designed to implement guidelines for community-acquired pneumonia. ¹⁵⁸ The radiology department uses speech recognition technology that decreases the time necessary to transcribe radiographic reports. ¹⁵⁸ As soon as the radiologist completes his or her dictation, SymText identifies patients who may have pneumonia based on their radiology reports and assesses the severity of their pneumonia. ¹⁵⁸ These findings are combined with other clinical and laboratory data to generate management recommendations to clinicians in compliance with clinical practice guidelines. An evaluation of this automated guideline showed that SymText was similar to physicians in identifying patients eligible for the guideline, but worse than the physicians in extracting information about the location and extent of the infiltrates (patient outcomes were not assessed). ¹⁵⁸

Summary: Radiologic systems

Our search identified 4 IT systems designed to improve radiographic diagnoses or incorporate data from radiology reports into diagnostic or management DSSs. Their utility in recognizing illnesses caused by bioterrorism is unknown, as none has been formally evaluated for this purpose.

The system from the University of Chicago has established utility for the diagnosis of community-acquired pneumonia. However, because the radiologic findings for most bioterrorism-related illness will be identical to pulmonary diseases of other etiologies and because the presence of a specific radiologic finding associated with bioterrorism-related illness is the exception rather than the rule, it is not clear that these systems could help clinicians, beyond alerting them to the presence of a pulmonary infiltrate, pleural effusion, or widened mediastinum. For radiologic systems to have a significant effect on clinicians' diagnostic decisions in regards to a bioterrorism event, they would have to raise the clinician's index of suspicion that a biothreat agent may be causing the radiologic findings. Incorporating information from these systems with other information from patients' medical records and knowledge bases about the clinical presentations of bioterrorism-related illnesses could be a useful innovation. Specifically, radiologic systems could serve as a component of an integrated management system that incorporates radiologic as well as other clinical information with clinical practice guidelines for the management and reporting of suspected bioterrorism-related illness.

Background

Telemedicine is the use of telecommunications technology for medical diagnostic, monitoring, and therapeutic purposes when distance separates the users. ¹⁶⁰ We direct readers interested in this topic to a recent Evidence Report from AHRQ entitled "Telemedicine for the Medicare Population." ¹⁶⁰ Briefly, this Report describes 3 types of telemedicine systems: "(1) store-and-forward services that collect clinical data, store them, and then forward them to be interpreted later; (2) self-monitoring/testing services that enable clinicians to monitor physiologic measurements, test results, images, and sounds, usually collected in a patient's residence or care facility; and (3) clinician-interactive services that are real-time distance clinician-patient interactions." ¹⁶⁰ They found that telemedicine consults increased steadily throughout the 1990s with most programs designed to serve rural populations, veterans, and the elderly. ¹⁶⁰ Additionally, they report that teledermatology is the most-studied clinical specialty in store-and-forward telemedicine; its diagnostic accuracy and patient management decisions are comparable to those of in-person clinical encounters. ¹⁶⁰

Evaluation criteria

We evaluated each of the reports of telemedicine systems for the following information (Table 2 -- Diagnosis; Evidence Table 4): the purpose of the system, the settings in which they are used, the sensitivity and specificity of the diagnoses provided by consultants using the system, the type of hardware required, and the system's security measures.

Findings

Our search identified 4 telemedicine/teleradiology systems with potential relevance to bioterrorism; clinical evaluations for 2 of these have been presented in peer-reviewed evaluations (Tables 3 and 11; Appendix H). Since our search strategy was neither designed specifically for telemedicine nor teleradiology, the systems identified may not be representative of the systems that are available. We present these systems in this section because they are related to the radiologic interpretation systems just described, although they share many similarities with the communication systems described later in this Report.

Table

Table 11. Diagnostic systems using telemedicine.

Three of the 4 telemedicine systems were designed by the military to provide telemedicine consultation for military personnel at sites distant from military hospitals. Similarly, the other system, MERMAID, was designed for the European Union to provide telemedicine consultations to members of the merchant marine. None was designed or evaluated specifically for providing telemedicine consultations for disease resulting from bioterrorism. The sensitivity and specificity of the diagnoses provided by consultants using the system, the type of hardware required, and the system's security measures were not described in any report.

The Walter Reed Army Medical Center (WRAMC) Telemedicine Service system was evaluated in a retrospective case review of 171 telemedicine consultations. ¹⁶¹ Of these, 114 consults were reviewed: 39 percent were for dermatology, 16 percent for surgical subspecialties and 15 percent for orthopedics. Telemedicine was felt to affect the diagnosis in 30 percent, the treatment in 32 percent, and the overall patient status in 70 percent of cases.^{161, 162}

Summary: Diagnostic systems using telemedicine

No telemedicine system has been evaluated specifically for bioterrorism. Telemedicine systems are most useful in areas with limited direct access to medical specialists. Since acts of bioterrorism against civilian populations may be less likely to occur in remote areas than in population centers, these systems may be of limited value against bioterrorism. However, since few practicing primary care or emergency physicians have ever seen the rashes associated with smallpox or other bioterrorism-related illness, the use of teledermatology technologies may increase the likelihood of a timely diagnosis by facilitating access to dermatologic experts. In the event of a widespread epidemic reaching geographically isolated areas, existing telemedicine infrastructures could be used by public health officials to relate public health information and alerts to clinicians.

Background

In this section, we present a variety of other types of diagnostic systems. Unlike the general diagnostic DSSs discussed earlier, most of these systems are specifically for the diagnosis of infectious diseases (thereby limiting their use to only those circumstances in which the clinician suspects an infectious etiology).

Evaluation criteria

We evaluated each of the reports of other kinds of diagnostic DSSs for the following information (Table 2 -- Diagnosis; Evidence Table 4): the purpose of the system, the type of information required by the DSS, diagnostic sensitivity and specificity, whether the biothreat agents and their associated illnesses are included in the knowledge base, whether the system uses a standard vocabulary, the method of reasoning used by the inference engine, information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, the type of hardware required, and the system's security measures.

Findings

In this section, we present a brief description of 9 other diagnostic systems, 7 of which have been described in at least 1 peer-reviewed evaluation (Tables 3 and 12; Evidence Table 2; Appendix H).

Table

Table 12. Other diagnostic DSSs.

The included systems were designed for a variety of purposes: 4 diagnostic DSSs specifically for infectious diseases (The Computer Program for Diagnosing and Teaching Geographic Medicine, GIDEON, a fuzzy logic program to predict the source of bacterial infection from demographic variables, and the Texas Infectious Disease Diagnostic DSS); 2 systems that facilitate the prompt diagnosis of patients with active pulmonary tuberculosis (the first is a neural network-based system from the State University of New York at Buffalo, and the other is based on natural language processing of electronic medical record information from Columbia University); and 3 diagnostic systems with other purposes. We included the tuberculosis diagnostic systems primarily because tuberculosis serves as a model for bioterrorism-related agents that present as pneumonia and require respiratory isolation during the initial treatment period. Those systems that incorporate diagnostic functions with management recommendations are not presented in Table 12 but are described with Management and Prevention systems later in this Report. The methods used by these systems to generate diagnoses include probabilistic and rules-based inference engines and neural networks.

Of all the diagnostic DSSs, we could only verify that GIDEON and The Computer Program for Diagnosing and Teaching Geographic Medicine specifically include most of the worrisome bioterrorism-related organisms in their knowledge bases.^{168, 169} Both of these systems provide differential diagnoses of infectious diseases based on clinical parameters regarding a patient that are entered into the program. The Computer Program for Diagnosing and Teaching Geographic Medicine also provides general information about infectious diseases, anti-infective agents, and vaccines.

The evaluation of GIDEON compared the diagnostic accuracy of the DSS to that of medical house officers admitting 86 febrile adults to the Boston Medical Center. The house officers listed the correct diagnosis first in their admission note 87 percent (75/86) of the time compared with 33 percent (28/86) for GIDEON. ¹⁶⁹ In a study to evaluate the diagnostic accuracy of The Computer Program for Diagnosing and Teaching Geographic Medicine, 6 infectious disease specialists (blinded to the patients' actual diagnoses) were asked to record all positive and negative clinical data for 295 consecutive patients with established diagnoses and 200 hypothetical cases. The computer program correctly identified 75 percent (222 of 295) of actual cases and 64 percent (128 of 200) of hypothetical cases. The clinical diagnosis was included in the computer differential diagnosis list in 94.7 percent of cases. Among the cases included in this evaluation, several were for the causative agents of: anthrax, brucellosis, cholera, cryptosporidiosis, Hantavirus respiratory distress syndrome, Lassa fever, plague, Q fever, Rocky Mountain spotted fever, shigellosis, and tularemia. However, this system was only tested on cases for which the diagnosis was known; therefore, there is no information on how it would perform for cases with unknown outcomes. ¹⁶⁸

If a system produces a single diagnosis for a given case as its output, the sensitivity and specificity of the system can be readily determined if the case's actual diagnosis is known. Frequently, systems are designed to provide a list of many possible diagnoses, often ranked according to their probability of being the actual diagnosis. Under these circumstances the clinician will have to determine the lower threshold of probability for which they will make a diagnostic or therapeutic decision (i.e., if a system generates a list of possible diagnoses for a case and suggests that smallpox is on the differential but highly unlikely, he or she may not choose to send a viral culture or notify the local public health official). When diagnostic systems provide a list of possible diagnoses, it may be more appropriate to calculate receiver operator characteristic (ROC) curves to evaluate the performance of the system over a range of probability thresholds. We direct interested readers to an article by Fraser and colleagues measuring the performance of systems that generate differential diagnoses using ROC curves and other methods. ¹⁷⁰ Only the neural network for the diagnosis of active pulmonary tuberculosis was evaluated with ROC curves.

For those programs that require a user to input case-specific information, we again found that the differential diagnoses provided by the systems were highly dependent upon the information input about the cases. This was particularly true for DERMIS, where generalists' inputs were less likely than those of specialists to result in a correct diagnosis. This is an unfortunate finding because patients with bioterrorism-related skin lesions are more likely to present to general clinicians than dermatologic specialists and because the early recognition of skin lesions associated with smallpox, Glanders, bubonic plague, and tularemia could significantly reduce bioterrorism-related morbidity and mortality.

None of the reports of these general DSSs discussed barriers to the use of the systems, whether the system uses a standard vocabulary, information regarding the ability to update the probability of biothreat-related illness as the epidemic progresses, the type of hardware required, or the system's security measures.

Summary: Other diagnostic systems

The role of this heterogeneous group of diagnostic DSSs in a bioterrorism response is to enhance the likelihood that clinicians consider the possibility of bioterrorism-related illness. Therefore, these systems could contribute to the detection of a previously unrecognized release of biothreat agents. However, the reports of general diagnostic DSSs have several important limitations that prevent conclusions regarding their ability to serve this role.

As was true for the general diagnostic DSSs, if cases associated with biothreat agents are not included in the system's knowledge base, the diagnosis of bioterrorism-related illness will not be included in a system's differential diagnosis. GIDEON and The Computer Program for Diagnosing and Teaching Geographic Medicine are the only systems for which we were able to obtain a complete list of the diseases included in the knowledge base and could verify that all potential biothreat agents were included. All of the systems presented in Table 12 are limited in that they are not general diagnostic systems but specific for either infectious diseases or another specialized application; thus, if the patient does not present with either a fever or a rash, the clinician may not choose to use these specialized DSSs. Additionally, many of these systems require clinicians to manually enter data -- a laborious step that may be a barrier to the use of these systems and has been demonstrated to increase inter-user variability.

Background

The earliest signs and symptoms caused by most biothreat agents are flu-like illness, acute respiratory distress, gastrointestinal symptoms, febrile hemorrhagic syndromes, and febrile illnesses with either dermatologic or neurologic findings. Therefore, patients with these syndromes are the targets of bioterrorism-related syndromal surveillance programs. There is no widely accepted definition for any of these syndromes. As a result, syndromal surveillance systems are widely heterogeneous with respect to the syndromes under surveillance and the definitions of each syndrome.

Findings

We found 7 syndromal surveillance systems, none of which has been described in a peer-reviewed evaluation report (Tables 3, 14 and 15; Figures 3 to 10; Appendix H). We have no specific information about the usefulness of these systems in terms of detecting known infectious disease outbreaks (e.g., no information about whether the syndromal surveillance system detected last season's influenza outbreak). Additionally, we have no specific information on any of these systems' flexibility, acceptability, representativeness, or the direct costs of implementation.

Figure

Figure 3. Examples of data sources for surveillance systems.

Figure

Figure 10. RSVP^®: Flu-like illness screen. With written permission from: RSVP ^® Project, Sandia National LaboratoryThis screen enables clinicians to report additional clinical data about the patient (more...)

The purposes of the systems vary: some systems are designed for ongoing surveillance for infectious disease outbreaks; whereas, others are for short-term surveillance for bioterrorism-related illness before, during, and after major political, economic, or entertainment events. For example, the Border Infectious Disease Surveillance Project (BIDS) collects syndromal surveillance information along the U.S.-Mexican border. BIDS performs ongoing surveillance by routinely collecting and analyzing data at regular intervals (Table 14). In contrast, systems such as the Lightweight Epidemiology Advanced Detection and Emergency Response System (LEADERS), have typically been used to perform event-based surveillance -- that is, they begin collecting data for a brief period before an event thought to be a target for bioterrorists (e.g., a major sporting or political event) and continue collection through a specified time after the event has finished (Tables 14 and 15; Figures 4 and 5). LEADERS has been used for event-based surveillance for the 1999 World Trade Summit, the 2000 Democratic and Republican National Conventions, and the 2001 Presidential Inauguration. (LEADERS can also be used for ongoing surveillance.)

Table

Table 14. Surveillance systems collecting syndromal reports.

Figure

Figure 4. LEADERS main page. With written permission from: LEADERS ConsortiumUsers select from among the LEADERS tools in the upper left section of this screen. For example, The Critical Care Tracking System allows (more...)

Figure

Figure 5. LEADERS syndromal surveillance page. With written permission from: LEADERS ConsortiumThis is an example of a syndromal surveillance form. The user can customize the list of syndromes or use one of the preprogrammed (more...)

The systems also differ with respect to the type of syndromal data they collect. For example, public health officials are evaluating several methods for collecting syndromal surveillance data from clinical personnel. The Syndromal Surveillance Tally Sheet is a paper-based ongoing syndromal surveillance tool used in Santa Clara County, California for the surveillance of 6 syndromes (Tables 14 and 15; Figure 6). ²⁴³ Triage nurses in urgent care centers and emergency departments enter data on each patient whom they evaluate and fax the sheet to the local health department at the end of each shift (typically 3 times a day). Health Buddy^® is a device that can be used to display syndromal surveillance questions to users in a variety of clinical areas (Tables 14 and 15; Figure 7). ²⁴⁴ It was recently piloted in the Stanford University Medical Center Emergency Department. It is likely that similar tools have been developed and implemented by county health departments throughout the U.S.; however, a comprehensive survey of the syndromal surveillance methods currently in use was outside the scope of our project.

Figure

Figure 6. Syndromal surveillance tally sheet.

Figure

Figure 7. Health Buddy^® device. With written permission from: HealthHeroTriage nurses use this device to answer syndromal surveillance questions presented on the screen. Additionally, public health officials (more...)

The Early Warning Outbreak Recognition System (EWORS),^{245, 246} developed collaboratively by the Naval Medical Research Unit-2, Department of Defense Global Emerging Infections System (DOD-GEIS), and the Indonesian Ministry of Health, is a global syndromal surveillance system (Tables 14 and 15). EWORS uses a simple computer program designed to enable untrained personnel to collect basic demographic and symptom data that are downloaded daily from remote sites to the Indonesian Ministry of Health.

The Rapid Syndrome Validation Project (RSVP^®) is a Web-based collection tool for use by clinicians (currently primarily in use in emergency departments but could be used in any clinical area with a personal computer and Internet connection) (Tables 14 and 15; Figures 8 to 10). Data are entered on only those patients whom clinicians believe fit into 1 or more of 6 syndromal categories. ²⁴⁷ RSVP^® facilitates (but does not require) the collection of more detailed information about a particular patient than the other systems (although several of the other systems have flexible interfaces that allow customization and the design of more detailed collection forms). In addition, RSVP^® enables public health officials to send alerts and public health information to clinicians.

Figure

Figure 8. RSVP^®: Opening screen. With written permission from: RSVP ^® Project, Sandia National LaboratoryClinicians use this screen to log into the system. They enter their name and password (more...)

Figure

Figure 9. RSVP^®: Demographics screen. With written permission from: RSVP ^® Project, Sandia National LaboratoryOn the left side, clinicians enter the patient's zip code, occupation, gender, (more...)

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) was developed by the DOD-GEIS (Tables 14 and 15). ²⁴⁸ This system downloads ambulatory diagnosis codes grouped into "syndromal clusters." Initially, ESSENCE collected data from the primary care and emergency clinics serving the DOD health care beneficiaries living in and around Washington, DC. The data were downloaded daily onto a server and automatically analyzed with both traditional epidemiologic methods and time-space analyses. Following the events of September 11, 2001, ESSENCE was expanded to include virtually the entire Military Health System. Currently, ESSENCE downloads outpatient data on a daily basis from 121 Army, 110 Navy, 80 Air Force, and 2 Coast Guard installations around the world. Over 2,700 syndrome- and location-specific graphs are prepared each day and automatically analyzed for patterns that suggest a need for further investigation. ²⁴⁸ Beyond these centralized assessments, the graphs are available daily to approved DOD public health professionals on a secure Web site. ²⁴⁸ Two evaluations of ESSENCE are currently ongoing: one to address the issue of data quality, and the other to test the system's sensitivity and specificity over a range of outbreak scenarios. DARPA has awarded a $12-million 4-year grant to the ESSENCE consortium to construct a more powerful system for the Washington, DC area that includes both military and civilian data.

None of these syndromal surveillance systems has been evaluated in a comprehensive manner. However, current evaluations of ESSENCE and LEADERS are ongoing to determine their sensitivity, specificity, and timeliness. In Table 15, we present a comparison of 6 of the syndromal surveillance systems. (The paucity of available data on BIDS precludes its inclusion in this table.) The advantages of the tally sheet and Health Buddy^® device are that they are rapidly deployable at relatively low cost, requiring only a fax or phone line. They can be relatively easily integrated into the triage nurses' workflow, thereby increasing the likelihood of compliance with data collection. RSVP^® and EWORS are somewhat more costly to install since they require personal computers with Web access (RSVP^®) or a modem (EWORS) in the clinical area. For RSVP^®, clinicians enter data on only those patients whom they suspect of having one of the syndromes; therefore, the cases collected from this system will have a higher probability of disease than data collected from every patient presenting to a clinical area. Because RSVP^® collects more detailed information than the tally sheet or Health Buddy^®, it requires more of the clinician's time. ESSENCE or similar systems that perform surveillance on diagnostic codes have the enormous advantage of not adding any additional burden to clinical work flows. Additionally, ESSENCE was relatively low cost to implement and maintain and has demonstrated good scalability. The extensive epidemiologic analyses that ESSENCE performs on a daily basis far exceed what has been described for any of the other systems. However, systems like ESSENCE require that the diagnostic codes be available in an electronic format in a timely manner. Hospital or ambulatory diagnosis codes are not likely to be as sensitive as clinician reports in detecting suspicious cases. LEADERS is the most expensive system -- requiring personal computers with Web access and a subscription fee. It may also be difficult to integrate into the hospital and health departments' IT environments. However, it can incorporate multiple streams of data and is quickly deployable. In terms of data security, LEADERS and RSVP^® reported being compliant with the Health Insurance Portability and Accountability Act of 1996 (HIPAA); no other reports specified the security measures of other systems.

Table

Table 15. Comparison of syndromal surveillance systems.

Summary: Syndromal surveillance systems

Syndromal surveillance systems could provide an early indication of cases resulting from a bioterrorism event. However, we have no evidence to determine which of the methods of collecting syndromal data provides the most sensitive, timely, acceptable, and low cost data. Local public health officials are the primary users of these data. Therefore, the ability of the system to present syndromal data in a timely manner, and minimize the necessity for additional analyses, will enhance the system's usefulness and acceptability.

We found no published standard definitions for the most common syndromes under surveillance. Additionally, none of the systems that we found that rely on clinicians to enter syndromal data provided definitions of the syndromes used by the collection tool (i.e., on the data entry screen, there was no definition of "flu-like illness"). A well-accepted list of the key syndromes for surveillance and detailed definitions of these syndromes will facilitate the integration of numerous sources of surveillance data. For example, the definition of "flu-like illness" should include its clinical characteristics so that triage nurses and clinicians can clearly identify patients with the syndrome, the specific ICD9 codes and other administrative data likely to be associated with it, the pharmaceuticals likely to be used to treat it, and the laboratory tests likely to be ordered to diagnose it. Then, each source of syndromal surveillance data can be systematically mapped to each of the syndromes, enabling the integration of all of these data into a single surveillance system.

As public health officials consider which of these systems to implement for ongoing or event-based syndromal surveillance, they are likely to find that a hospital's IT infrastructure may affect the acceptability of a syndromal collection tool. Hospitals and clinics with nascent IT infrastructures and an associated culture of paper may be better suited to the Tally Sheet or Health Buddy^®. Within a given county, some hospitals may want to report ICD9 codes and others may prefer reporting triage nurse counts on the Tally Sheet. If different hospitals within a county are reporting syndromal surveillance data from different collection tools, public health officials will need to have the capacity to collect, analyze, and present these data in single system. The Oracle database associated with LEADERS has this capacity but is limited by the lack of standard definitions for the syndromes.

Background

Although most industrialized nations mandate the reporting of selected infectious diseases, compliance is typically poor. ²⁵² A study of 176 clinicians in the United Kingdom (U.K.) found that 123 (70 percent) did not know where to obtain notification forms and most were unaware of reporting requirements. ²⁵³ For example, 79 clinicians (45 percent) were unaware that cases of pneumococcal meningitis should be reported. ²⁵³ Notable exceptions to this trend are the "astute clinicians" who, by reporting suspicious cases or clusters of cases to their local public health officials, have been largely responsible for the timely detection of infectious disease outbreaks (as in the case of West Nile Virus outbreak in New York City in late August 1999 ¹⁵ and the recent cases of mail-associated anthrax^{254, 255}). Efforts to enhance the awareness of clinicians, educate them about bioterrorism-related illness, and remind them of their reporting obligations are outside the scope of this Report. We refer interested readers to a comprehensive Evidence Report entitled "Training of Clinicians for Public Health Events Relevant to Bioterrorism Preparedness." ²⁵⁶

Increasingly, public health officials have realized that the establishment of networks of practitioners with training in disease reporting could improve the quality of data obtained from voluntary communicable disease notification systems. A sentinel network is a disease surveillance program that involves the collection of health data on a routine basis by clinicians with some training (albeit minimal) in reporting communicable disease. The growth of such sentinel systems has generated demand for information systems capable of automating data collection, analysis, reporting, and communication. In this section, we describe systems for the electronic collection and analysis of clinical reports from individual clinicians and sentinel networks.

Findings

Our search identified 6 IT/DSSs used for the reporting of clinical cases for diseases other than influenza (which we present in a separate section of this Report); 2 of these have been described in peer-reviewed evaluation reports (Tables 3 and 16; Appendix H).

The purpose of each system for collecting data varies. Some monitor the incidence of specific infectious diseases (such as tuberculosis); others collect surveillance data on groups of communicable diseases or on emerging infectious diseases. Accordingly, they also vary with respect to method and timeliness of data collection, type of hardware required, and geographic area under surveillance. In the paragraphs that follow and in Table 16, we present information about these characteristics of the surveillance systems collecting clinical reports. None of the reports of these systems described security measures, information about how an outbreak is determined, or the direct costs needed to operate the system.

Table

Table 16. Surveillance systems collecting clinical reports.

In 1983, France took a technological lead in electronic disease surveillance with a national telecommunications program that provided videotext home terminals free of charge to French citizens. ²⁵² In 1984, the Institut National de la Santé et de la Recherche Médicale (INSERM), in collaboration with the French Ministry of Health, initiated a program for electronic monitoring of communicable diseases called the French Communicable Disease Network (FCDN).^257-259 Today, a volunteer sample of about 1 percent of French general practitioners enters weekly reports on personal computers with either modem or videotext terminals.^257-259 These data are available online through a Web site called SentiWeb. A major strength of the FCDN system for surveillance of infectious diseases is the timeliness of the collection, analysis and distribution of data. The reports of influenza-like illness from sentinel general practitioners, combined with information on viral isolates from the French Reference Centers, provide timely information for developers and evaluators of the influenza vaccine. ²⁶⁰ FCDN's early detection system is based on a regression model, which has been demonstrated in a retrospective study to forecast epidemics with a delay of only 1 week. ²⁶¹ A 1998 evaluation of 500 sentinel practices of the FCDN found that although the system quickly offered estimates of the effectiveness of the influenza vaccine, and all results were available on the Internet within 1 week of data collection, delay could be shortened by updating data from the clinicians daily instead of weekly. ²⁶²

The Eurosentinel project depends on an international network of sentinel general practitioners to monitor influenza and other syndromes and diseases in Europe. Volunteer physicians submit weekly reports to a coordinating center in Belgium. Outputs for influenza are within minutes of reporting, while data for other syndromes and diseases are released in a quarterly newsletter. A report describing the experiences of the first 3 years of the project found that discrepancies in disease reporting practices, particularly the use of different denominators, between the sentinel networks of different countries made it difficult to compare the data from each network. ²⁶³

In the U.S., each state health department uses a standardized weekly form submitted by e-mail to CDC through the National Electronic Telecommunications System for Surveillance (NETSS).^264-267 In the past, CDC returned analyses of this information back to the state health departments via an electronic text message system of the Morbidity and Mortality Weekly Report (MMWR), which these departments received earlier than other subscribers. CDC then established the Public Health Network (PHNET) to provide a tool featuring up-to-date maps and graphs to return information alerts to state health departments. ²⁵² Data on selected notifiable infectious diseases continue to be published weekly in the MMWR and at year-end in the annual Summary of Notifiable Diseases of the United States. An observational study monitoring the delay between the date of disease onset and date of report to CDC found that for states providing the date of disease onset for at least 70 percent of reported cases, the median reporting delay was 23 days for shigellosis, 22 days for salmonellosis, 33 days for hepatitis A, and 20 days for bacterial meningitis. ²⁶⁴ Within 8 weeks of disease onset, more than 75 percent of reports were made for each of these diseases. Reporting delays varied widely between states, particularly for salmonellosis and shigellosis. ²⁶⁴

In addition to the major surveillance efforts just described, we report on 3 other surveillance projects. EUROTB collects data on tuberculosis from multiple European countries and reports these data to the WHO.^268-270 EMERGEncy ID NET is a collection of sentinel physicians from 11 U.S. emergency departments who collect data on patients with selected clinical syndromes through multiple-choice forms. The information is entered into a desktop computer and electronically transferred to a central database at the Olive View-UCLA Medical Center for later analysis.^{271, 272} The Global Emerging Infections Sentinel Network (GeoSentinel)^{273, 274} collects reports from 25 sentinel clinics to monitor geographic and temporal trends in morbidity among travelers and other mobile populations. Reports are either faxed or electronically submitted to a central database in Georgia. GeoSentinel can also be used to send alerts and surveys to a widespread network of providers. The only additional information available about these systems describe the incidence and prevalence of the diseases under surveillance. It is difficult to determine their potential utility during a bioterrorism event from these data.

Summary: Surveillance systems collecting clinical reports

Because clinicians may be the first to recognize unusual or suspicious illnesses, reports from clinician networks are an essential source of surveillance data for detection of bioterrorism-related diseases. However, the usefulness of the systems described in this section is difficult to assess. First, none of these systems has been evaluated for its ability to detect illness caused by bioterrorism events. Thus, the sensitivity, specificity, and timeliness of the systems have not been documented. Second, the descriptions of many of the systems were published in the early 1990s. It is possible that some have been upgraded to rely on electronic reporting, but current descriptions have not been published.

Systems for detection of bioterrorism require more rapid response times than do systems designed for some other purposes. The timeliness of systems that collect clinical reports depends on the time required to report data, analyze the data, and communicate the results of such analyses to decision makers who can respond appropriately. Systems that collect data weekly will be substantially less useful than systems that can provide more rapid collection and analysis. Of the systems in this section, Eurosentinel provides the timeliest data (but only for influenza, data on other diseases and syndromes has a longer delay). The timeliness of the other systems varies from days to months. Investigation of whether the systems can be modified to increase timeliness should be a high priority.

Background

On June 26, 1957, 1,688 delegates from 43 states and foreign countries arrived at a church conference at Grinnell College. ²³⁶ Among them were 100 Californian delegates who had made the trip to Iowa in a single chartered railroad coach. ²³⁶ Cases of influenza had developed en route and when the 100 Californians interacted with the other delegates, an epidemic exploded. ²³⁶ By July 1, more than 200 clinical cases of influenza were documented, the delegates ended their conference early, and returned home -- taking the virus with them. ²³⁶

Often perceived as a comparatively low-level threat, the viruses that cause influenza are continually evolving and occasionally undergo sufficient genetic drift that they cause significant morbidity and mortality. ²⁴¹ For example, the 1918-1919 pandemic killed more than 20 million people around the world. ²⁴¹ The less severe pandemics in 1957 and 1968 killed a total of 1.5 million people and caused an estimated $32 billion in economic losses worldwide. ²⁴¹ Established in 1948, the WHO's global network for influenza surveillance currently includes 110 collaborating laboratories in 82 countries, continuously monitoring locally isolated influenza strains.^{240, 284} These data are used to make recommendations on the 3 virus strains to be included in the next season's influenza vaccine.^{240, 284} The WHO has also created FluNet, an Internet site for reporting and monitoring clinical cases of influenza.

The U.S. Air Force has actively contributed to the global influenza surveillance effort since 1976 through the efforts of Project Gargle, based at Brooks Air Force Base in San Antonio, Texas.^{245, 285, 286} In June 1996, President Clinton issued Presidential Decision Directive NSTC-7, which formally expanded the mission of the DOD to support global surveillance, training, research, and response to emerging infectious disease threats.^{245, 285, 286} The DOD-GEIS was formed in 1997 in response to this Presidential Directive. During the 1999-2000 influenza surveillance season, Project Gargle processed 3,825 throat swabs from 19 sentinel U.S. military bases, 49 nonsentinel bases, and 3 DOD overseas laboratories.^{245, 285, 286} The Air Force also correlates the immunization status of Air Force personnel with influenza morbidity, providing a marker for vaccine efficacy.^{245, 285, 286}

The lessons of the past 50 years of global surveillance for influenza are applicable to the problem of surveillance for biothreat agents. Effective surveillance for bioterrorism-related diseases requires similar global "networks of networks" integrating clinical and laboratory data collected by governmental, charitable, military, private and professional organizations and reported to local, national, and international public health organizations.

Findings

Our search identified 11 IT/DSSs for influenza surveillance, 3 of which have been described in peer-reviewed evaluation reports (Tables 3 and 17; Appendix H).

The AAH Meditel United Kingdom General Practitioner Reporting System has been used for disease surveillance across the U.K. ²⁸⁷ Each night, data are downloaded from the computers of hundreds of sentinel general practitioner offices to a mainframe computer at AAH Meditel. ²⁸⁷ The system was originally intended to collect prescribing data but has been extended to collect other patient data. A 10-week study compared the AAH Meditel database of clinical reports of influenza cases to the manual influenza surveillance system of the Royal College of General Practitioners' and found the slope of the influenza epidemic curve to be nearly identical for both systems. ²⁸⁷ This suggests that the system was able to detect an outbreak of influenza at least as well as the sentinel General Practitioners against whom it was compared (we have no additional sensitivity or timeliness information on this group). However, the incidence of disease as detected by the General Practitioners was 3 to 4 times higher than that derived from AAH Meditel. ²⁸⁷ The authors suggest that the underreporting of the AAH Meditel system might be due to clinicians inexperienced in surveillance methods, especially in comparison with the trained sentinel Practitioners of the Royal College, and overestimation of the surveillance population. ²⁸⁷

The French Regional Influenza Surveillance Group (GROG) system consists of almost 1500 voluntary general practitioners, pediatricians, pharmacists, and military medical officers who provide a weekly activity summary via telephone.^{288, 289} A regression-based method for early recognition of influenza epidemics based on time series analysis was used to evaluate surveillance data from various indicators. Sick-leave as prescribed by physicians and recorded by the "Assistance Publique" (Health and Social Security Services), emergency house calls, and numbers of patients with influenza-like illness seen by general practitioners and pediatricians were the most sensitive indicators for the early recognition of influenza epidemics. The detection of influenza epidemics using drug consumption data was 1 week delayed relative to house call and sick-leave indicators.^{288, 289} The annual epidemic threshold criteria are derived from a combination of these indicators (with virologic data) designed to minimize the false positive rate.^{288, 289} During the 1992-1993 season, using the threshold criteria developed from the 1984-1992 historical data, GROG detected an influenza B epidemic on week 52 -- 3 weeks after the virus began circulating in the population.^{288, 289}

In 1991, the National Influenza Centre of the Netherlands set up an electronic influenza surveillance project using the existing Medimatica computer network that connected several health care institutions via a central server. The 3000 clinician and 1300 pharmacist subscribers to the Medimatica service receive low cost e-mail service and access to medical database services (including access to patient medical records and current information on a variety of diseases). ²⁹⁰ A new database was added to the Medimatica system providing current information on Dutch influenza outbreaks and served as a means for clinicians and pharmacists to report suspected influenza cases. A prospective evaluation of this influenza reporting system during the 1991-2 influenza season found that no clinicians or pharmacists reported any influenza cases. ²⁹⁰ The investigators suggested that this disappointing result might have been caused by a clinical culture in the Netherlands in which electronic reporting has not yet been embraced. ²⁹⁰

None of the other influenza surveillance systems has been reported in a peer-reviewed evaluation. Many of them collect data on a daily basis with weekly reporting of data to public health officials. The systems vary with respect to their methods of data collection, including telephone, fax, mail, and e-mail. Most incorporate several sources of surveillance data (e.g., virologic data, clinician reports, hospital admissions data, pharmacy data, and work absenteeism data). No reports of these systems discussed information on methods to maintain the security of data or direct costs associated with the program.

Summary: Surveillance systems for influenza

Surveillance systems that collect influenza data are relevant to bioterrorism surveillance in 3 ways. First, sentinel clinicians who report on patients with suspected influenza are experienced at applying a case definition to a clinical population for the collection of public health data. Because many bioterrorism-related illnesses present with a "flu-like illness," this network of trained sentinel clinicians could provide valuable surveillance data. Second, most influenza surveillance systems integrate clinical and laboratory data for the detection of influenza outbreaks. Surveillance for bioterrorism may be aided by similar integration of multiple data sources. Finally, influenza surveillance is a coordinated global effort. Given the current state of international travel, importation of food products, and trade in pharmaceuticals and blood products, infectious diseases from natural sources or from acts of bioterrorism can readily travel to the U.S from remote areas of the world. New programs for the surveillance of bioterrorism-related illness could be derived from the existing IT infrastructures and the historical relationships that have been developed for influenza surveillance. Several of the influenza systems rely on weekly reporting by clinicians -- for bioterrorism surveillance, every effort should be made to reduce this lag.

Background

Laboratories are critical to the detection of naturally occurring emerging infectious diseases, identification of biothreat agents, and monitoring for unusual patterns of antimicrobial resistance. The laboratory that encounters the first case of a covert bioterrorist attack may be located in a community hospital. ³⁰² Many of the biothreat agents are morphologically similar to the organisms that normal inhabit the human respiratory tract. ³⁰² For example, B. anthracis is morphologically identical to other Bacillus species -- most of which are contaminants of clinical samples. ³⁰² Therefore, if the organism is not speciated, the laboratory technician may consider the isolate a contaminant and discard the sample. ³⁰²

APHL and CDC are developing the Laboratory Response Network, a network of civilian public health and private laboratories in the U.S. for routine disease surveillance and detection of biothreat agents. ³⁰² In the Laboratory Response Network, laboratories in local hospitals (Level A) can rule out biothreat agents or refer organisms to Level B laboratories. ³⁰² Level B laboratories can identify certain pathogens, such as anthrax, but must refer other organisms to Level C laboratories for further evaluation. ³⁰² Level C laboratories can definitively identify a broader range of pathogens, but must also refer some samples (such as smallpox and hemorrhagic fever viruses) to Level D laboratories. ³⁰² Only 2 Level D laboratories exist in the U.S., at the CDC and at the U.S. Army Medical Research Institute of Infectious Diseases (USAMRIID). ³⁰² An information system is currently being established for the Laboratory Response Network that includes: a password-restricted site (using NEDSS standards and systems) for ordering reagents, distributing procedural information, and standard messaging for results. ³⁰² The Laboratory Response Network has recently received additional federal funds enabling its expansion to 120 laboratories, each equipped with advanced laboratory technologies and trained staff. ³⁰³

Findings

Our search identified 12 systems for the surveillance of laboratory data (4 of which were described in peer-reviewed evaluation reports) and 11 systems for the surveillance of antimicrobial data (1 of which was described in a peer-reviewed evaluation report) (Tables 3, 18 and 19; Appendix H).

Table

Table 18. Surveillance systems for laboratory data.

Table

Table 19. Surveillance systems for antimicrobial resistance data.

A clinical evaluation of the Electronic Communicable Disease Reporting System (ECDRS) operated by the Hawaii Department of Health demonstrated that this automated laboratory reporting system improved communicable disease reporting to public health officials relative to conventional reporting methods. ³⁰⁴ At a predetermined time each day, a computer dedicated to reporting purposes in each of the participating laboratories automatically connected to the laboratory information system in routine use at that facility. This connection launched a data extraction program that then transmitted the laboratory data to a public health database. During a 6-month evaluation period, 325 (91 percent) of 357 laboratory reports were reported through ECDRS, compared with 156 (44 percent) reported through conventional methods. ³⁰⁴ For the combined 124 reports received from both reporting mechanisms, electronic reports were received an average of 3.8 (95 percent confidence interval: 2.6-5.0) days earlier than were corresponding reports sent by mail or fax. ³⁰⁴ Of the 21 data fields per report, 12 were significantly more likely to be complete in the electronically reported data; however, the field describing the type of specimen (e.g., from blood or stool) was more likely to be reported by conventional methods. ³⁰⁴

The National Enteric Pathogen Surveillance Scheme (NEPSS)/Salmonella Potential Outbreak Targeting System (SPOT) is designed for the early detection of potential Salmonella outbreaks in Australia. ³⁰⁵ A preliminary retrospective analysis of this system demonstrated a sensitivity of 100 percent (15 out of 15 Salmonella outbreaks), compared with 50 percent for another surveillance method in which epidemiologists "eye-balled" the data. ³⁰⁵ During a separate 3-year retrospective study, NEPSS/SPOT detected 124 out of 134 potential Salmonella outbreaks, thereby demonstrating a sensitivity level greater than 90 percent and a positive predictive value in the range of 53 to 68 percent. ³⁰⁶

The Public Health Laboratory Information System (PHLIS) is used for the investigation and surveillance of outbreaks of specific notifiable diseases in the U.S. ³⁰⁷ Outbreak-specific Data Entry Screens are created and distributed to all reporting sites electronically. Data input and reporting occurs within hours. In addition, there is relational database software on individual personal computers with connections to laboratory and public health computers and databases. An evaluation of PHLIS used different methods to calculate expected values of Salmonella serotype Enteritidis. ³⁰⁷ This retrospective analysis demonstrated that using a 5-week mean as the threshold for what constituted an outbreak was more sensitive than using either the 5-week median or the 15-week mean. Using the 5-week mean as the threshold for determining an outbreak, the system had 3 false negatives, 76 percent sensitivity, 95 percent specificity, and 77 percent false-positive rate. ³⁰⁷ The authors remarked that their evaluation may have been affected by their definition of an outbreak (the system would have missed an outbreak of 3 cases if it occurred in a season with a high background number of reported cases). ³⁰⁷ The authors report that this system, which detected an outbreak of Salmonella serotype Stanley in May 1995, resulted in a more timely investigation of the outbreak than would have occurred with conventional surveillance -- the investigation implicated alfalfa sprouts as the vehicle of infection resulting in "the prompt development of prevention measures." ³⁰⁷

Among the laboratory systems specifically designed for surveillance of antimicrobial resistance, only The Surveillance Network™ (TSN™) Database-USA has been clinically evaluated. Laboratory data obtained from 229 laboratories "chosen for participation on the basis of their geographic and demographic characteristics" and ability to perform antimicrobial susceptibility testing are sent to the TSN™ Database-USA. One study compared in vitro susceptibility testing of isolates from 27 hospital laboratories with surveillance data for 200 laboratories from TSN™ Database-USA for the current status of fluoroquinolone activity against gram-negative species. Despite the fact that the comparison data was from different time periods and for different samples, the 2 surveillance systems agreed in many areas. The authors suggested that the discrepancies between the systems may be due to reporting practices of clinical laboratories, the geographic distribution or number of isolates, or the number of laboratories involved in the studies. ³⁰⁸

In general, the evaluative and descriptive reports of the systems collecting laboratory and antimicrobial resistance data suggest that the electronic systems improve the timeliness and sensitivity of conventional methods. Few reports specifically described how laboratory samples are handled, methods for confirmation of laboratory samples, acceptability, or cost of implementation.

Summary: Surveillance systems collecting laboratory and antimicrobial resistance data

Laboratory testing will be an essential component of any bioterrorism response effort. Systems that facilitate the collection, analysis, and reporting of notifiable pathogens and antimicrobial resistance data could potentially facilitate the rapid detection of a biothreat agent. However, none of these systems has been evaluated for such use.

Limitations of the current U.S. laboratory system for bioterrorism surveillance include lack of staffing and equipment for rapid detection in local laboratories and lack of a robust communication infrastructure among the different levels of laboratories. Efforts are ongoing to correct some of the gaps. Specifically, the Laboratory Response Network, which builds on existing laboratory capacity and is currently under active expansion, was designed so that it can be integrated into surveillance networks (such as NEDSS) and communication networks (such as California's Rapid Health Electronic Alert, Communication, and Training system (RHEACT)).

Background

The primary objective of hospital surveillance is to track hospital-acquired infections, not to identify undiagnosed infections from the community. However, hospital epidemiology systems could play 2 roles in the early detection of a covert bioterrorist attack: the identification of a cluster of recently admitted cases suggestive of a community-based outbreak, and/or the identification of a cluster of cases within the hospital suggestive of patients with an unrecognized communicable disease.

A bioterrorist attack resulting in a large number of people seeking medical attention at the same time would likely be identified by other means. However, a smaller release of a biothreat agent may result in the hospitalization of a few sentinel patients who could be detected by a hospital surveillance system. For example, if a biothreat agent that causes meningitis was released in a covert attack, patients receiving a large inoculum may present to the emergency department of their local hospitals and be admitted to the ICU. Patients receiving a small inoculum may present to their internists or urgent care clinics and be admitted to the medical wards of the same community hospital. Another group of patients may present to a neurologist and be admitted to the neurology service of the hospital. Under these circumstances, no single team of clinicians will have cared for these patients and each is likely to be unaware of the other similar cases within the hospital. The detection of such a cluster would fall to the hospital infection control service.

Alternatively, if cases of smallpox with its characteristic rash present to the emergency department, astute clinicians may recognize the disease and institute appropriate isolation measures. However, if a few patients present with acute onset of fever, chills, myalgia, cough, and chest X-ray revealing patchy infiltrates, they are not likely to be placed in respiratory isolation until Y. pestis is found in their blood cultures. The early identification of patients with contagious infections and their prompt isolation depend on the effectiveness of the hospital epidemiology service.

The hospital epidemiology IT/DSSs that are likely to identify a bioterrorism event are those that function in a timely and sensitive manner. Traditional hospital infection control surveillance relies on the manual review of suspected cases and the retrospective analysis of aggregated surveillance data. This approach tends to be labor intensive, expensive, and slow. Efforts to improve hospital surveillance with IT/DSSs have focused on automated alerts of abnormal microbiology cultures (see also the section of this chapter on Reporting and Communication Systems), identification of high-risk patients, and the detection of infection rates above a statistical background rate.

Findings

We found 16 IT/DSSs designed specifically for hospital surveillance, 10 of which have been described in peer-reviewed evaluation reports (Tables 3 and 20; Appendix H). Several of these systems were principally designed to detect hospital-specific antimicrobial resistance patterns and therefore are similar to systems presented in the preceding section. We have presented them in this section because, unlike the laboratory-based surveillance systems for antimicrobial resistance, whose purpose is largely to provide public health officials with surveillance data, the purpose of these systems is to provide information for use by hospital infection control officers. These data typically are not a part of a public health surveillance system. Several of these systems send alerts to clinicians and hospital infection control personnel (similar to some of the reporting and communication systems discussed below). We present them here because their primary purpose is to detect and reduce hospital infections.

Table

Table 20. Surveillance systems for hospital-based infections data.

Five of the 16 systems have been evaluated for their detection capabilities (others have been evaluated for other outcomes such as user acceptability which will be discussed at the end of this section). As we discussed earlier in this Report, the HELP system at the LDS Hospital in Salt Lake City uses the Data Mining Surveillance System (DMSS), which has the demonstrated ability to identify unusual patterns in surveillance data from sources including the microbiology laboratory, nurses' charts, chemistry laboratory, surgical record, and pharmacy.^{186, 191}

Researchers at the University of Alabama at Birmingham have developed a system that they also call the Data Mining Surveillance System (DMSS).^{213, 214} They performed an evaluation of the DMSS using all positive inpatient microbiology cultures for a 15-month period. Each month of the study, 475 to 677 records were used in the algorithm with a running time of less than 4 minutes. The system detected 2 outbreaks of a highly resistant strain of Acinetobacter baumannii and changes in the incidence of multi-drug resistant Klebsiella pneumoniae that were not detected by conventional hospital surveillance. It also provided geographic information suggesting that in some units (e.g., the surgical ICU) patterns of antibiotic use may have been associated with changes in antimicrobial resistance, which was unknown before the use of the system. ²¹⁴

GermWatcher is a system designed to detect both outbreaks of new infections and rising endemic rates of preexisting infections in hospitals.^340-343 Each morning, positive laboratory results are automatically transferred from the hospital database to GermWatcher. The system makes recommendations to the infection control officer to keep, discard or watch the cultures, based on the CDC's criteria for potential nosocomial infections. The system was evaluated by comparing these recommendations to those of the hospital control officers. Changes in the detection algorithms between GermWatcher Versions 1 and 2 resulted in improvement from 14 percent to less than 2 percent in rates of disagreement between the system and infection control officers. The final version of the system misclassified 3.5 percent of the 1851 cultures evaluated (2.8 percent false positives and 0.7 percent false negatives). ³⁴¹ The interpretation of these results is difficult because no additional information was provided regarding the use of the infection control officer as a gold standard (i.e., the sensitivity and specificity of their decision making).

The Danish National Hospital Discharge Registry, a registry of all non-psychiatric patients admitted to Danish hospitals, was developed to determine if hospital discharge data (clinician-provided ICD10 codes) could be used for the detection of bacteremia. ³⁴⁴ Out of 45,000 patients discharged from Aalborg Hospital, Denmark in 1994, a diagnosis of septicemia or sepsis was found in the discharge database 207 times for 186 patients. Of these, 183 episodes (88 percent) were not in the bacteremia database maintained by the regional department of clinical microbiology. Using the clinical microbiology bacteremia database as the gold standard, the sensitivity of septicemia and sepsis registration in the Danish National Hospital Discharge Registry ranged from 4.4 percent to 5.9 percent (18 to 24 cases out of 406) depending on the definition of bacteremia used. The positive predictive value of the registry was 21.7 percent (95 percent confidence interval: 12.8 to 30.5 percent) since only 18 out of 83 episodes of septicemia found in the Danish National Hospital Discharge Registry were confirmed by the clinical microbiology data. ³⁴⁴ These data suggest that the use of ICD10 data as collected in that hospital discharge registry do not significantly add to the information already available in the clinical microbiology data.

The Tucson VA Nosocomial Infection Surveillance System was designed to identify potentially preventable nosocomial infections. ³⁴⁵ During a 6-month study, of the 19 clusters of infections and 3 outbreaks of intravenous catheter-related bacteremias identified by a combination of the system and epidemiological investigations, only 3 of the 22 were identified by standard surveillance methods. Standard surveillance found no additional infections. ³⁴⁵ We conclude that this system significantly improved the detection of nosocomial infections; how this relates to the detection of inpatients with bioterrorism-related illness is unclear.

The other evaluations of the hospital surveillance systems reported similar improvements in the ability to detect hospital-based infections above what would have been detected by manual methods alone. Several evaluation reports did not report detection outcomes, rather, they primarily presented data regarding the acceptance of the system by hospital infection control officials. For example, the Hospital Infection Standardized Surveillance (HISS)^{346, 347} system was designed to improve the timeliness and standardization of data collected about nosocomial infections. Data are entered into a handheld computer, and then electronically transferred to a desktop personal computer for storage and analysis using specialized software. In an evaluation of HISS for the reporting of procedure-related infections in public acute care hospitals in New South Wales, Australia, the authors reported that although all 9 infection control officers who used the handheld device "had initial difficulties in adapting to the new technology, personnel at 7 sites consider it a useful tool. The near 100 percent completion rate for the data sets achieved by all 9 sites testify that the handheld computer assisted ICPs (infection control professionals) to collect large data sets." ³⁴⁶

In general, the systems collecting hospital-based infections data with the most compelling evidence for effectiveness are those that are integrated into a robust hospital IT infrastructure. The different systems had different purposes and therefore varied considerably with respect to the information they collected and the kinds of alerts sent to hospital infection control officials. None was specifically designed for integration into a national bioterrorism surveillance program, and it is not clear how to evaluate which of the systems presented would be best suited for that purpose. These incorporate local estimates of pathogenic prevalence and resistance with clinical data to provide hospital infection control personnel with timely, sensitive, and relatively specific analyses. None of the included systems described cost of implementation.

Summary: Surveillance systems collecting hospital-based infections data

The hospital surveillance systems that automate the collection of data from hospital-based laboratories and clinical records (i.e., the LDS and University of Alabama DMSSs, CDR, DHCP in VA hospital, and WING) are likely to be more timely than the manual systems that they replaced.

From the reports of the 5 systems that have been evaluated for their detection capabilities, we conclude that some of these systems could be a valuable tool for hospital infection control officers. However, there is little evidence demonstrating that they have sufficient sensitivity, specificity, or timeliness to detect a community-based bioterrorism event.

The integration of data from hospital-based surveillance systems, already collected in electronic format for use by hospital infection control officers, could be a valuable addition to a surveillance system organized by local public health officials. Similarly, the collection and reporting of hospital-based infections data from networks of hospitals (such as the VA) could contribute to a national bioterrorism surveillance system.

Background

On September 17, 1984, the Wasco-Sherman Public Health Department received the first case reports of the 751 victims of the intentional contamination of salad bars in The Dalles, Oregon with Salmonella typhimurium. ⁷ The lengthy epidemiologic and criminal investigations that followed demonstrated that the nation's largest foodborne outbreak of 1984 was the result of a bioterrorist attack by followers of Bhagwan Shree Rajneesh. ⁷

As evidenced by this event, terrorists can exploit vulnerabilities to our food supply. Agroterrorists may be motivated to cause human morbidity and mortality through contamination of foods during harvest, processing, or preparation, or they may be interested in creating the economic burden resulting from reduced food supply. ³⁶⁰ Attacks may be made against food and agriculture transportation systems, on water supplies, on farm workers, on food handlers, or on processing facilities. ³⁶⁰ Insect hosts for diseases affecting crop plants or livestock may be imported and released with the intent of creating an epidemic or influencing a nation's ability to export agricultural products abroad. ³⁶⁰ U.S. reliance on imported fresh fruits and vegetables has grown to such an extent that the safety of imported foods has become a source of major public health concern. ³⁶⁰

Most industrialized nations mandate the reporting of foodborne illnesses and have established surveillance systems for foodborne illnesses that collect data from health officials or clinical laboratories. In the U.S., reportable foodborne diseases and organisms include: botulism, brucellosis, cholera, E. coli O157:H7, hemolytic uremic syndrome, post-diarrheal salmonellosis, shigellosis, typhoid fever, Hepatitis A, cryptosporidiosis, cyclosporiasis, and trichinosis. ³⁶¹

Ours is not a comprehensive review of agroterrorism. We direct interested readers to other reports on the topic.^{7, 82, 360, 361} Our search produced 2 types of IT/DSSs for surveillance of foodborne illness: those that collect and analyze reports from clinicians and laboratories about the incidence and laboratory characteristics of foodborne pathogens, and those that model microbial growth responses to various food production methods. Our search also found 2 studies on active monitoring systems using implantable sensors in livestock to allow for identification of animals and surveillance of the animals' vital signs.^{362, 363} Additionally, we found reports on 3 DSSs designed to assist veterinarians and animal handlers with diagnosis and management of common animal diseases (EPIZOO, Associate, and BOVID-3).^{364, 365} However, these systems are outside the scope of our project and are not described further in this section.

Findings

In this section, we describe 10 IT/DSSs for surveillance of foodborne illnesses, 2 of which have been described in peer-reviewed evaluation reports (Tables 3 and 21; Appendix H). We note that many of the surveillance systems described in other sections (particularly those collecting laboratory data and clinician reports) also collect and report information about Salmonella species and other foodborne pathogens. The 10 surveillance systems for foodborne illness described in this section differ from these general systems in that they are designed specifically for the detection of foodborne pathogens or illness. In this section, we will first describe the systems that collect and analyze reports from clinicians and laboratories about the incidence and laboratory characteristics of foodborne pathogens. Then, we will present information about those systems that model microbial growth responses to various food production methods.

Table

Table 21. Surveillance systems for foodborne illnesses.

In the U.S., the national system for surveillance of food-borne illnesses is dependent on voluntary reporting from clinicians and laboratories. In addition, the Foodborne Diseases Active Surveillance Network (FoodNet), the principal foodborne-disease component of the CDC's Emerging Infections Program (EIP), is an active system that collects data from clinical laboratories and public health officials to estimate the burden and sources of specific foodborne diseases in the U.S.^366-372 FoodNet is a collaborative project among the CDC, the 8 EIP state health department sites, the Food Safety and Inspection Service of the United States Department of Agriculture (USDA), and the Food and Drug Administration (FDA). Results from FoodNet are used to identify infection control points, focus future prevention strategies and decision making within food safety regulatory agencies, measure changes in the burden of disease, and evaluate the effects of interventions on rates of infections over time. FoodNet collects data about 9 foodborne diseases (Campylobacter, Cryptosporidium, Cyclospora, E. coli O157, Listeria, Salmonella, Shigella, Vibrio, and Yersinia) in 8 U.S. sites (California, Connecticut, Georgia, Maryland, Minnesota, New York, Oregon, and Tennessee).

Evaluation data about FoodNet are limited to those comparing the detection of the organisms of interest in one year versus another. In Table 21, we describe 5 related systems for the collection and analysis of foodborne diseases and contaminations. Similarly, the reports of these systems provide estimates of disease incidence as identified by the system but do not further describe the systems' sensitivity, specificity, or timeliness.

The CDC has established several networks of laboratories with capabilities to perform DNA fingerprinting of bacterial strains. These methods were instrumental in demonstrating that the strain of Salmonella typhimurium in the Rajneeshpuram was indistinguishable from the organism cultured from the contaminated salad bars in The Dalles. We present 2 of these IT networks: the Salmonella Outbreak Detection Algorithm (SODA), which tracks the 50,000 clinical isolates of Salmonella that are routinely serotyped by state public health laboratories each year, and the National Molecular Subtyping Network for Foodborne Disease Surveillance (PulseNet), a similar IT for the analysis of strains of 4 foodborne pathogens (Table 21).

Other food safety ITs use mathematical models to predict microbial responses to different environmental conditions (e.g., cooking temperature) in order to recommend food preparation methods that minimize microbial contamination. These systems usually specify a hazard (i.e., pathogen), an exposure (i.e., likely amount of consumption), a hazard characterization (i.e., evaluation of the nature of the adverse effects), and a risk characterization (i.e., estimation of adverse effects given the population at risk). ³⁷³ These models are used in the production of a variety of foods, including milk, eggs, ground beef, poultry, and cheese products.^374-377 Our search found 3 IT/DSSs that incorporate these kinds of microbial prediction models (Table 21). We have included them because they could be used to model intentional contamination of the food supply by bioterrorists during the production process, although none was specifically designed with this purpose in mind. We present them in the same table with the surveillance systems for foodborne illnesses, recognizing that they are not surveillance systems, per se, but that they use surveillance data for their models.

In general, the relatively small number of organisms for which data are collected limits the usefulness of systems that perform surveillance of specific foodborne pathogens. Additionally, the lack of published evaluative information prevents a clear understanding of how these systems would function in the event of a bioterrorist attack on the food supply.

Summary: Disease surveillance of foodborne illnesses

Technologies like SODA and PulseNet have been used extensively in foodborne outbreak investigations with success -- even with investigations of outbreaks resulting from intentional contamination of the food supply. However, none of the foodborne illness surveillance systems that collect disease incidence data was specifically designed for the early detection of bioterrorist attacks on the food supply, nor has any been evaluated for that purpose. We found no evidence regarding the potential sensitivity, specificity, or timeliness of FoodNet, the active surveillance system collecting data to estimate the burden of foodborne illnesses in the U.S. Moreover, even if FoodNet was sufficiently sensitive and timely to be useful for agroterrorism detection, it is limited in that it only collects data from 8 states on 9 foodborne illnesses. The primary means for detecting an agroterrorist attack outside these states or using a different organism would be based on the analysis of voluntary reports from clinicians and laboratories.

Background

On the morning of March 18, 1996, John Major, then Prime Minister of the U.K., was presented with a memo from the Health and Agriculture Ministers confirming a link between bovine spongiform encephalopathy (BSE) in cattle and Creutzfeldt-Jakob disease (CJD) in humans. ³⁸⁵ The memo described reports of people contracting CJD by consuming beef infected with BSE. ³⁸⁵ Response to BSE in the U.K. has cost the country an estimated £4 billion, caused significant damage to the agricultural industry, and harmed Britain's relations with its European Union partners, who rapidly banned the importation of British beef. ³⁸⁵ Similarly, concerns exist that a bioterrorist attack could involve the dissemination of a zoonotic illness among animal populations with the intention of infecting humans or livestock to cause economic and political chaos.^386-388

Findings

We found 6 ITs designed to collect, process, and disseminate information on zoonotic and animal diseases, none of which has been described in a peer-reviewed evaluation (Tables 3 and 22; Appendix H).

Table

Table 22. Zoonotic and animal disease surveillance systems.

The National West Nile Virus Surveillance System (ArboNet) ³⁸⁹ is an electronic-based surveillance and reporting system, developed by the CDC to track West Nile virus activity in humans, horses, other mammals, birds, and mosquitoes. The California Encephalitis Program ³⁹⁰ monitors mosquitoes and flocks of sentinel chickens for mosquito-borne viruses such as West Nile Virus, St. Louis encephalitis, and Western equine encephalomyelitis. ArboNet and the California Encephalitis Program are the only zoonotic surveillance systems in our Report. Reports of these programs describe the type of data they collect and report the weekly incidence of the diseases under surveillance. They provide no additional information about the sensitivity, specificity, representativeness, acceptability, or other criteria by which we evaluated surveillance systems.

The National Animal Health Monitoring System (NAHMS) is an integrated national (U.S.) surveillance system that collects data on animal disease incidence and prevalence, mortality, management practices, and disease costs.^{391, 392} One of its surveillance programs is the Veterinary Diagnostic Laboratory Reporting System. Data are compiled from several sources including national animal disease control and eradication programs, information on patterns of disease based on laboratory data, selected etiologic agents, global disease distribution and reports of "unusual" laboratory findings. Results are published quarterly in DxMonitor. Another animal health surveillance program within NAHMS is the Sentinel Feedlot Monitoring program, which relies on health data collected from veterinary practitioners, who report inventory and morbidity by cause. A pilot study, involving a large commercial beef feedlot, was performed in 1988. ³⁹³ NAHMS also conducts national studies of livestock, which involve processing data obtained from questionnaires, logs, and laboratory analysis of biological specimens. ³⁹¹ These studies include the swine health survey in 1990 ³⁹² and the 1995 National Swine Study. ³⁹⁴ Data from these principally describe animal disease incidence data and related mortality and cost information but not evaluative information about the system itself.

In New Zealand, an epidemiological information system has been developed to assist disease control authorities in the containment and eradication of animal disease outbreaks. The system was initially developed to control a potential incursion of foot and mouth disease, and is called EpiMAN-FMD. By combining disease and viral spread models, farm maps, and epidemiologic parameters, the system provides statistical reports and decision support to help with the management of an outbreak. While the system has not been tested in an actual foot and mouth disease outbreak (or any bioterrorism events), it has been tested in hypothetical scenarios, and the developers hope to adapt EpiMAN-FMD to other veterinary issues.^{395, 396}

None of the reports of the systems in this section described the hardware requirements, the system's security measures, or the costs of operating the system.

Summary: Zoonotic and animal disease surveillance systems

Bioterrorism involving zoonotic and animal diseases represents a substantial threat to our national security and economy. Early detection of such an event requires effective rapid detection systems for use by farm workers, meat inspectors, and veterinarians with real-time reporting capabilities to public health officials. The surveillance systems described above provide some organizational support for national and international anti-agroterrorism efforts. However, none has been evaluated for this purpose. Our search found reports of only 2 zoonotic surveillance systems -- a major gap in bioterrorism surveillance efforts. Most of the reports provided little or no information about the timeliness of these systems; those that did suggest lag times that would be too long for effective bioterrorism surveillance.

Background

In this section, we present information about surveillance systems that met our inclusion criteria but collected sufficiently different surveillance data that they could not readily be described in the previous sections. These systems could be valuable additions to surveillance networks that integrate surveillance data from clinicians, hospitals, and laboratories. There are likely to be many other systems collecting data with potential utility for bioterrorism surveillance that have not been described in published reports or fall outside the scope of our primary research focus. For example, by September 2002, the American Association of Poison Control Centers will require all of its approximately 60 member centers to electronically report all call data at the time it is collected to the Toxic Exposure Surveillance System (TESS) database. ⁴⁰¹ Efforts are currently underway to develop analysis algorithms to systematically search this database for clusters of cases that may represent public health problems.

Findings

We present 11 surveillance systems that collect other kinds of data, 2 of which have been described in peer-reviewed evaluation reports (Tables 3 and 23; Appendix H).

Table

Table 23. Surveillance systems for other data.

These systems differ with respect to the types of surveillance data they collect and therefore differ with respect to sensitivity, timeliness, and cost. For example, EPIFAR collects drug prescription data, which may be more sensitive for a bioterrorism event than a system like Data Web, which performs surveillance on administrative databases of demographic, economic, environmental, and health data collected by a variety of organizations in the U.S. However, Data Web uses data already collected for other purposes and therefore minimizes collection burden.

Two of these 11 systems have been clinically evaluated. EPIFAR is designed to track individual prescription histories in order to provide estimates of disease prevalence in Italy. ⁴⁰² All drug prescriptions are routinely collected and processed by the Italian National Health Service (NHS). EPIFAR is a computer program used to analyze these data and determine the prevalence of selected diseases. A retrospective study of 2,550 patients who received 1 of 12 drug regimens designed for tuberculosis found that 7 times as many tuberculosis patients were identified by EPIFAR as were officially reported by clinicians. ⁴⁰² However, 88 of 250 total notified cases (35.2 percent) for the study period were not found in the EPIFAR system. ⁴⁰² Also, in a survey of physicians prescribing tuberculosis drugs for the included patients, physicians denied any tuberculosis-related problem for nearly 25 percent of the EPIFAR-identified patients. The positive predictive value of the model differed between drug regimes, with a range of 50 to 76.9 percent. ⁴⁰² To understand whether a system such as EPIFAR is likely to be useful for bioterrorism surveillance, it would be important to have a detailed understanding of the sensitivity, specificity, and timeliness of each of the drugs (or combinations of drugs) under surveillance. For example, if EPIFAR collected surveillance data exclusively for a medication such as isoniazid, which is typically used exclusively for the treatment of mycobacterial diseases, it is likely to have better specificity (albeit perhaps decreased sensitivity) than a combination of medications including those that are used for many diseases. Given the current evidence, it is not clear which drugs (or combination of drugs) should be selected for a bioterrorism surveillance program that attempts to achieve a given degree of sensitivity for all or most of the most worrisome biothreat agents.

Medical Historian electronically asks patients questions about their medical histories to derive a historical database of patient information that can be queried for the purpose of disease surveillance. The system performs a computerized elicitation of a review of symptoms. "Diseases" are defined according to symptom-clusters. The database is then scanned to search for outbreaks of symptom-clusters. In a retrospective evaluation of 288 patients with cough and rhinitis, the computer diagnosis was within the 95 percent confidence interval of expert panel diagnosed-conditions for upper respiratory infection. Sensitivity varied from 27 to 90 percent depending upon the symptom-cluster used. No assessments of specificity were provided. Data on the sensitivity of Medical Historian for 4 other computer-surveyed diseases were not stated, but the authors did note that "the number of encounters identified electronically was always fewer than the number identified by the physician panel." ⁴⁰³ It is difficult to interpret these results in terms of surveillance of bioterrorism-related illness.

The HAWK system for surveillance of reportable diseases in Kansas is a promising system that collects notifiable disease reports from clinicians and a variety of public health agencies in the state using a data warehouse model.^{404, 405} Authorized public health officials can remotely access the data via the Internet to perform analyses. No evaluative data are available about HAWK, which could serve as a model for similar statewide reporting systems.

None of the reports of the systems in this section described the type of hardware required, the system's security measures, or the costs of operating the system.

Summary: Surveillance systems collecting other kinds of data

None of the specific systems described in this section has been designed or evaluated for surveillance for bioterrorism, and only EPIFAR and Medical History have been clinically evaluated. When we consider the systems in this and all the preceding Surveillance sections, we have described at least 1 surveillance system that collects each of the data sources described in Figure 3. However, there are significantly fewer descriptions of systems collecting the earliest surveillance data -- a significant gap in the literature. For example, given that the 53.1 million school-aged children (aged 5 to 17) represent 18.9 percent of the total U.S. population (281.4 million) and that their absenteeism is collected on a daily basis nationally (albeit not always electronically), an evaluation of the sensitivity, timeliness, and cost of this source of surveillance data seems warranted. ⁴⁰⁶ The surveillance of work absenteeism rates is complicated by the decline in the types of industries that require employees to clock in and out of work and by the lack of these data in an electronic format. Systems that collect pharmaceutical data, like EPIFAR, hold particular promise for bioterrorism surveillance. Pharmaceutical data, particularly over-the-counter medication sales data, could provide an early, if not specific, indication of an outbreak. Additionally, most pharmaceutical sales are tracked electronically. The evaluation of EPIFAR emphasizes the complexity of selecting specific pharmacy data for bioterrorism surveillance. A careful analysis of the detection characteristics of common prescription and non-prescription medications used for the bioterrorism-related syndromes (e.g., antipyretics, cough suppressants, and antidiarrheals) will be required to determine the utility of these data for bioterrorism surveillance.

Background

Once surveillance data are collected, analysis -- typically by public health officials -- is required to identify patterns suggestive of bioterrorism-related illness. Our search identified a number of methods for analysis of surveillance data. We note however, that we have not conducted a systematic review of surveillance analysis methods per se but have identified methods that have been or could be applied to the analysis of bioterrorism-related illnesses. In this section, we describe the important features of analytic methods for bioterrorism surveillance data.

The primary analytic question in prospective disease surveillance is whether the currently observed disease process differs from the expected process. Before answering this question, the analyst must determine what elements of the data will be modeled: individual attributes (e.g., gender, age), time, and/or space. Additionally, the analyst must consider how the expected process (i.e., the baseline) will be modeled and what threshold will be used to establish a significant change from baseline (i.e., an aberrant disease process). Traditionally, time has been the major modeling consideration, ⁴¹⁶ but recent developments in technology and statistics have greatly facilitated the consideration of spatial information in surveillance analyses. We first examine modeling decisions concerned with generating the expected disease process, and then discuss developments in spatial analysis and their implications for bioterrorism detection.

Findings: Modeling the baseline characteristics of surveillance data

Models of the expected disease process are typically derived from the historical pattern of disease, either over the immediately preceding time interval (e.g., 30-day moving average), or over one or more historically corresponding intervals (e.g., the mean rate for the first week in January over the past 5 years).^{417, 418} This approach is straightforward, but it ignores the underlying dynamics of disease propagation through the population. Several articles identified in our search describe methods for stochastically modeling the spread of communicable disease epidemics.^419-423 These methods have not been widely used for the modeling of disease surveillance, but they may allow more accurate determination of the expected disease rates and deviations from the expected.

More accurate descriptions of the expected and observed disease rates should enable more accurate identification and characterization of data aberrations and disease outbreaks. Because the need for timeliness is particularly acute in bioterrorism surveillance, application of these methods to disease surveillance merits further attention. It should also be possible to apply similar methods to model the disease processes associated with outbreaks of non-communicable diseases of bioterrorism, such as anthrax. Potential drawbacks of this modeling approach include its added complexity and the need to collect or estimate data for a number of parameters, such as the proportion of susceptible individuals, herd immunity, and mixing rate.

Findings: Consideration of space in surveillance analyses

The importance of spatial location for understanding disease processes has been appreciated for many years. Spatial location has a central role in the epidemiological triad of "person, place and time," and maps have long been used to identify important patterns in diseases such as cholera, cancer, leprosy, pneumonia, and smallpox. ²⁵² Consideration of the spatial dimension of surveillance data may enable more timely identification and characterization of important patterns. Recent developments in IT and statistics, most notably the development of user-friendly geographic information system (GIS) technology, can facilitate consideration of space in surveillance data; however, the effective application of these developments to surveillance analysis is not yet well established. ⁴²⁴ In this section, we examine how GIS technology and other spatial analysis tools have been applied to bioterrorism-related diseases.

A GIS is an automated system for collecting, organizing, analyzing and presenting geographically referenced data. Beyond simple mapping, a GIS can perform complex functions such as automated address matching, distance calculations, buffer analyses (i.e., calculation of a buffer zone of variable width around a point, line, or area), spatial queries (i.e., the ability to select observations based on their geographic characteristics), and linking data sources by spatial location. ⁴²⁵ The primary effort to develop global mapping capability for public health data has come from the joint WHO/UNICEF program HealthMap -- a data management, mapping and GIS system for public health. The program was initially created in 1993 to establish a GIS to support management and monitoring of the Guinea Worm Eradication Programme. ⁴²⁶ Since 1995, however, in response to the increasing demand for mapping and GIS technologies from a much wider range of public health administrators, the scope of the work has been broadened to include the promotion and use of GIS technology for other disease control programs. ⁴²⁶

Most applications of GIS that we identified in our review performed retrospective analyses to identify spatial patterns of disease and/or disease vectors. However, several of the syndromal surveillance systems described in this Report (such as ESSENCE and RSVP^®) use GIS or spatial analysis in an ongoing and systematic manner to identify disease outbreaks or data aberrations. Some studies used Global Positioning System technology to locate disease cases.^{427, 428} Analytic methods employed within a GIS environment ranged from simple, descriptive spatial analyses,^429-431 to more complex spatial^{432, 433} and space-time modeling.^{427, 428} The main objective of the analyses was usually to characterize the relationship between disease vectors and cases. In some instances, this relationship was used to forecast future disease outbreaks based on predicted disease vector ecology.^433-436

Summary: Analysis and presentation of surveillance data

The studies presented, with their focus on predicting new outbreaks from previous outbreaks and disease vector distributions, may seem unrelated to bioterrorism surveillance. The lack of previous large-scale bioterrorism attacks, and the fact that most agents of bioterrorism are not vector-borne, prevent this type of predictive modeling for bioterrorism. However, a number of the specific technologies and GIS/spatial analytic methods employed in these studies may be directly applicable to the analysis of bioterrorism surveillance data. These methods and technologies include remote sensing, data integration, spatial interpolation, and space-time statistical analysis. For example, remote sensing data may help to identify potential effects of meteorological conditions for airborne dispersal of a bioterrorism agent. The ability of a GIS to integrate disparate data sources (e.g., pharmacy sales data, ambulance activity, and ICD9 codes) according to their spatial location may enable the identification of important relationships that indicate early disease behavior. Spatial interpolation could be used to predict risk in locations for which similar data are not available. Finally, advanced space-time analytic methods can take advantage of the spatial dimension of data to detect aberrations with greater sensitivity and timeliness. The combination of these analytic methods with rich descriptions of the expected disease process (as described previously) may provide even greater sensitivity for identifying bioterrorism attacks in the midst of normally occurring disease outbreaks. We note that no published report has specifically evaluated whether a surveillance system that uses both temporal and spatial analyses is likely to be more timely or sensitive than a system that performs only temporal analyses.