Conceptual model

We developed a conceptual model to specify the decisions and tasks involved in diagnosis, management, prevention, surveillance, and communication by clinicians and public health officials in the event of a bioterrorist attack. We used a process called task decomposition to specify the data requirements that need to be incorporated into an IT/DSS for it to assist clinicians and public health officials make these decisions. This list of tasks and data requirements served as the foundation of our evaluation system of the currently available IT/DSSs.

Inclusion and exclusion criteria

Based on input from our expert advisory panel, the conceptual model, the task decomposition, and practical considerations, we developed an inclusion-exclusion strategy to identify articles that described or evaluated IT/DSSs.

Selection of quality scales

We used a scale developed at McMaster University to rate the quality of evidence from peer-reviewed evaluations of IT/DSSs for diagnosis, management, and communication. For reports of surveillance systems, we used an evaluation scale published by the Centers for Disease Control and Prevention (CDC).

Literature sources

In consultation with professional research librarians, we developed a search strategy for references from 3 sources: peer-reviewed articles, government reports, and Web-based information. For the peer-reviewed articles, we identified 5 databases of medical, scientific, and government references likely to contain reports of relevant IT/DSSs: MEDLINE^® (January 1985 to April 2001), the Catalog of U.S. Government Publications, GrayLIT, the Library of Congress, and the National Technical Information Service. We identified the 16 government agencies most likely to fund, develop, or use IT/DSSs that could also be used by clinicians or public health officials. We also planned Internet searches to retrieve reports of potentially relevant IT/DSSs from sites other than those operated by government agencies (e.g., academic and commercial sites).

Search strategies

We developed separate search strategies, one for MEDLINE^® and one for the government documents and Web-based information. Each included terms such as bioterrorism, biological warfare, information technology, decision support system, diagnosis, management, therapeutics, communication, surveillance, public health, and epidemiology. Additional articles were identified by members of our expert advisory panel, from conference proceedings, and by review of reference lists.

Data collection and analysis

We reviewed titles, abstracts, and full-length articles as necessary to identify potentially relevant articles. All peer-reviewed articles that met our inclusion criteria were blinded to the investigators, 2 of whom independently abstracted study information onto a data-abstraction and quality-assessment form.

We abstracted the following data from all included articles: the purpose and description of the system (e.g., detection, diagnosis, management, surveillance, or communication), whether the system had been clinically evaluated and the results of these evaluations, what security measures the system uses, what kind of reasoning the system uses, and information about the quality of the report. A draft Evidence Report was critiqued by 16 expert advisors and 17 peer-reviewers who had expertise in nursing, clinical medicine, public health, hospital management, informatics, diagnostics, emergency management, epidemiology, national security, toxicology, and food safety.

Detection systems

We identified 55 detection systems that collect and identify potential biothreat agents within environmental and clinical samples. Many of these systems were developed for use by the military and some were adapted for civilian purposes. Few reports compare detection systems to a gold standard, and their sensitivity (i.e., the likelihood that the detection system will give a positive result when testing a sample containing a biothreat agent) and specificity (i.e., the likelihood that the detection system will give a negative result when testing a sample that does not contain a biothreat agent) remain poorly characterized in the publicly available literature. Most identification systems are limited in that each test cycle can evaluate a sample for only a single biothreat agent, often run only a limited number of samples at a time, and cannot test for many of the most worrisome agents (e.g., smallpox). We found no reports that directly compared 2 or more of the commercially available systems in any given category. The paucity of comprehensive evaluative information about these systems prevents conclusions about whether or not these systems are likely to serve the detection needs of first-responders, clinicians, and public health officials during a bioterrorist event.

Diagnostic systems

We identified 23 diagnostic systems with potential utility for enhancing the likelihood that clinicians consider the possibility of bioterrorism-related illness. These systems are generally designed to assist clinicians in developing a differential diagnosis for a patient who has an unusual clinical presentation. We found 6 general diagnostic systems, 4 systems designed to improve radiologic diagnoses, 4 telemedicine systems, 4 systems for the diagnosis of infectious diseases, 1 system for the diagnosis of dermatologic lesions, 1 system for the diagnosis of community-acquired pneumonia, and 3 systems for other purposes. None of these DSSs has been evaluated formally with respect to bioterrorism response. In an evaluation of a DSS for infectious diseases that has more than 20 biothreat agents in its knowledge base, the system was able to list the actual diagnosis in an output of possible diagnoses for nearly 95 percent of 495 actual and hypothetical cases. However, this system is limited in that it is specific for infectious diseases; consequently, even those clinicians with access to this technology may not use it if the patient does not present with a fever or other signs or symptoms of infectious disease.

Three of the general diagnostic DSSs have been evaluated for their intended (non-bioterrorism related) purposes. In these evaluations, the general diagnostic DSSs typically performed better than physicians-in-training but not as well as experienced clinicians. However, the accuracy of the DSSs decreased for difficult cases. The need for clinicians to manually enter patients' signs and symptoms into diagnostic DSSs -- a laborious step that may be a barrier to the use of these systems and has been demonstrated to increase inter-user variability -- is eliminated by the few systems that automatically collect patient data from an electronic medical record. For example, there are diagnostic DSSs currently available in hospitals with electronic medical records that provide clinicians with an estimate of the likelihood of community-acquired pneumonia or active pulmonary tuberculosis based exclusively on data collected from the medical record. Many diagnostic DSSs use probabilistic information about the likelihood of disease. Because bioterrorism-related illness is relatively rare, in the event of bioterrorism these systems will have inappropriately low pretest probabilities for biothreat agents. None of the reports of diagnostic DSSs described the ability to change the probability of disease based on information about suspected bioterrorism events.

Management and prevention systems

Management and prevention systems are designed to make recommendations to clinicians by abstracting clinical information from electronic medical records to make patient-specific recommendations. None of the 18 systems identified in this review has been specifically designed or evaluated for utility in providing management or prevention recommendations during a bioterrorism event; however, 10 of them have been evaluated for their intended purpose. These evaluations demonstrate that the expert systems that continuously search electronic medical records (including data from the laboratory, radiology reports, and clinician notes) for new evidence of infection and apply clinical practice guidelines to those data are able to affect clinicians' antibiotic selection decisions and increase compliance with clinical practice guidelines. We have no information as to whether the knowledge bases of these systems include comprehensive information about bioterrorism-related illnesses. The systems that are not linked to electronic medical records share many of the limitations of the general diagnostic systems -- including, that clinicians may not use the system to seek advice for patients presenting with common viral syndromes (i.e., the bioterrorism-related syndromes). Antibiotic recommendation programs are typically designed to provide recommendations for antibiotics with the narrowest possible spectra, thereby reducing the risk of developing resistant organisms. If clinicians make antibiotic selection decisions while unaware of the true bioterrorism-related diagnosis and select narrow-spectrum antibiotics, they may not be effective against biothreat agents. Therefore, whether the use of these systems would be helpful or detrimental is not known.

Surveillance systems

We identified 90 surveillance systems that collect a variety of surveillance reports: 7 for syndromal surveillance, 6 for reports from clinicians, 11 for influenza-related data, 23 for laboratory and antimicrobial resistance data, 16 for hospital-based infections data, 10 for food-borne illness data, 6 for zoonotic illness data, and 11 for other types of surveillance data. For a surveillance system to detect a covert bioterrorist event, it must collect data that are sensitive and specific for biothreat agents, analyze the data, and report results to public health decision makers in a timely manner. None of 90 included surveillance systems has been evaluated for its utility in detecting a bioterrorism event. Forty of 61 reports of evaluations or descriptions of surveillance systems described the timeliness, importance of the health event under surveillance, and usefulness of the system. However, less than one-third of the reports of evaluations of surveillance systems described the representativeness, simplicity, sensitivity, specificity, acceptability, or flexibility of the system. The quality of the evidence regarding the effectiveness of the systems reported by these articles is therefore limited. Most of the evaluations of surveillance systems demonstrated that the electronic collection and reporting of surveillance data improved detection over older, manual methods. When we consider the 90 surveillance systems described in this Report, there are relatively few systems collecting the earliest surveillance data -- such as school and work absenteeism, calls to telephone care nurses, over-the-counter pharmacy sales, or veterinary or zoonotic illness -- a potentially significant gap in available surveillance systems.

Communication systems

Eight of the 26 communication systems were designed for communication among public health officials at local, state and federal levels (e.g., Web-based discussion and reporting of surveillance data). In pilot evaluations directed by individual state health departments, these systems securely managed the disease reporting needs of local and state public health officials. However, these systems were limited to communication within a state. We found no single system that effectively links members of the public health community at national, state and local levels. However, there are ongoing efforts (such as the Urban Security Initiative project of Los Alamos National Laboratory, EpiX, Health Alert Network and RHEACT) designed to integrate communication of public health information vertically and horizontally within the U.S. public health system. Five systems were designed for the automated communication of information from hospital-based electronic medical records to clinicians (e.g., alerting systems to notify clinicians of abnormal laboratory tests). These systems have been subjected to the greatest evaluation of all the communication systems. Despite being limited to institutions with electronic medical records, they could potentially play an important role in improving the timely recognition of bioterrorism-related illness. Three systems facilitated communication between emergency departments and first-line emergency response personnel. ProMED^© has demonstrated the capacity for rapid reporting and dissemination of information on a wide range of infectious diseases resulting from both naturally occurring and bioterrorism-related events. During a bioterrorism event, clinicians must be able to rapidly communicate with their patients. Systems exist that enable Web-based communications between these parties in a manner compliant with the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Robust security measures that ensure patient confidentiality and resist cyberattack will be a necessary component of any bioterrorism-related communication system.

Key Question 4: In areas where existing IT/DSSs do not meet the information needs of clinicians or public health officials, what functional and technical considerations are important in the design of future IT/DSSs to support response to bioterrorism events?

We did not identify evaluations or studies that directly assess the functional and technical requirements that are important for future IT/DSSs. In this section, we provide our interpretation of factors that could be considered for the design of future IT/DSSs.

• IT/DSSs for bioterrorism need to have documented sensitivity, specificity, and timeliness that are appropriate for their intended use. Because both false-positive and false-negative results can result in serious adverse outcomes, sensitivity and specificity should generally be high. Similarly, timeliness is of critical importance for IT/DSSs that aid with detection, diagnosis, management, communication, and surveillance. Systems should have measures to maintain security of samples and data collected.
• Detection and diagnostic systems must be in use in the affected area. In the event of a covert attack collection systems will have to be in place in areas of likely attack. In the event of a known attack, these systems must be portable and sufficiently rapid that they can be used in a variety of field and clinical situations.
• Clinicians would be helped by detection methods that include all of the most worrisome bio-threat agents, by systems that can test a individual sample for multiple biothreat agents simultaneously, and by systems that can run multiple samples simultaneously.
• Because the individuals collecting and analyzing the environmental and clinical samples are often at considerable distance from public health decision makers, detection systems could benefit from the capacity for secure transmission of data to these decision makers.
• Efforts to link diagnostic and management or prevention DSSs to other hospital information systems would reduce the data entry burden substantially.
• The knowledge bases of diagnostic and management systems need to include current information and clinical practice guidelines about bioterrorism-related illness. The systems need to be able to appropriately adjust the probability of disease caused by biothreat agents if a known bioterrorism event has occurred.
• Efforts to integrate surveillance data may benefit from definitions of the syndromes under surveillance; comprehensive analysis of the sensitivity, specificity, and timeliness of each source of surveillance data; improved spatial and temporal analysis methods; and systems that collect sources of data reflecting disease earlier in the course of illness (e.g., school and work absenteeism and over-the-counter pharmacy sales).
• Communication systems that protect patient confidentiality and have adequate security measures would be useful for the rapid dissemination of outbreak-related information among all relevant decision makers, including public health officials, clinicians, and the public.