Key Question (KQ) 1. What clinical conditions are associated with the greatest number and highest risk of ED diagnostic errors and associated harms?

Although limited by biases in the data towards diseases causing more severe harms when missed, the top 20 individual diseases associated with diagnostic errors (independent of harm severity), in approximate rank order, were found to be fracture, stroke, myocardial infarction, appendicitis, venous thromboembolism, spinal cord compression and injury, aortic aneurysm and dissection, meningitis and encephalitis, sepsis, traumatic brain injury and traumatic intracranial hemorrhage, arterial thromboembolism, lung cancer, ectopic pregnancy and ovarian torsion, pneumonia, testicular torsion, gastrointestinal perforation and rupture, spinal and intracranial abscess, open and non-healing wounds, cardiac arrhythmia, and intestinal obstruction (with or without hernia). It is likely that this list of misdiagnosed diseases, which derive from two large “numerator-only” studies (one malpractice-based and one incident report-based), is strongly skewed by reporting bias towards diseases that, when missed, lead to serious harms. It is also likely that the list is skewed towards false negatives relative to false positives; many “benign” diseases that do not cause immediate threat to life or limb are likely missed in far higher total numbers than most of these disorders. Finally, it may also be partly skewed towards errors likely to be confirmed in hindsight by radiographic review, including both fractures and lung cancer.

Best available evidence indicates that the top 15 individual diseases associated with the greatest number of serious misdiagnosis-related harms in the ED, in rank order, were (1) stroke, (2) myocardial infarction, (3) aortic aneurysm and dissection, (4) spinal cord compression and injury, (5) venous thromboembolism, (6/7 – tie) meningitis and encephalitis, (6/7 – tie) sepsis, (8) lung cancer, (9) traumatic brain injury and traumatic intracranial hemorrhage, (10) arterial thromboembolism, (11) spinal and intracranial abscess, (12) cardiac arrhythmia, (13) pneumonia, (14) gastrointestinal perforation and rupture, and (15) intestinal obstruction. These data derive from a large, nationally representative study of malpractice claims in the United States¹⁷ and are bolstered by corroborating data from a similarly large, nationally representative incident reporting system in the United Kingdom¹⁶; together these two studies represent 78 percent of the diagnostic error cases analyzed for KQ1 (n=4,561 of 5,817). These results are further bolstered by results from a recent malpractice study that was not included in the report, because it was identified after completion of our grey literature search. This was a report on ED diagnostic errors from The Doctor’s Company which also identified stroke as the top category, stating, “The top categories for final diagnosis among the settled claims differed slightly. The highest classification remained cerebrovascular disease, but at a larger percentage (18 percent).”³¹⁴ Unsurprisingly, spinal abscess, myocardial infarction, aortic aneurysm and dissection, arterial thromboembolism, and sepsis all also appeared among the top missed conditions. It is possible that missed myocardial infarctions and lung cancers may be overrepresented in malpractice claims, so their ranks could be overstated. However, it is also likely that this is a relatively unbiased list of diseases leading to serious misdiagnosis-related harms. The source data (which are organized by the final, correct diagnosis) likely reflect almost entirely false negatives, but this is probably still an accurate reflection of serious misdiagnosis-related harms. Put differently, death or permanent disability is probably a rare outcome among patients with non-life- or limb-threatening diseases mistaken for dangerous ones (e.g., migraine mistaken for stroke and leading to excess imaging and hospital admission). Nevertheless, complications (including death) can certainly occur, especially when invasive procedures are involved, such as when surgery is performed because of a false positive appendicitis diagnosis.³¹⁵ The precise frequency of such adverse outcomes is unknown, but it is likely that such cases would appear in medicolegal claims with equal or greater odds relative to false negative diagnoses of dangerous illnesses, since the legal claim must present evidence that the patient’s outcome would have differed but for the diagnostic error, and this is more easily proven for a patient whose misdiagnosis is a false positive (i.e., “healthy” without the disease) who suffers a complication from treatment for an incorrect diagnosis than for a patient whose misdiagnosis is a false negative (i.e., “sick” with the disease) who suffers from the disease itself.

Taken together, these 15 diseases account for an estimated 68 percent of all serious harms from diagnostic error in the ED. The so-called “Big Three” disease categories (vascular events, infections, and cancers), in their totality, account for an estimated 72 percent of all ED diagnostic errors resulting in serious misdiagnosis-related harms. However, major vascular events (42%) and infections (23%) substantially outnumber cancers (8%) in the ED clinical setting. Pediatric populations have fewer high-severity harms than adults and, unlike adults, more infections than vascular events; less is known about the ranks of specific disease distributions.¹⁷

When considering ED diagnostic errors of mixed severity, missed fractures are the most frequent conditions reported in malpractice claims and incident reports.^{16, 31, 71, 80, 90} However, the level of harm associated with most missed fractures is generally lower than that for missed major medical and neurologic events,¹⁷ so they are not among the more common causes of serious misdiagnosis-related harms to patients. Perhaps more importantly, they may be overrepresented in claims as well as incident reports due to ascertainment and reporting biases, perhaps related to the relative ease with which radiographic misdiagnosis can be documented (i.e., using the tangible artifact of the radiograph), even well after the fact (see KQ1a above for details). Epidemiologic data suggest that other diagnostic errors (e.g., for conditions producing lower-severity harms and unaccompanied by radiographs) are likely far more frequent than fractures yet go unaccounted for in malpractice claims or incident reports. For example, missed diagnoses of inner ear diseases are likely an order of magnitude more frequent than fractures, yet do not appear on “top ten” lists of the most commonly missed conditions (see KQ1a). Missed appendicitis is also commonly noted in such reports, but data on frequency are conflicting.

The most commonly misdiagnosed clinical presentations may be abdominal pain, trauma, and neurological symptoms (e.g., dizziness, headache, back pain). However, data are sparse.

Gaps filled: Prior to this report, there was a clear evidence gap regarding the most frequent diseases missed in the ED, and data from different sources appeared conflicting. Best available evidence regarding the most frequent causes of serious misdiagnosis-related harms has now been synthesized, and clearly points to missed vascular events and infections as the principal causes, with stroke the undisputed leader in total serious harms, particularly permanent disability. Just 15 diseases likely account for more than two-thirds of all serious harms; this means that eliminating preventable patient harms from ED diagnostic error is more tractable than previously imagined.

Gaps identified: A number of gaps were identified in preparing this report. These are described below in the section on Strengths and Limitations (Evidence subsection).

KQ2. Overall and for the clinical conditions of interest, how frequent are ED diagnostic errors and associated harms?

Although based on just a few higher-quality studies less likely to be impacted by systematic under-ascertainment bias, best available evidence indicates that an estimated 5.7 percent (95% confidence interval [CI] 4.4 to 7.1) of all ED visits will have at least one diagnostic error. The overall (not disease-specific), per ED visit, potentially preventable diagnostic adverse event rates were estimated as follows: any harm severity 2.0 percent (95% CI 1.0 to 3.6), serious misdiagnosis-related harms (i.e., permanent, high-severity disability or death) 0.3 percent (plausible range [PR] 0.1 to 0.7), and misdiagnosis-related deaths 0.2 percent (PR 0.1 to 0.4). For each misdiagnosis-related death, it is estimated that there are roughly 0.41 (PR 0.27 to 0.60) ED patients suffering non-lethal, permanent, serious disability. If generalizable to all U.S. ED visits (130 million), that translates to over 7 million ED diagnostic errors, over 2.5 million diagnostic adverse events with preventable harms, and over 350,000 serious misdiagnosis-related harms, including more than 100,000 serious, permanent disabilities and 250,000 deaths. This is equivalent to a diagnostic error every 18 patients, a diagnostic adverse event every 50 patients, a serious harm (serious disability or death) about every 350 patients, and a misdiagnosis-related death about every 500 patients. Put in terms of an average ED with 25,000 visits annually and average diagnostic performance, each year this would be over 1,400 diagnostic errors, 500 diagnostic adverse events, and 75 serious harms, including 50 deaths. These estimates corroborate the National Academy of Medicine (NAM) position that improving diagnosis is a “moral, professional, and public health imperative.”⁵

The overall preventable diagnostic adverse event rate of 2.0 percent and misdiagnosis-related death rate of 0.2 percent both come from the only high-quality, prospective study to look at diagnostic adverse events using systematic phone and chart review follow-up on 503 patients both discharged and admitted from the ED. The death rate from such a small study is necessarily imprecise, but supported by corroborating, indirect evidence from other sources, including the other high-quality prospective study of mortality (see report text of KQ2 for elaboration). Retrospective trigger-based studies included many more ED visits (sometimes hundreds of thousands) and often revealed substantially lower rates, but this was almost certainly due to systematic under-ascertainment, as described in the report text for KQ2. Estimates of diagnostic adverse events varied more than 100-fold across studies (i.e., across hospitals) from 0.01 percent at a large, U.S.-based tertiary care ED to 1.6 percent at a small regional ED in Denmark. It is unknown how much of this high degree of variation is real versus study design related.

Variation in diagnostic error rates by disease were striking, with the lowest per-disease diagnostic error rate being for myocardial infarction (false negative rate 1.5%), well below the estimated average diagnostic error rate across all diseases (5.7%). Most of the top harm-producing dangerous diseases are initially missed in the ED at rates of 10 to 36 percent, but spinal abscess is likely the principal high outlier with 56 percent missed initially. There is roughly an inverse relationship between annual disease incidence and diagnostic error rates, although myocardial infarction is clearly a low outlier. Among the diseases producing frequent death or disability, myocardial infarction stands alone as an exemplar for which ED miss rates have been reduced to a near-zero level, and its rank in malpractice studies may be overstated.

Gaps filled: Prior to this report, there was a clear evidence gap regarding the frequency of diagnostic errors and misdiagnosis-related harms, and data from different sources were highly variable and difficult to compare. Importantly, specific studies identified during the review strongly point to a high degree of systematic under-ascertainment of both errors and harms in the most common types of retrospective studies. Best available evidence regarding the frequency of diagnostic errors and harms both per ED visit and per disease case has now been synthesized. Evidence clearly points to a large public health burden of ED diagnostic errors and rates of diagnostic error for most dangerous diseases that offer a fair amount of “room for improvement.” We also present the first meta-analytically supported data on increased mortality from diagnostic error. Finally, demonstrating what appears to be large inter-ED variability in diagnostic error rates suggests many errors are likely remediable, rather than “the price of doing business.”

Gaps identified: A number of gaps were identified in preparing this report. These are described below in the section on Strengths and Limitations (Evidence subsection).

KQ3. Overall and for the clinical conditions of interest, what are the major causal factors associated with ED diagnostic errors and associated harms?

Best available evidence indicates that cognitive errors dominate. Although errors were often multifactorial, nearly 90 percent of cases involved failures of clinical decision-making or judgment, regardless of the underlying disease present. Key process failures were errors in diagnostic assessment, test ordering, and test interpretation. Most often these were attributed to inadequate clinical knowledge, skills, or reasoning, particularly in “atypical” cases.

Atypical presentations, non-specific symptoms, and diseases that seem “out of place” (e.g., stroke in a younger patient or appendicitis in an older patient) were among the strongest and most consistent predictors of increased risk for a missed diagnosis across diseases. In other words, clinicians do not miss diagnoses when they are obvious, they miss them when they are subtle. Therefore, solution-making to eliminate preventable harms from diagnostic error must be focused entirely on subtler disease presentations, not obvious ones. For example, it is thoroughly insufficient to attempt to tackle missed stroke in the ED by strengthening existing stroke treatment pathways and reducing door-to-needle times for administration of thrombolytic therapies. Instead, it is essential to create mechanisms that rapidly identify patients with subtle stroke symptoms which are prone to be missed (e.g., dizziness and headaches), in order to bring such patients into stroke treatment pathways so they too may benefit from prompt therapy (e.g., dual antiplatelet therapy for early secondary prevention, which, if applied in the first 24 hours, lowers risk of major stroke after minor stroke or transient ischemic attack by 34% over the next 21 days³¹⁶).

Taken together, this suggests that interventions to reduce harm from ED diagnostic error must directly tackle problems in bedside diagnostic skills and clinical reasoning for atypical presentations of the 15 diseases producing the most harm. If substantial headway is to be made, we must develop system-wide solutions to address these cognitive problems.² Options fall into three basic mechanisms that all target increasing the availability of diagnostic expertise:

(1)

enhance the expertise of ED clinicians through deliberate practice training and feedback;

(2)

support ED clinicians’ decision-making through teamwork, including access to experts;

(3)

minimize cognitive load by deploying technologies that digitally encapsulate expertise.

Achieving equity in diagnosis by addressing diagnostic health disparities is of acknowledged importance to achieving diagnostic excellence.³¹⁰ Studies that normalized for baseline risk of having the target disease often found an association with gender, race, or ethnicity, with a roughly 20 to 30 percent increased risk of diagnostic error for women and minorities.

Gaps filled: Prior to this report, there was a clear evidence gap regarding the overall causes of diagnostic errors and misdiagnosis-related harms in the ED, including both root causes and contextual risk factors. Clear results here point to a high frequency of cognitive errors in cases with subtle or atypical clinical presentations. This identifies a clear target for systems-based interventions that target cognitive error—increase the availability of diagnostic expertise at the point of care for dangerous diseases with a known high rate of misdiagnosis-related harms.

Gaps identified: A number of gaps were identified in preparing this report. These are described below in the section on Strengths and Limitations (Evidence subsection).

Evidence

Overall, the evidence available supported answers to all three Key Questions, including a majority of the sub-questions. On KQ1 (diseases), the literature was relatively strong for diseases causing more severe harms but fairly weak on the disease distribution for lower-severity errors. On KQ2 (frequency), the literature was strong on false negatives but relatively weak on false positives. Estimates for overall error and harm rates were drawn principally from three smaller studies (combined n=1,758), none U.S.-based, but these were the only studies that did not restrict patients by disease and still conducted systematic patient follow-up to minimize under-ascertainment of diagnostic errors. There is reason to believe that both the overall and disease-specific results generalize to U.S.-based EDs (see Applicability Section). On a disease-specific basis, literature about error frequency was strongest for stroke, myocardial infarction, and aortic aneurysm and dissection; weaker for venous thromboembolism, meningitis and encephalitis, sepsis, arterial thromboembolism, spinal abscess, pneumonia, appendicitis, fractures, and testicular torsion; and absent for endocarditis, necrotizing enterocolitis, sudden cardiac death, arrythmias, congenital heart disease, ectopic pregnancy, and pre-eclampsia/eclampsia. On KQ3 (causes), the literature was strongest for patient and illness characteristics and relatively weaker on clinician characteristics, fixed systems factors, and dynamic systems factors. Overall, there is a relative paucity of literature on diagnostic errors among pediatric ED populations. More studies are warranted, including research on how the distribution of diseases (KQ1), rates of diagnostic error (KQ2), and causes/risk factors (KQ3) differ from those in adult patients. Specific gaps identified for each question with potential remedies are described below for each KQ.

The list of diseases under consideration for the overall search and, specifically, KQ2, was prespecified on the basis of prior literature and informed by the use of a TEP and Key Informant interviews. This approach was chosen because, in the timeframe for conducting the work, it was not possible to complete the KQs “in series” (i.e., to do KQ1 first and then start the search anew for KQ2 and KQ3). Thus, this was the only methodologically feasible approach. This represents a limitation (particularly as relates to the list of diseases assessed in KQ2). We have assessed the impact of this limitation through the final results derived from KQ1, and the impact appears to have been modest. The prespecified list appears to have been fairly complete vis-à-vis the most common causes of misdiagnosis-related harms—for example, in the largest incident report study of ED diagnostic errors (n=2,288) (which was not used to determine the prespecified list), all top 12 conditions found in that study (Hussain et al., Table 1¹⁶) appeared in our prespecified list. No other conditions identified in that study had higher individual frequency, and, collectively, all of those other conditions combined accounted for just 30 percent of the total incidents reported (n=679/2,288) (i.e., our list embraced more than 70% of the total incident reports related to diagnostic error and all of the top conditions). Our prespecified list included searches for error rates for 14 of the top 20 diseases identified in malpractice claims and 10 of the top 15 associated with the largest number of serious misdiagnosis-related harms. While some conditions (particularly those affecting children) may have been underrepresented (e.g., missed child abuse/non-accidental trauma), we found no evidence to suggest that using a prespecified list based on prior literature, Technical Expert Panel and Key Informant interviews appreciably affected our results. However, because of the constrained focus on the most common conditions, we do not have data on misdiagnosis of less common conditions that may nevertheless be of importance to ED clinicians (non-accidental trauma, necrotizing fasciitis, compartment syndrome, brain tumors, obstructive hydrocephalus, ovarian torsion, post-partum hemorrhage, etc.); this is a limitation. We also do not know whether exclusion of smaller studies (n<50) by design influenced results.

Most studies did not directly address issues surrounding measurement of diagnostic error (e.g., validity, reliability, determination of causes, preventability, or attribution of harms). In clinical practice, many disease reference standards are insufficiently understood, developed, and implemented, so diagnosticians often disagree on final patient diagnoses. To the extent that manual chart reviews were used to identify errors, original studies are likely to suffer from problems of poor chart documentation,^{318, 319} low inter-rater reliability,^{320, 321} and hindsight bias.³²² The problem of author bias in choice of definition or method of measurement (e.g., specialists [or diagnostic error “advocates”] determining ED misdiagnosis and favoring more lax definitions of error/harm, or the reverse, with ED clinicians favoring more stringent definitions) is difficult to ascertain. Our use of the NAM definition of diagnostic error mitigates some of these concerns, since there is less subjectivity inherent in a diagnostic label change (e.g., discharged with “musculoskeletal chest pain” returns with “aortic dissection” within 24 hours) than in the determination of preventability, which is known to be highly subjective.³²⁰ Also, many included studies used stringent measurement protocols or objective statistical methods (e.g., Symptom-disease Pair Analysis of Diagnostic Error [SPADE]¹⁴⁵). Nevertheless, poorly standardized or low-reliability measurements are important limitations.

Review

Neither the study team nor the TEP prospectively identified five conditions that ultimately appeared among the 15 most harmful conditions identified as part of KQ1. As a result, spinal cord compression and injury (#4), lung cancer (#8), traumatic brain injury and traumatic intracranial hemorrhage (#9), gastrointestinal perforation and rupture (#14), and intestinal obstruction (#15) were not included in the original, prespecified disease-specific searches related to KQ2 and KQ3. Thus, rates are not available. It is likely that the causes of missed lung cancer differ somewhat from the studied vascular events and infections. Cognitive errors are likely to have been errors in interpreting radiographs (missed lung nodules) and systems errors are likely to have been errors in communication or handoffs for follow-up of incidental findings.

Considerations for Clinical Practice and Policy

Implications for Operational Quality Measurement and Benchmarking

A recent issue brief from AHRQ outlines the full palette of options for operational measurement of diagnostic errors.³⁷³ Below, based on ED measurement needs derived from our systematic review and meta-analysis, we offer specific suggestions from among the list of possibilities mentioned in that brief. We divide these into methods that are disease-specific and those that are disease-agnostic. We also note measures that are “numerator only” (i.e., they are all “events” and the precise population from which these events are drawn is ill-defined) and briefly summarize the use of each data type considering findings from this report. These are followed by some general recommendations on measurement based on our findings. Because no single measurement method can address all types of diagnostic errors, ED diagnostic errors should be tracked using a portfolio of metrics that include the following:

1. Disease-Specific Data Sources/Metrics for Diagnostic Error. Disease-specific measurement facilitates targeted quality improvement efforts and assessment of their impact. These measures should be used to address symptoms, diseases, or symptom-disease pairs that are either common or frequently misdiagnosed.

   1. SPADE (Symptom-disease Pair Analysis of Diagnostic Error)¹⁴⁵ - We identified multiple studies using SPADE or related methods for missed stroke,^{64, 120, 155} myocardial infarction,^{63, 77, 120} aortic aneurysm/dissection,¹²⁰ sepsis,^{78, 93, 94, 256} and meningitis,⁹³ but it can be applied to any acute disease which confers excess short-term risk of an adverse clinical outcome when left untreated after an initial treat-and-release visit. The look back method (from diseases to symptoms) can be used to discover clinical presentations (often “atypical” ones) at high risk of misdiagnosis, as well as other risk factors for misdiagnosis, such as age, gender, or race. The look forward method (from symptoms to diseases) can be used to measure absolute rates of misdiagnosis-related harms and monitor performance in response to diagnostic improvement initiatives. SPADE is a clinically valid, methodologically sound, statically robust,¹⁵⁴ and operationally viable¹⁵⁵ method of identifying misdiagnosis-related harms from electronic health record or billing/administrative data—importantly, without the requirement of manual chart review (although chart review can inform root cause analysis if so desired). However, SPADE relies on detecting adverse events. From the studies we identified, these are relatively infrequent (typically less than 1 percent of treat-and-release cases), so stable measurement generally requires thousands of encounters. That means that at a medium to large-sized ED, relatively common symptoms (e.g., abdominal pain, chest pain, dizziness, headache, back pain) can be mined using SPADE for misdiagnosis-related harms linked to more common dangerous diseases such as stroke, myocardial infarction, sepsis, or pneumonia using a rolling 6- to 12-month window. Smaller hospitals or rarer diseases generally require longer assessment time windows. Also, related symptoms^{77, 256} or diseases¹²⁰ can be aggregated to increase the sample size. If insufficient data are available for stable measures, SPADE can be used as an electronic trigger mechanism to identify cases for manual chart review.
   2. Change from ED admitting diagnosis to final hospital discharge diagnosis - We found many studies that measured false positives among patients admitted to the hospital for a specific target disease. For example, this included studies of the rate at which admissions to a medical unit with suspected myocardial infarction turned out to be incorrect or the rate at which the cardiac catheterization lab consulting service was activated unnecessarily. These studies tended to focus on overutilization of clinical services or hospital admission. This method is likely to be more helpful in assessing overall diagnostic accuracy for a given disease if paired with a search for false negatives, at least among admitted patients (e.g., admission for “fall” that turns out to be a missed myocardial infarction). This involves a search for when the ED admitting diagnosis differs from the final hospital discharge diagnosis for a given dangerous disease, as done retrospectively for myocardial infarction using Medicare data²⁰⁶ and in robust prospective fashion by Hautz et al., 2019 across medical conditions.⁷ Even more robust would be to combine this with 30-day disease-specific hospitalizations after ED treat-and-release for a more complete capture of false negatives, although we did not identify any disease-specific studies that combined these sorts of data.
   3. Unannounced standardized patients³⁷⁴ (“secret shoppers”) - Although no studies of this type were identified in our review, standardized “fake” patients can be used to assess diagnostic quality for specific symptoms or diseases in clinical practice.³⁷⁵ This approach decreases the variance in measurement, allowing very direct comparisons, down to the individual ED clinician level. However, the effort and expense required makes this an option that should be reserved for very high-stakes scenarios (e.g., pay-for-performance benchmarking for diagnosis of a specific clinical presentation).
2. Disease-Agnostic Data Sources/Metrics for Diagnostic Error. Disease-agnostic measurement facilitates inquiry into overall error and harm trends but is less actionable. These measures should be used to track the impact of broad interventions likely to affect the overall diagnostic error rate (e.g., change in staffing model or access to specialists) and, when possible, as a general benchmarking tool to compare across institutions.

   1. Malpractice claims (numerator only) - At most institutions, ED claims are readily captured and thoroughly analyzed. Based on this review, claims should be presumed to be both biased towards dangerous diseases and to substantially underrepresent total errors, but also to be mostly representative of diagnostic errors resulting in serious harms (barring perhaps overrepresentation of heart attacks and radiographically determined misdiagnoses). Tracking changes in the frequency or severity of claims in response to diagnostic improvement interventions may work for more common conditions in claims (e.g., stroke) or using long-term averages over time for less common ones, but the latter may be impacted by other secular trends.
   2. Incident reports (numerator only) - These are most useful if there is a structured mechanism for identifying the incident as a diagnostic error and concerted efforts are made to encourage reporting by clinicians.^{100, 339, 376} This includes physicians (who rarely report but are best positioned to report on diagnostic issues),³⁷⁶ as well as other team members such as nurses (who routinely report but do not routinely view diagnostic errors as within the scope of their reporting duties^{100, 339, 376}). Their value is principally in identifying unexpected errors or latent risks. Incident reports can be combined with similar data (e.g., patient complaints, autopsy, morbidity and mortality rounds cases^{377, 378}). Incident reports can be enhanced and made more informative via the use of common formats that permit aggregation of data at the local, regional, or national levels.³⁷⁹ The AHRQ Common Formats for Event Reporting (CFER) now include a special common format for Diagnostic Safety event reporting (CFER-DS) that has recently been developed for use by patient safety organizations (PSOs).^{379, 380} The CFER-DS (and all of the other AHRQ Common Formats) are available in the public domain to encourage their widespread adoption. An entity does not need to be listed as a PSO or working with one to use the Common Formats. However, it should also be noted that the Federal privilege and confidentiality protections only apply to information developed as patient safety work product by providers and federally listed PSOs working under the Patient Safety and Quality Improvement Act of 2005.
   3. Electronic triggers for ED treat-and-release visits (unplanned revisits or outcomes) - Electronic triggers represent an important mechanism for identifying potential diagnostic adverse events that then trigger manual chart review. After case review and confirmation as diagnostic errors, the rate can be tracked over time for any changes. Typical triggers include short-term revisits, hospitalizations, or adverse patient outcomes (e.g., non-hospice death), if available. Based on our review, 72 hours provides an enriched sample but will substantially underestimate totals. The range of reasonable time frames is estimated at 7 to 30 days, but it appears 14 days is sufficient to capture the majority of adverse events following diagnostic errors. Ideal ascertainment time windows are likely disease-specific in relation to natural history.
   4. Electronic triggers for ED admissions (unplanned escalation in care or change in treating service) - Typical triggers include intensive care unit transfers after routine (ward) admission, transfer of admitting service (e.g., from medicine to neurology), or adverse patient outcomes (e.g., in-hospital death). Because our review found that diagnostic errors are enriched among admitted patients and associated with both hospital mortality and increased length of stay, it would not be unreasonable to screen any chart with a change in diagnosis from ED admitting diagnosis to hospital discharge for diagnostic errors, as in Hautz et al., 2019.⁷
   5. Routine or sampled follow-up outreach to patients (e.g., Leveraging Patient’s Experience to improve Diagnosis [LEAPED]⁸) - Although methods for determining diagnostic error using routine or sampled patient follow-up contact are still being optimized, feasibility has recently been established.^{8, 381} Our review identified very few studies assessing failures in communicating diagnoses to patients (which are also defined as diagnostic errors by the NAM). Direct patient outreach post-visit is likely the only method by which diagnostic errors due to communication failures with patients can be ascertained. If follow-up is obtained very early (e.g., less than 72 hours), communication failures will predominate. If follow-up is obtained later (e.g., 30 days), it is likely that both communication failures and diagnostic accuracy can be captured. Such later phone calls are likely to serve as an important source of information regarding diagnostic errors with less severe consequences, including temporary harms, which our review found were poorly captured by existing methods.
3. General Recommendations for Measuring Diagnostic Error. Below are general insights about measurement derived from our systematic review of the literature.

   1. False negatives and false positives - It would be optimal to measure all four aspects of diagnostic accuracy (true positives, true negatives, false positives, false negatives). This permits calculation of all accuracy statistics – sensitivity (false negative rate), specificity (false positive rate), negative predictive value (false omission rate), positive predictive value (false discovery rate), and total diagnostic accuracy (total diagnostic error). However, doing so requires combining multiple types of data, and we found no studies that did this. This can be done more easily for a single disease than for all diseases simultaneously.
   2. Balancing measures – Diagnostic process improvements (e.g., through use of new or different test batteries, structured clinical pathways, or teamwork in diagnosis) that increase total diagnostic accuracy will generally lead to reductions in both false negatives and false positives.³³⁷ However, one potential ED clinician response to concerns over (false negative) diagnostic errors is to simply “do more of the same” by changing their personal threshold for ordering diagnostic tests; this tends to produce diagnostic test overuse and excessive hospital admissions, rather than more accurate diagnosis.³³⁷ All diagnostic error-related measures should be accompanied by balancing measures that address rates of diagnostic test utilization and hospitalization.
   3. Outcome ascertainment - Optimal outcome ascertainment involves prospective data collection, as seen in the few very high-quality studies of diagnostic error on which our overall error and harm estimates are most heavily based. Ideally, this would be built into the process of routine care—systematically recording presenting symptoms, admitting diagnoses, discharge diagnoses, plus follow-up events and outcomes. With modern electronic health records, EDs can generally secure all but the last of these data points. Systematic ascertainment of outcomes is a crucial addition. All errors requiring chart review should be analyzed using diagnosis-specific root cause analysis procedures (e.g., specialized diagnostic error fishbone diagram³⁸²).
   4. Pitfalls in measurement - Based on the review, we identified several pitfalls in measurement that some studies failed to address, leading to heterogeneity, apparently conflicting results, and, in some cases, false conclusions.

      1. Finding discrepant diagnoses versus errant processes:⁶ The literature is admixed with studies that examine diagnostic errors very differently. A key aspect is whether studies require only an incorrect diagnosis label (or even a communication failure despite the correct label, as in the NAM’s definition), mandate some identifiable diagnostic process failure, require preventability, or only consider it an error when outcomes were judged to have been impacted. As expected, the highest frequency of errors will be measured when a label failure is all that is required and the lowest when resulting harms must have been judged by clinicians to have been preventable. However, as demonstrated nicely by Hautz, 2019, even a label failure (regardless of process) is associated with 2.4-fold increased mortality and 3.4-day increased hospital length of stay.⁷ Thus, even without ascertaining diagnostic process failures, label failures alone portend worse outcomes. This suggests that identifying label failures (which is easier and has greater inter-rater reliability than identifying process failures⁷) is preferable as a starting point for measuring errors from a quality improvement standpoint. It also suggests that studies or results should not be directly compared when different definitions are used. We recommend using the NAM definitions, which reflect all diagnosis label failures as errors, without regard to process.⁵
      2. Counting errors versus harms: When assessing post-treat-and-release ED returns or hospitalizations, it is incorrect to label these “diagnostic errors” because they are actually misdiagnosis-related harms. The severity of harms may be judged minor (e.g., temporary inconvenience and loss of confidence in the healthcare system³⁸³), but they still represent harms. Furthermore, many patients who suffer diagnostic errors “get lucky” temporarily (i.e., suffer no short-term consequences of the diagnostic error), but these patients are nevertheless at risk of delayed harms from lack of secondary prevention. For example, mislabeling a transient ischemic attack as “benign positional vertigo” may prevent the patient from getting secondary stroke prophylaxis. Untreated, 10 to 20 percent of such patients will suffer a major stroke in the subsequent 90 days,^{113, 258, 384} but even if the patient is one of the fortunate ones who do not have a major stroke in that time frame, the diagnostic error may nevertheless prevent the patient from being recognized as needing long-term stroke prophylaxis or risk factor modification.
      3. Tracking hospital crossovers: Not all patients return to the same hospital (or even health system) when they develop new or worsening symptoms after having been treated and released from an initial ED. This means there is systematic under-ascertainment of diagnostic adverse events (e.g., subsequent hospitalizations) when out-of-network crossovers are not considered. One estimate using a regional health information exchange found that 25 percent of patients who visit the ED more than once will cross over to another hospital or health system.³⁸⁵ When patients are misdiagnosed, they may be more likely to return to a different ED than if they were correctly diagnosed.¹⁴⁵ One study included in our review found a 37 percent crossover rate.¹⁹⁵ The importance of this for measurement is that hospitals should recognize that their true misdiagnosis-related harm rate could be more than 1.5-fold higher than measured using intra-hospital data. The implication for national benchmarking, payment incentives, and other high-stakes accountability initiatives is that data sources that capture out-of-network follow-ups are critical to ensure comparability of case ascertainment. Potential data sources include (a) insurance-based billing data such as Medicare, (b) linkable state-level data such as AHRQ’s State Emergency Department Databases (SEDD)³⁸⁶ and State Inpatient Databases (SID),³⁸⁷ or (c) regional health information exchanges such as Maryland’s Chesapeake Regional Information System for our Patients (CRISP).³⁸⁸
      4. Including morbidity in addition to mortality: From this review, we estimate that 29 to 41 percent of the serious misdiagnosis-related harms from ED diagnostic error are permanently disabling, rather than lethal. This means that mortality statistics alone will understate the total by 1.4- to 1.7-fold and diseases that confer a high rate of morbidity relative to mortality will be underrepresented in summaries of mortality. In particular, this includes neurologic diseases with a high proportion of serious harms that are morbid but not mortal—spinal abscess (82%), stroke (71%), and meningitis (48%). It is expected that untreated neurologic disorders tend to produce more permanent disability than death (and this likely includes two other top 15 neurologic conditions associated with serious misdiagnosis-related harms for which we were unable to ascertain the breakdown of morbidity versus mortality (i.e., spinal cord compression and injury; traumatic brain injury and traumatic intracranial hemorrhage)). Given that the organ system most often involved in diagnostic errors leading to serious harms is the nervous system (34%, Table 4), mortality alone will be a particularly poor health outcome proxy and will tend to substantially understate both individual diseases and total harms.
      5. Controlling for initial severity (“misdiagnosis is protective paradox”): Illness severity is often a confounder of the relationship between diagnostic error and health outcomes for patients (i.e., is causally linked to both the risk of misdiagnosis and the risk of a bad health outcome). An observational study that directly compares a population of all correctly diagnosed and all incorrectly diagnosed patients will generally find that initial case severity is higher among the correctly diagnosed population, skewing health outcomes for these patients in an unfavorable direction. This effect will tend to nullify the unadjusted, measured impact of diagnostic error or even reverse it (“misdiagnosis is protective” paradox).¹ We found a number of studies which failed to control for initial case severity and, as a result, drew erroneous inferences about lack of impact of diagnostic error on patient outcomes. No measures of this type should be considered valid unless appropriate statistical controls (e.g., matching or adjustment) are used to account for initial case severity or its proxies.
      6. Addressing preventability of harms: There is moderate inter-rater variability in clinician ratings of preventability.³²⁰ This issue is more complex for diagnostic errors than treatment errors because there is dual uncertainty—first, whether the diagnostic errors themselves can be prevented and, second, whether treatments for the diagnosed underlying diseases would prevent any associated untoward outcomes. Although combined diagnosis-treatment studies are recommended by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) group as ideal, this two-step link from diagnosis to health outcome is rarely assessed when diagnostic interventions are put to the test.^{389, 390} The strongest evidence of preventability of harms will come from prospective (preferably randomized) studies that measure the health outcomes of interventions to improve diagnosis and that demonstrate both greater diagnostic accuracy and a link between that greater accuracy and improved patient outcomes. Absent this level of rigor, measurements of inter-institutional variability (adjusted for likely confounders) may be a good proxy for preventability, with lower-performing institutions striving to match outcomes from higher-performing institutions.
      7. Differences in causal inferences based on different denominators: In the same way that inferences about error rates may differ dramatically depending on the denominator used (e.g., false negative rate [denominator all with disease] versus false omission rate [denominator all at-risk patients]), the same is true for causal inferences. For example, much of the apparent heterogeneity in our KQ3 results for demographic predictors likely stems from confusion about the inferences to be drawn from different study designs. For example, one study showed that being a woman or a minority is a risk factor for misdiagnosis of heart attack when looking back from heart attack admissions to antecedent treat-and-release ED visits, but not when looking forward from chest pain discharges to subsequent heart attack hospitalizations.⁷⁷ These results seem conflicting, but they are not. The reason for this difference is as follows. The look back method normalizes overall risk for heart attacks by starting with heart attack hospitalizations as the denominator; this, in turn, allows investigators to assess the impact of gender or race on the likelihood of misdiagnosis, given equal baseline risk of the underlying disease. However, the look forward method uses chest pain discharges as the denominator; here the distribution of heart attacks is uneven by gender and race, with the largest number of heart attacks being among white men. Since the impact of disease prevalence is greater than the impact of misdiagnosis risk, the result is that white men are more likely to return having had their heart attack initially missed. Thus, the look back method speaks to the relative risk or odds of a misdiagnosis conferred on a patient based solely on their gender or race, while the look forward method estimates the absolute risk of a misdiagnosis based on a mix of disease and misdiagnosis prevalence. Because ED clinicians are likely to calibrate their decision-making to baseline disease prevalence, this may contribute to some proportion of the demographic disparities seen in diagnosis. Since that proportion is unknown, additional research should be done to assess the impact of prevalence-based reasoning on demographic disparities in diagnosis.
4. Approaches to Measurement at the Institutional Level. No single measurement method or individual measure will suffice. A “portfolio” approach is needed. A one-size-fits-all approach is unlikely to be equally appropriate for all institutions. Offered below are a few different ways that an institution might choose to approach measuring diagnostic errors.

   1. Tailored-risk portfolio: An institution with limited measurement resources and a need to convince institutional leadership of the return on investment for measuring diagnostic errors might take a tailored-risk approach. This could begin with numerator-only measures (e.g., malpractice claims or incident reports) to identify specific symptoms or diseases which have been a known source of institutional risk. These could then spark a disease-specific approach to measurement such as SPADE for those clinical presentations, with balancing measures related to false positives and resource utilization (e.g., test frequency, hospital admission rates, false discovery rate). After addressing one or more diseases and showing improvement, the entire process could be repeated to identify current risks and then address new conditions.
   2. Top-harms portfolio: An institution with intermediate measurement resources and institutional recognition of the importance of diagnostic error might develop a local dashboard for the top conditions generally causing the greatest misdiagnosis-related harms from “undercalls” (i.e., false negatives with dangerous diseases)—top five most harmful vascular events (stroke, myocardial infarction, aortic aneurysm and dissection, venous thromboembolism, arterial thromboembolism) plus top five most harmful infections (meningitis/encephalitis, sepsis, spinal/intracranial abscess, pneumonia, necrotizing fasciitis). They could (i) use SPADE look-back metrics to identify high-risk clinical presentations, (ii) design interventions to address the most pressing of these, and (iii) measure impact of these interventions using SPADE look-forward metrics. They could use balancing measures related to false positives and resource utilization (e.g., test frequency, hospital admissions, false discovery rate) for each of these high-harm diseases to address “overcalls” (i.e., false positive diagnoses of dangerous diseases or inappropriate resource use in pursuit of those diseases).
   3. Comprehensive portfolio: An institution with more substantial measurement resources and leadership support to pursue institutional diagnostic excellence might combine the tailored-risk and top-harms portfolio approaches (described above) with systematic sampling of patient feedback (e.g., using LEAPED⁸) and systematic use of disease-agnostic e-triggers to identify (i) 7-day hospital admissions after ED treat-and-release visit to identify additional high-risk diseases or clinical presentations and (ii) unplanned escalation in care or change in treating service for admitted patients. Taken together, this comprehensive approach would address almost all potential opportunities to improve diagnostic performance in pursuit of diagnostic excellence.
5. High-Stakes Measurement for Accountability, Payments, and National Benchmarking. Based on the results of this review, high-stakes, cross-institutional comparisons require greater standardization and efficiency than can be achieved using most of the available data sources and methods listed above.

   1. Data source (likely Medicare or HCUP databases): While integrated health plans (e.g., Kaiser Permanente, Intermountain Healthcare, and Geisinger Health System) and the Veterans Administration have electronic medical record data sources that are fairly complete and comparable within their respective systems, there are no such data sets for all hospitals nationally. Currently, the most promising data for high-stakes measurement in the United States are from Medicare beneficiaries, since Medicare billing data are gathered in fairly consistent fashion, from a relatively unbiased sample of older patients, at almost all U.S. hospital EDs. They are unconstrained by health system crossovers or geographic boundaries, and they incorporate death data. They do not, however, represent children, so cannot be used to assess pediatric diagnostic error. They also represent only a subset of ED cases (roughly 24 percent in 2018³⁹¹), which means the sample size for some hospital-level analyses will need to sacrifice temporal resolution for smaller hospitals. It is possible that with greater state-level engagement in maintaining linkable ED visit (SEDD) and hospitalization (SID) patient databases that these two obstacles can be overcome, and the preferred data source would then likely become the AHRQ family of HCUP databases (though integration with the national death index for out-of-hospital mortality would be an important addition to increase capture of important outcomes). Both data sources would benefit by the addition of ongoing health-related quality of life (HRQoL) metrics, but implementation of this could prove cumbersome. An alternative would be for the Centers for Disease Control and Prevention to adapt the National Hospital Ambulatory Medical Care Survey (NHAMCS) to include short-term patient follow-up from their nationally representative sample of ED visits.
   2. Measurement method (likely SPADE): We found no methods of measurement other than SPADE using a statistically robust approach to measuring diagnostic error without the reliability challenges and high costs faced by triggered manual chart review or routine patient follow-up assessment (e.g., phone calls at 30 days). This appears to be the most promising method currently available for achieving valid, high-stakes measurement that can easily incorporate case mix severity adjustments or propensity score case matching.¹⁶³ Missed cancer (including lung cancer) may require alternative monitoring methods, since the temporal risk profile of adverse events after a lung cancer misdiagnosis are very different than those after a missed vascular event or infection, making it less readily amenable to current SPADE methods.
   3. Disease metrics (Top 10+ for Serious Misdiagnosis-Related Harms): A reasonable place to start for national ED quality measurement would be to create metrics for the top five most harmful vascular events (stroke, myocardial infarction, aortic aneurysm and dissection, venous thromboembolism, arterial thromboembolism) and top five most harmful infections (meningitis/encephalitis, sepsis, spinal/intracranial abscess, pneumonia, necrotizing fasciitis). Diseases most appropriate to pediatric misdiagnosis, such as appendicitis and testicular torsion, could be added if the data source were changed to one that was not age restricted (i.e., if it were not Medicare data). Standardized ICD code sets for each disease could be derived from the Elixhauser system used by AHRQ in its Clinical Classifications Software,¹⁰¹ with appropriate modification to match the diseases in question (as done recently by Newman-Toker, et al.¹⁷). Ideally these measures would be endorsed by Emergency Medicine specialty societies (e.g., American College of Emergency Physicians, Society for Academic Emergency Medicine) and national quality and safety organizations (e.g., National Quality Forum, The Joint Commission).
   4. National Diagnostic Performance Dashboard: AHRQ, other government bodies (e.g., Centers for Disease Control and Prevention’s National Center for Health Statistics³²⁴), or non-governmental organizations could monitor the overall epidemiology and variability of diagnostic performance (specifically, diagnostic outcomes, which can be adjusted for case mix severity) across the nation (analogous to the Dartmouth Atlas Project for utilization of healthcare services³⁴⁷). For the 10+ diseases noted above, disease-specific metrics could be combined into a National Diagnostic Performance Dashboard. For simplicity, this might initially use only a look-back approach and ignore specific symptoms (as done recently by Waxman, et al¹²⁰). Later, for greater precision and monitoring of diagnostic quality and safety performance, ICD symptom code sets could be added for the most common ED symptoms. This would allow realization of the full potential of SPADE analysis, using both look-back (identifying high-risk presentations and disparities in diagnosis) and look-forward (measuring absolute harm rates and monitoring impact of solutions) approaches, which have been shown to vary substantially by hospital (e.g., for acute myocardial infarction, where misdiagnosis-related adverse event rates varied 3.3-fold from 0.6% to 1.9% across individual EDs, P < 0.001⁷⁷) and permit observed minus expected analysis to detect statistically valid excess adverse events above the base rate.²⁵⁶ The purpose of a national monitoring mechanism would be multiple: (i) providing a benchmarking tool for individual institutional ED performance; (ii) monitoring national diagnostic quality and safety (e.g., temporal trends and health disparities) to help guide policy decisions; (iii) assessing the impact of major policy interventions (e.g., payment reforms that incentivize better diagnostic performance).