1.1. Background

Healthcare algorithms are frequently used to guide clinical decision making both at the point of care and as part of resource allocation and healthcare management. For this review, algorithms are defined as mathematical formulas or models that combine different input variables or factors to inform a calculation or an estimate, such as an estimate of disease or risk of a particular health outcome. Algorithms are often incorporated into healthcare decision tools, such as clinical guidelines, pathways, clinical decision support programs in electronic health records (ERs), and operational systems used by health systems and payers; our use of “algorithm” includes algorithm-informed tools. End-users, such as clinicians, integrated delivery networks, payers, and consumers, use algorithms for at least six broad purposes: screening, risk prediction, diagnosis, prognosis, treatment planning, and resource allocation. While algorithms have long been derived from traditional statistical techniques, such as regression analysis, their use in predictive analyses is increasingly fueled by artificial intelligence techniques, including machine learning.

Algorithms commonly include clinical and sociodemographic input variables and measures of healthcare utilization. Race and ethnicity are often used as input variables;^1–3 however, because race and ethnicity are not biological concepts but socially constructed and represent a variety of other factors, their use in algorithms that influence clinical decision making can have a wide range of effects on patient outcomes. Some effects could be desirable (e.g., improved allocation of resources or access to care), and some could be harmful (e.g., exacerbation or perpetuation of health and healthcare disparities).^4–6 Many effects are unknown.

In a seminal review published in 2020, Vyas et al.¹ examined race-based algorithms commonly used in eight clinical specialties. The review observed that while use of race and ethnicity as an input variable was often driven by primary studies that noted a difference in health between racial and ethnic groups, little, if any, actual evaluation has measured potential race-based harms of using such algorithms. The authors noted that including race and ethnicity might direct more resources toward White patients and thus exacerbate health and healthcare inequities.

Algorithm developers often include race and ethnicity as input variables, intending to increase diagnostic or predictive accuracy by capturing genetic predispositions due to racial and ethnic differences that affect clinical outcomes. However, race and ethnicity are poor proxies for genetic predisposition. Greater genetic variation typically exists within groups classified as the same race or ethnicity than between them.^7–9 Numerous purported genetic predisposing differences between races and ethnicities regarding muscle mass, pain sensitivity, lung function, and similar biomarkers have been debunked.¹⁰ Mounting research details nonbiological root causes of biological phenomena (e.g., epigenetics) and observed differences in health between racial and ethnic groups. Specifically, chronic exposure to interpersonal discrimination, coupled with structural racism or biases intrinsic to societal systems, create unearned disadvantage or advantage depending on one’s identity. This leads to unequal opportunities for health and wellbeing through disparities in social determinants of health (SDOH).^11–14

While individual self-identification of race and ethnicity is considered the preferred method for defining and collecting these data, the sensitivity and specificity of this method depends on the response categories presented, in particular for individuals who may identify with more than one racial and ethnic group (e.g., Hispanic and Asian populations).^15,16 This further highlights challenges with operationalizing U.S.-centered, socially constructed racial and ethnic categories, such as those used by the federal Office of Management and Budget (OMB). In response to the challenges with OMB categories, the National Academy of Medicine in 2009 issued standards for optimal collection of Race, Ethnicity, and Language (REaL). REaL standards improved on existing OMB race categories by incorporating Hispanic ethnicity as a racial category and recommended capture of granular ethnicity to better approximate ancestry or country of origin.^16–18 However, implementation of REaL standards by health systems, researchers, and the broader medical community is variable. For this report, we use race and ethnicity to represent the socially constructed OMB categories of race and Hispanic ethnicity, with a multiple-choice option as distinct from granular ethnicity in accord with the 2009 standards. Unless otherwise specified, we follow the convention of capitalizing all racial and ethnic categories.

Algorithm developers often justify including race and ethnicity input variables by citing observational studies or post hoc analyses of randomized controlled trial data that demonstrated differences in outcomes among different race and ethnicity subgroups. These studies may be small and unrepresentative, serve to reinforce misconceptions, or assume that race and ethnicity are fundamental causes, even though other factors may be causative, confounding, or modifying the effects of race and ethnicity.^19,20 A robust example examines a “race-correction” coefficient that raises the threshold of concern or action for a given estimated glomerular filtration rate (eGFR), a key biomarker in examining kidney function and diagnosing and treating chronic kidney disease, only for Black patients. Recent studies have modeled the effect of removing the race-based coefficient from eGFR and concluded that Black patients would be more likely to receive more timely referrals for kidney transplants compared with race-based eGFR calculations.^21–23 However, controversy around this issue remained, as the evidence base lacks prospective trials comparing differing approaches to assessing kidney function and subsequent need for treatments, including transplant.^24–26 Accordingly, the National Kidney Foundation and the American Society of Nephrology convened a task force to address this topic. In September 2021, the task force released its final report recommending against use of the race coefficient, and supporting the use of a race-independent biomarker, cystatin C, to confirm eGFR.²⁷

Algorithm input variables other than race and ethnicity may also perpetuate, contribute to, and/or exacerbate health disparities and inequities. For example, an algorithm used to allocate access to disease management support programs included prior healthcare costs as an input variable to serve as a proxy for clinical needs and subsequent healthcare utilization.⁵ This algorithm led to a disproportionate enrollment of White patients with less severe disease into a chronic disease management program compared with Black patients with greater disease severity. These stark racial and ethnic disparities were a consequence of selecting a proxy for disease severity and healthcare needs, given that healthcare expenditure is higher, on average, for White patients than for Black patients with the same conditions reflecting barriers to accessing care. Replacing healthcare costs with a better indicator for disease severity corrected this race and ethnicity disparity for access to an indicated chronic disease management program. Therefore, due to structural racial and ethnic biases and other forms of racism in healthcare, algorithms that do not include race and ethnicity input variables may nonetheless contribute to or perpetuate healthcare and health disparities.

Evidence gaps regarding the impact of many algorithms that include race and ethnicity input variables on potential racial and ethnic disparities in healthcare delivery remain, with few studies comparing the effects of alternative algorithm strategies on important health outcomes. Nevertheless, several academic societies have issued position statements or guidelines supporting removal of race and ethnicity input variables in their algorithms. Notable examples include recommendations to remove race from pediatric urinary tract infection treatment guidelines^10,19 and to remove race considerations in hypertension management guidelines.^28–31 Moreover, little is currently known about how algorithms that do not explicitly include race and ethnicity input variables may affect racial and ethnic health and healthcare disparities.

1.2. Purpose and Scope of the Review

In September 2020, the Agency for Healthcare Research and Quality (AHRQ) received a request from Congress to review the evidence on the use of race and ethnicity in clinical algorithms and the potential of algorithms to contribute to disparities in healthcare, and commissioned this evidence review. AHRQ also issued a public Request for Information³² that generated responses related to algorithms from 42 organizations, agencies, and individuals.³³ This evidence review is intended to:

• Examine how algorithms, with or without race and ethnicity as input variables, affect racial and ethnic differences in access to care, quality of care, and health outcomes.
• Evaluate strategies to mitigate any racial and ethnic bias in the development and use of algorithms.
• Explore contextual factors, including the role of algorithm developers and end-users; identify available or emerging guidance on preventing racial and ethnic bias during algorithm development; clarify stakeholder awareness of and perspectives on potentially racially and ethnically biased algorithms; and determine incentives and barriers affecting use and evaluation of algorithms.

With guidance from AHRQ and in collaboration with Subject Matter Experts (SMEs), Key Informants (KIs), and Technical Expert Panel (TEP) members, we developed two Key Questions (KQs) and four Contextual Questions (CQs). KQ 1 assesses the effects of algorithms on racial and ethnic disparities in health and healthcare. An algorithm might create, perpetuate, exacerbate, reduce, or have no effect on health and healthcare disparities. We excluded studies of algorithms that did not examine their effect on disparities. KQ 2 focuses on strategies to mitigate algorithmic bias. Focal points for mitigation could be on datasets used to develop or train algorithms, input variables included in algorithms, or processes for validating, implementing, disseminating, or adapting algorithms. KQ 2 includes studies of algorithms that were redesigned to mitigate algorithmic bias in response to prior associations between these algorithms and racial and ethnic health or healthcare disparities. We aimed to identify and describe strategies to address potential racial and ethnic algorithmic bias and evaluate the effect on racial and ethnic health and healthcare disparities. To address KQs 1 and 2, we conducted a systematic literature search.

The CQs were designed to explore the context and capture insights on practical aspects of these issues. CQ 1 examines the problem’s scope within healthcare. CQ 2 addresses recently emerging standards and guidance, from within and outside healthcare, on how racial and ethnic bias can be prevented or mitigated when healthcare and non-healthcare algorithms are developed and deployed. CQ 3 explores key stakeholders’ knowledge and perspectives about the interaction of algorithms and racial and ethnic disparities in health and healthcare. CQ 4 conducted an in-depth analysis of a sample of six healthcare algorithms not previously widely evaluated in published literature to understand how their design and implementation might contribute to racial and ethnic health and healthcare disparities.

To address CQs 1, 2, and 3, we searched for studies, standards, frameworks, white papers, and other relevant resources and sought input from our SMEs, KIs, and TEP members. As a supplement to KQs 1 and 2, a separate conceptual model was developed for CQ 4, and an objective four-step, a priori approach was used to identify the sample of algorithms currently in use whose effects on racial and ethnic health and healthcare disparities were not previously studied.

The conceptual model for CQ 4 was motivated by the fact that although the selected algorithms are being used to make clinical decisions, little is known about their development, stakeholders involved in development, validation, performance testing, translation and implementation into clinical practice, and process for updating. We therefore conducted a deeper analysis to understand these considerations, using a representative sample of algorithms that were likely to affect large populations. We focused on algorithms not identified in KQs 1 and 2 to contextualize the development, validation, and impact of algorithms. This evaluation is complementary to findings from the KQs and may provide a fuller understanding of the opportunities and challenges faced by policymakers and healthcare providers for more just and equitable care with the use of algorithms.

Findings from this review are intended to inform: (1) policymakers, providers, payers, health systems, and patients seeking to understand or address the role of algorithms in racial and ethnic health and healthcare disparities; (2) future research opportunities exploring the effects of algorithms on racial and ethnic health and healthcare disparities; and (3) current and emerging guidance and best practices for developing, validating, implementing, and evaluating algorithms to mitigate potential racial and ethnic bias.

For this report, we define commonly used terms in Table 1.

Table 1

Definition of terms and report-specific considerations.

1.3. Key Questions

Key Question 1. What is the effect of healthcare algorithms on racial and ethnic differences in access to care, quality of care, and health outcomes?

Key Question 2. What is the effect of interventions, models of interventions, or other approaches to mitigate racial and ethnic bias in the development, validation, dissemination, and implementation of healthcare algorithms?

1. Datasets: What is the effect of interventions, models of interventions, or approaches to mitigate racial and ethnic bias in datasets used for development and validation of algorithms?
2. Algorithms: What is the effect of interventions, models of interventions, or approaches to mitigate racial and ethnic bias produced by algorithms or their dissemination and implementation?

1.4. Contextual Questions

Contextual Question 1. How widespread is the inclusion of input variables based on race and ethnicity in healthcare algorithms?

1. What types of algorithms used in healthcare include input variables based on race and ethnicity? How widely are they used?
2. Who develops algorithms used in healthcare that might include input variables based on race and ethnicity?
3. Who are the end-users of these algorithms used in healthcare? What incentives and barriers are there to implementation or de-implementation?
4. What patient populations are included?
5. What clinical conditions, processes of care, and healthcare settings are included?

Contextual Question 2. What are existing and emerging national or international standards or guidance for how algorithms should be developed, validated, implemented, and updated to avoid introducing bias that could lead to health and healthcare disparities?

1. Within these standards or guidance, what are the recommendations about the use of input variables or datasets that include race and ethnicity to develop or validate algorithms?
2. What are the recommendations about input variables used or sought in place of race and ethnicity (e.g., genetic markers and biomarkers, social determinants of health, the experience of individual and structural racism), including standards or guidance for how to define and collect data on these variables, and their impact on exacerbating or mitigating bias?
3. What are the recommendations for identifying and addressing other types of input variables that could introduce bias leading to disparities, such as measures of healthcare use or SDOH?
4. What are the recommendations regarding transparency or disclosure of information related to algorithm development, validation, use, and outcomes?

Contextual Question 3. To what extent are patients, providers (e.g., clinicians, hospitals, health systems), payers (e.g., insurers, employers), and policymakers (e.g., healthcare and insurance regulators, State Medicaid directors) aware of the inclusion of input variables based on race and ethnicity in healthcare algorithms?

1. Is there evidence of how these types of algorithms might contribute to biases in provider and payer perceptions of affected populations and their clinical care?

Contextual Question 4. Select a sample of approximately 5–10 healthcare algorithms that have the potential to impact racial and ethnic disparities in access to care, quality of care, or health outcomes and are not included in KQs 1 or 2. For each algorithm, describe the type of algorithm, its purpose (e.g., screening, risk prediction, diagnosis), its developer and intended endusers, affected patient population, clinical condition or process of care, healthcare setting, and information on outcomes, if available. This question’s intent is to consider the use of healthcare algorithms that may be perpetuating racial and ethnic bias but have not been previously linked to disparities in health or healthcare.

1. If race and ethnicity is included as an input variable, how is it defined? Are definitions consistent with available standards, guidance, or important considerations identified in CQ 2?
2. For healthcare algorithms that include other input variables in place of or associated with race and ethnicity, how were these other variables defined? Are these definitions consistent with available standards, guidance, or important considerations as identified in CQ 2? Were racial and ethnic variables considered during initial development or validation?
3. For each healthcare algorithm, what methods were used for development and validation? What evidence, evidence quality, data sources, and study populations were used for development and validation?
4. Are development and validation methods consistent with available standards, guidance, and strategies to mitigate bias and reduce the potential of healthcare algorithms to contribute to health disparities?
5. What approaches and practices are there to implement, adapt, or update each healthcare algorithm?

1.5. Organization of This Report

In the following Methods section, we describe in detail the methods used to address the KQs and CQs. In the Results section, we first provide results of the literature searches for KQ 1 and KQ 2. This includes descriptions of eligible research studies, key points, and syntheses of findings. Results for CQs 1 through 4 follow KQ results. The Discussion section reviews key findings, examines general applicability of the findings, identifies evidence gaps, and describes strengths and limitations of the evidence review and evidence base. The report’s main body is followed by six appendixes: Appendix A. Methods for Search Strategy; Appendix B. List of Excluded Studies; Appendix C. Characteristics of Key Question 1 and 2 Included Studies; Appendix D. Key Question 1 and 2 Evidence Tables; Appendix E. Contextual Question 4 Detailed Supplement; Appendix F. Appendix References.