Levels of Evidence for Adult and Pediatric Cancer Treatment Studies (PDQ®)

This PDQ cancer information summary for health professionals provides comprehensive, peer-reviewed, evidence-based information about the formal ranking system used by the PDQ Editorial Boards to assess evidence supporting the use of specific interventions or approaches. It is intended as a resource to inform and assist clinicians in the care of their patients. It does not provide formal guidelines or recommendations for making health care decisions.

This summary is reviewed regularly and updated as necessary by the PDQ Adult Treatment Editorial Board, which is editorially independent of the National Cancer Institute (NCI). The summary reflects an independent review of the literature and does not represent a policy statement of NCI or the National Institutes of Health (NIH).

Introduction

A variety of end points may be measured and reported from clinical studies in oncology. These may include total mortality (or survival from the initiation of therapy), cause-specific mortality, quality of life, or indirect surrogates such as event-free survival, disease-free survival, progression-free survival, or tumor response rate. End points may also be determined within study designs of varying strength, ranging from the gold standard—the randomized, double-blinded, controlled clinical trial—to case series experiences from nonconsecutive patients.

The PDQ editorial boards use a formal ranking system of levels of evidence to help the reader judge the strength of evidence linked to the reported results of a therapeutic strategy. For any given therapy, results can be ranked on each of the following two scales: (1) strength of the study design and (2) strength of the study end points. Together, the two rankings inform the overall level of evidence. Depending on perspective, different expert panels, professional organizations, or individual physicians may use different cut points of overall strength of evidence in formulating therapeutic guidelines or in taking action. However, a formal description of the level of evidence provides a uniform framework for the data, leading to specific recommendations.

The PDQ Adult Treatment Editorial Board and the PDQ Pediatric Treatment Editorial Board add information on levels of evidence, described below, to the PDQ Adult Cancer Treatment Summaries and the PDQ Pediatric Cancer Treatment Summaries when appropriate.

Strength of Study Design

The various types of study design are described below in descending order of strength:

1. Randomized controlled clinical trials (RCTs).
   The double-blinded RCT is the gold standard of study design. To meet this designation, the study allocation must be blinded to the physician both before and after the randomization and treatment assignment take place. This design provides protection from allocation bias by the investigator and from bias in assessment of outcomes by both the investigator and the patient. Unfortunately, most clinical trials in oncology cannot be double-blinded after treatment allocation because procedures or toxic effects often vary substantially among study allocations in ways that are obvious to both the health care professional and the patient.
2. Meta-analyses of RCTs.
   Meta-analyses of randomized studies offer a quantitative synthesis of previously conducted studies. The strength of evidence from a meta-analysis is based on the quality of the conduct of individual studies. Moreover, meta-analyses can magnify small systematic errors in individual studies. A study comparing the results of single, large, randomized trials with those of meta-analyses of smaller trials published earlier on the same topics showed only fair agreement (kappa statistic, 0.35). Outcomes of the large RCTs were not predicted accurately by the meta-analysis 35% of the time.[1,2] Meta-analyses performed by different investigators to address the same clinical issue can reach contradictory conclusions.[2] Therefore, meta-analyses of randomized studies are placed in the same or lower category of strength of evidence as randomized studies, not at a higher level.
3. Nonrandomized controlled clinical trials.
   This category includes trials in which treatment allocation was nonrandomized and prospective. The allocation would be known to the investigator before informed consent is obtained from the patient. An imbalance can occur in treatment allocation under such circumstances. Subset analyses within randomized trials often fall into this category of evidence.
   Subset analyses of randomized studies are subject to errors inherent in multiplicity (i.e., statistically significant results are to be expected because of random variation of measured effects in multiple subsets). Therefore, subset analyses do not represent the same strength of evidence as the overall analysis of a randomized trial as designed unless explicit prospective hypotheses are made for the analyzed subset. Subset analyses should be placed in the next lower category of study design, nonrandomized controlled clinical trials.
4. Case series or other observational study designs.
   These clinical experiences are the weakest form of study design, but they may be the only available or practical information in support of a therapeutic strategy. This is especially true in the case of rare diseases or when the evolution of the therapy predates the common use of randomized study designs in medical practice. Case series or observational designs may also provide the only practical design when treatments in study arms are radically different (e.g., amputation vs. limb-sparing surgery). Nevertheless, they always raise issues of patient selection and comparability with other populations. In terms of generalizability to other populations, the strongest to weakest study designs are the following: population-based studies that have a definable population, nonpopulation-based but consecutive series, and nonconsecutive cases. Some study designs (e.g., cohort and case-control studies) have internal-control study subjects, while others do not (e.g., case-only series with no internal comparison group or case-only series that are compared with historical controls).
   Even large, population-based, observational studies with internal controls that compare therapeutic strategies in oncology should be interpreted with extreme caution. In a study that directly compared observational studies with RCT results, investigators performed a systematic MEDLINE search for observational studies published from 2000 to 2016 using data from the Surveillance, Epidemiology, and End Results (SEER) Program, SEER-Medicare, or the National Cancer Database that compared treatment regimens for any diagnosis of cancer.[3] The investigators matched 350 treatment comparisons to 121 RCTs that made the same comparison. They found no significant correlation between the hazard ratios (HRs) of the observational studies and the matching RCTs (concordance correlation coefficient, 0.08; 95% confidence interval [CI], -0.07 to 0.23). Only 40% of matched studies agreed with respect to treatment effects (kappa statistic, 0.037), and only 62% of the HRs in observational studies were within the 95% CI of the matched RCT. None of these correlations exceeded what would be expected by chance, and correlations did not improve in the studies that used the most sophisticated statistical methods of analysis, including propensity score weighting, instrumental variable adjustment, or sensitivity analysis. Of note, among the 70 observational studies ranked as rigorous and that reported overall survival, 35 reported a positive result, while the RCT reported either no difference or showed an effect in the opposite direction.

Strength of Study End Points

Commonly measured end points for adult and pediatric cancer treatment studies are listed below in descending order of strength:

1. Overall survival from a defined time (or total mortality).
   This outcome is arguably the most important one to patients and is also the most easily defined and least subject to investigator bias.
2. Cause-specific mortality (or cause-specific mortality from a defined time).
   Although this may be the most biologically important outcome in a disease-specific intervention, it is a more subjective end point than total mortality and more subject to investigator bias in its determination. This end point may also miss important effects of therapy that may actually shorten overall survival.
3. Carefully assessed quality of life.
   This is an extremely important end point to patients. Careful documentation of this end point within a strong study design is therefore sufficient for most physicians to incorporate a treatment into their practices.
4. Indirect surrogates.

   • Event-free survival.
   • Relapse-free survival.
   • Disease-free survival.
   • Progression-free survival.
   • Tumor response rate.
   These end points may be subject to investigator interpretation. More importantly, they may, but do not automatically, translate into direct patient benefit such as survival or quality of life. Nevertheless, it is rational in many circumstances to use a treatment that improves these surrogate end points while awaiting a more definitive end point to support its use.

Summation of Levels of Evidence

After considering the strength of the study design and the strength of the study end points, the following levels of evidence may be added to PDQ Adult Cancer Treatment Summaries and PDQ Pediatric Cancer Treatment Summaries:

• A1 Evidence. Randomized controlled clinical trial (RCT) (double-blinded or nonblinded) with an end point of overall survival (OS) from a defined time, total mortality, or cause-specific mortality.
• A2 Evidence. Meta-analysis of RCTs with an end point of OS from a defined time, total mortality, or cause-specific mortality.
• A3 Evidence. RCT (double-blinded or nonblinded) with an end point of quality of life that is well-collected, clinically meaningful, and carefully assessed.
• B1 Evidence. RCT (double-blinded or nonblinded) with an end point of event-free survival (EFS), relapse-free survival (RFS), disease-free survival (DFS), or progression-free survival (PFS) differences.
• B2 Evidence. Meta-analysis of RCTs with an end point of EFS, DFS, PFS, or carefully assessed quality of life.
• B3 Evidence. RCT (double-blinded or nonblinded) with an end point of tumor response rate or quality-of-life measurement that does not reach the level described in A3.
• B4 Evidence. Nonrandomized, multicenter, prospective, controlled clinical trial with a planned comparison of efficacy including an end point of OS from a defined time, total mortality, cause-specific mortality, carefully assessed quality of life, EFS, DFS, PFS, or tumor response differences.
• C1 Evidence. Case series or other observational study design, including trials with nonconsecutive cases, with an end point of OS from a defined time, total mortality, cause-specific mortality, or carefully assessed quality of life.
• C2 Evidence. Case series or other observational study design, including trials with nonconsecutive cases, with an end point of EFS, RFS, DFS, or PFS differences.
• C3 Evidence. Case series or other observational study design, including trials with nonconsecutive cases, with an end point of tumor response rate or quality-of-life measurement that does not reach the level described in A3.
• D Evidence. Anecdotal experience or expert opinion.

Conclusion

Beyond the level of evidence, all recommendations must take into account other issues that cannot be so easily quantified, such as toxicity, width of confidence intervals of observations, trial size, quality assurance in the trial, and cost. Nevertheless, the PDQ ranking system provides a categorization of strength of evidence as a starting point to evaluate study results.

Latest Updates to This Summary (08/03/2023)

The PDQ cancer information summaries are reviewed regularly and updated as new information becomes available. This section describes the latest changes made to this summary as of the date above.

Strength of Study End Points

Added text to state that relapse-free survival is a commonly measured end point for adult and pediatric cancer treatment studies.

Summation of Levels of Evidence

Revised text to define B1 evidence as a randomized controlled trial with an end point of event-free survival (EFS), relapse-free survival (RFS), disease-free survival (DFS), or progression-free survival (PFS) differences. Also revised text to define C2 evidence as a case series or other observational study design, including trials with nonconsecutive cases, with an end point of EFS, RFS, DFS, or PFS differences.

This summary is written and maintained by the PDQ Adult Treatment Editorial Board, which is editorially independent of NCI. The summary reflects an independent review of the literature and does not represent a policy statement of NCI or NIH. More information about summary policies and the role of the PDQ Editorial Boards in maintaining the PDQ summaries can be found on the About This PDQ Summary and PDQ® Cancer Information for Health Professionals pages.

About This PDQ Summary