Appendix FQuality Assessment Methods

Publication Details

Individual studies were rated as “good,” “fair” or “poor” as defined below:1-3

Studies rated “good” have the least risk of bias and results are considered valid. Good quality studies include clear descriptions of the population, setting, interventions, and comparison groups; a valid method for allocation of patients to treatment; low dropout rates, and clear reporting of dropouts; appropriate means for preventing bias; appropriate measurement of outcomes, and reporting results.

Studies rated “fair” are susceptible to some bias, but it is not sufficient to invalidate the results. These studies do not meet all the criteria for a rating of good quality because they have some deficiencies, but no flaw is likely to cause major bias. The study may be missing information, making it difficult to assess limitations and potential problems. The “fair” quality category is broad, and studies with this rating vary in their strengths and weaknesses: the results of some fair-quality studies are likely to be valid, while others are only probably valid.

Studies rated “poor” have significant flaws that imply biases of various types that may invalidate the results. They have a serious or “fatal” flaw in design, analysis, or reporting; large amounts of missing information; or discrepancies in reporting. The results of these studies are at least as likely to reflect flaws in the study design as the true difference between the compared drugs.

For Controlled Trials

Each criterion was given an assessment of yes, no, or unclear.

  1. Was the assignment to the treatment groups really random?
    Adequate approaches to sequence generation:
    • Computer-generated random numbers
    • Random numbers tables
    Inferior approaches to sequence generation:
    • Use of alternation, case record numbers, birth dates or week days
    Randomization reported, but method not stated
    Not clear or not reported
    Not randomized
  2. Was the treatment allocation concealed?
    Adequate approaches to concealment of randomization:
    • Centralized or pharmacy-controlled randomization (randomization performed without knowledge of patient characteristics).
    • Serially-numbered identical containers
    • On-site computer based system with a randomization sequence that is not readable until allocation
    • Sealed opaque envelopes
    Inferior approaches to concealment of randomization:
    • Use of alternation, case record numbers, birth dates or week days
    • Open random numbers lists
    • Serially numbered non- opaque envelopes
    • Not clear or not reported
  3. Were the groups similar at baseline in terms of prognostic factors?
  4. Were the eligibility criteria specified?
  5. Were outcome assessors and/or data analysts blinded to the treatment allocation?
  6. Was the care provider blinded?
  7. Was the patient kept unaware of the treatment received?
  8. Did the article include an intention-to-treat analysis, or provide the data needed to calculate it (i.e., number assigned to each group, number of subjects who finished in each group, and their results)?
  9. Did the study maintain comparable groups?
  10. Did the article report attrition, crossovers, adherence, and contamination?
  11. Is there important differential loss to followup or overall high loss to followup?

For Cohort Studies

Each criterion was given an assessment of yes, no, or unclear.

  1. Did the study attempt to enroll all (or a random sample of) patients meeting inclusion criteria, or a random sample (inception cohort)?
  2. Were the groups comparable at baseline on key prognostic factors (e.g., by restriction or matching)?
  3. Did the study use accurate methods for ascertaining exposures, potential confounders, and outcomes?
  4. Were outcome assessors and/or data analysts blinded to treatment?
  5. Did the article report attrition?
  6. Did the study perform appropriate statistical analyses on potential confounders?
  7. Is there important differential loss to followup or overall high loss to followup?
  8. Were outcomes pre-specified and defined, and ascertained using accurate methods?

For Case-control Studies

Each criterion was given an assessment of yes, no, or unclear.

  1. Did the study attempt to enroll all (or a random sample of) cases using pre-defined criteria?
  2. Were the controls derived from the same population as the cases, and would they have been selected as cases if the outcome was present?
  3. Were the groups comparable at baseline on key prognostic factors (e.g., by restriction or matching)?
  4. Did the study report the proportion of cases and controls who met inclusion criteria that were analyzed?
  5. Did the study use accurate methods for identifying outcomes?
  6. Did the study use accurate methods for ascertaining exposures and potential confounders?
  7. Did the study perform appropriate statistical analyses on potential confounders?