U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Chou R, Helfand M, Peterson K, et al. Comparative Effectiveness and Safety of Analgesics for Osteoarthritis [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006 Sep. (Comparative Effectiveness Reviews, No. 4.)

Cover
  • This publication is provided for historical reference only and the information may be out of date.

This publication is provided for historical reference only and the information may be out of date.

Cover of Comparative Effectiveness and Safety of Analgesics for Osteoarthritis

Comparative Effectiveness and Safety of Analgesics for Osteoarthritis [Internet].

Show details

Appendix EQuality Assessment Methods

The methods outlined in this document ensure that the products created in this process are methodologically sound, scientifically defensible, reproducible, and well-documented. This document has been adapted from the Procedure Manual developed by the Methods Work Group of the United States Preventive Services Task Force (version 1.9, September 2001), with additional material from the NHS Centre for Reviews and Dissemination (CRD) report on Undertaking Systematic Reviews of Research on Effectiveness: CRD’s Guidance for Carrying Out or Commissioning Reviews (2nd edition, 2001) and “The Database of Abstracts of Reviews of Effects (DARE)” in Effectiveness Matters, vol. 6, issue 2, December 2002, published by the CRD.

All studies or systematic reviews that are included are assessed for quality, and assigned a rating of “good”, “fair” or “poor”. Studies that have a fatal flaw in one or more criteria are rated poor quality; studies which meet all criteria, are rated good quality; the remainder are rated fair quality. As the “fair quality” category is broad, studies with this rating vary in their strengths and weaknesses: the results of some fair quality studies are likely to be valid, while others are only probably valid. A “poor quality” trial is not valid—the results are at least as likely to reflect flaws in the study design as the true difference between the compared drugs.

For Controlled Trials

Assessment of Internal Validity

  1. Was the assignment to the treatment groups really random?
    • Adequate approaches to sequence generation:
    • Computer-generated random numbers
    • Random numbers tables
    • Inferior approaches to sequence generation:
    • Use of alternation, case record numbers, birth dates or week days
    • Not reported
  2. Was the treatment allocation concealed?
    • Adequate approaches to concealment of randomization:
      • Centralized or pharmacy-controlled randomization
      • Serially-numbered identical containers
        • On-site computer based system with a randomization sequence that is not readable until allocation
        • Other approaches sequence to clinicians and patients
      • Inferior approaches to concealment of randomization:
      • Use of alternation, case record numbers, birth dates or week days
      • Open random numbers lists
        • Serially numbered envelopes (even sealed opaque envelopes can be subject to manipulation)
      • Not reported
  3. Were the groups similar at baseline in terms of prognostic factors?
  4. Were the eligibility criteria specified?
  5. Were outcome assessors blinded to the treatment allocation?
  6. Was the care provider blinded?
  7. Was the patient kept unaware of the treatment received?
  8. Did the article include an intention-to-treat analysis, or provide the data needed to calculate it (i.e., number assigned to each group, number of subjects who finished in each group, and their results)?
  9. Did the study maintain comparable groups?
  10. Did the article report attrition, crossovers, adherence, and contamination?
  11. Is there important differential loss to followup or overall high loss to followup? (give numbers in each group)

Assessment of External Validity (Generalizability)

  1. How similar is the population to the population to whom the intervention would be applied?
  2. How many patients were recruited?
  3. What were the exclusion criteria for recruitment? (Give numbers excluded at each step)
  4. What was the funding source and role of funder in the study?
  5. Did the control group receive the standard of care?
  6. What was the length of followup? (Give numbers at each stage of attrition.)

For Studies Reporting Complications/Adverse Effects

Assessment of Internal Validity

  1. Was the selection of patients for inclusion non-biased (Was any group of patients systematically excluded)?
  2. Is there important differential loss to followup or overall high loss to followup? (Give numbers in each group.)
  3. Were the events investigated specified and defined?
  4. Was there a clear description of the techniques used to identify the events?
  5. Was there non-biased and accurate ascertainment of events (independent ascertainer; validation of ascertainment technique)?
  6. Were potential confounding variables and risk factors identified and examined using acceptable statistical techniques?
  7. Did the duration of followup correlate to reasonable timing for investigated events? (Does it meet the stated threshold?)

Assessment of External Validity

  1. Was the description of the population adequate?
  2. How similar is the population to the population to whom the intervention would be applied?
  3. How many patients were recruited?
  4. What were the exclusion criteria for recruitment? (Give numbers excluded at each step)
  5. What was the funding source and role of funder in the study?

Systematic Reviews

1. Is there a clear review question and inclusion/exclusion criteria reported relating to the primary studies?

A good quality review should focus on a well-defined question or set of questions, which ideally will refer to the inclusion/exclusion criteria by which decisions are made on whether to include or exclude primary studies. The criteria should relate to the four components of study design, indications (patient populations), interventions (drugs), and outcomes of interest. In addition, details should be reported relating to the process of decision-making, i.e., how many reviewers were involved, whether the studies were examined independently, and how disagreements between reviewers were resolved.

2. Is there evidence of a substantial effort to search for all relevant research?

This is usually the case if details of electronic database searches and other identification strategies are given. Ideally, details of the search terms used, date and language restrictions should be presented. In addition, descriptions of hand-searching, attempts to identify unpublished material, and any contact with authors, industry, and research institutes should be provided. The appropriateness of the database(s) searched by the authors should also be considered, e.g. if MEDLINE is searched for a review looking at health education, then it is unlikely that all relevant studies will have been located.

3. Is the validity of included studies adequately assessed?

A systematic assessment of the quality of primary studies should include an explanation of the criteria used (e.g., method of randomization, whether outcome assessment was blinded, whether analysis was on an intention-to-treat basis). Authors may use either a published checklist or scale, or one that they have designed specifically for their review. Again, the process relating to the assessment should be explained (i.e. how many reviewers involved, whether the assessment was independent, and how discrepancies between reviewers were resolved).

4. Is sufficient detail of the individual studies presented?

The review should demonstrate that the studies included are suitable to answer the question posed and that a judgement on the appropriateness of the authors’ conclusions can be made. If a paper includes a table giving information on the design and results of the individual studies, or includes a narrative description of the studies within the text, this criterion is usually fulfilled. If relevant, the tables or text should include information on study design, sample size in each study group, patient characteristics, description of interventions, settings, outcome measures, follow-up, drop-out rate (withdrawals), effectiveness results and adverse events.

5. Are the primary studies summarized appropriately?

The authors should attempt to synthesize the results from individual studies. In all cases, there should be a narrative summary of results, which may or may not be accompanied by a quantitative summary (meta-analysis).

For reviews that use a meta-analysis, heterogeneity between studies should be assessed using statistical techniques. If heterogeneity is present, the possible reasons (including chance) should be investigated. In addition, the individual evaluations should be weighted in some way (e.g., according to sample size, or inverse of the variance) so that studies that are considered to provide the most reliable data have greater impact on the summary statistic.

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (933K)

In this Page

Other titles in these collections

Recent Activity

ClearTurn OffTurn On

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...