Guidelines for Quality Assessment and Data Extraction Pertaining to Efficacy of Interventions

(A) Quality assessment for systematic reviews:

Question (1) – Score as “yes” if the review states the search terms used in the electronic database search (e.g. the terms smoking cessation and tobacco use disorder were searched as subject headings and text words). The entire search strategy does not need to be provided.

Question (2) – To be scored as “yes” at least two electronic databases (e.g. Medline and Cancerlit) AND the reference lists of included studies must have been searched.

Question (3) – Score as “yes” if the review comments on either the level of evidence or the design (RCT, cohort, etc.) of the studies included in the systematic review.

Question (4) – To be scored as “yes” the review must rate formally the quality of the included studies AND that quality assessment must address at least 4 out of 9 of the criteria listed. In addition, if the review includes studies that are not of a randomized controlled trial (RCT) design the confounders criteria must be assessed to be scored as “yes”.

Question (5) – If the systematic review does not include a meta-analysis, the review must state the reason why a meta-analysis was not performed (due to the results of the test for heterogeneity or the diversity of outcomes assessed in the included studies) to be scored as “yes”. If the review does include a meta-analysis, the review must have tested for heterogeneity and used the results to decide whether it was appropriate to perform a meta-analysis for the review to be scored as “yes”.

Question (6) – Score as “yes” if the conclusions from the review can be supported from the results of the included studies reported in the review.

The quality of the reviews will be scored out of six (each question marked “yes” will receive one point). Reviews with a score of 5 or 6 will be rated as STRONG, reviews with a score of 3 to 4 will be rated as MODERATE and reviews with a score of 2 or less will be rated as WEAK.

Any questions marked as “uncertain” or any differences in scoring between the two independent assessors will be resolved by consensus.

(B) Data extraction:

Question (3) - Is there any reason not to proceed with data extraction?

Answer “yes” if the any of the following criteria apply:

1. The paper is not a systematic review (e.g. the review does not state the criteria studies it must meet to be included)
2. The review does not evaluate the effectiveness and/or efficacy of cancer control interventions that promote the uptake of one or more of the cancer control behaviors addressed by this evidence report (e.g. adult smoking cessation, adult healthy diet, mammography, cervical cancer screening, or control of cancer pain)
3. There are no extractable data, in terms, of topic-specific main results or conclusions

Please state the reason for not proceeding with data extraction in the space provided.

Question (4) - What was the purpose and/or objective(s) of the systematic review? Please use exact quotes from the review and provide the page number in the review where the quote was found. If possible, please highlight the quoted section in the review and mark Q4 beside it.

Question (5) - Total number of studies included in the systematic review. Please report the final number of studies included in the review. This number should reflect all of the studies included, regardless of whether some of the topics included are not addressed in our evidence report. Report the total number as “not clear” if the number is not provided and it cannot be determined from the evidence tables provided.

Number of studies included in each of the topic areas. Please report the number of included studies in the review for each of the five topics covered by this evidence report. Mark as “not clear” if the topic-specific number of included studies is not stated and cannot be determined from the evidence tables provided.

Question (6) - State the systematic review's inclusion criteria. Please use exact quotes from the review and provide the page number in the review where the quote was found. Please highlight the quoted section in the review and mark Q6 beside it. Be sure to indicate if the review limits the papers selected to English language only, or a specific country.

Question (7) - Sub Populations targeted? Please indicate if any minority groups were mentioned in the review (e.g.: ethnic, age, occupation). Gender may be highlighted in smoking, diet or cancer pain only.

Question (8) -Age criteria, all subjects. Please specify the ages covered in the review. If possible, state a minimum age and a maximum age. Numbers may be found in tables, you can report the mean age. Options for assume adult only or not clearly stated are also provided. If more than 1 topic is covered in the paper, please try to list the minimum and maximum age for each topic population, as they will be presented in separate evidence tables in our report.

Question (9) - What cancer-control intervention(s) were assessed? Some examples of cancer-control interventions are: patient reminders, physician prompts, and cost-free services. Please report all interventions that you see mentioned. Best places to look are the evidence tables and results section. If more than 1 topic area is included in the review, please group interventions by topic area, as they will be presented in separate evidence tables.

Question (10) - Report how the interventions were classified by the review? Most common classification is patient-directed, physician-directed, system-directed or multi-strategy. Best places to look are the evidence tables and results section. Please use exact terminology used in the paper.

Question (11) - Was a meta-analysis performed? If “yes”, please specify whether all studies included in the systematic review were included in the meta-analysis. If only a sub-set of the studies were included in the meta-analysis please state the criteria used to select the studies (e.g. all RCT studies or all studies of a certain intervention) and the number of studies included in the meta-analysis. If meta-analysis was NOT done, skip to question 12.

Question (12) - Main results from the systematic review. The main results reported should reflect the purpose and objective(s) of the systematic review reported in Question (4). If the systematic review covers more than one of our evidence report topic areas (e.g. mammography and cervical cancer screening) please record the results in separate topic-specific sections, as the results will be reported in separate topic-specific evidence tables. Please use exact quotes from the review and provide the page number in the review where the quotes were found. Please highlight the quoted section(s) in the review and mark Q10 beside the sections.

Question (13) - What are the main conclusions of the systematic review? The main conclusions reported should reflect both the purpose and objective(s) of the systematic review reported in Question (4) and the main results reported in Question (10). If the systematic review covers more than one of our evidence report topic areas (e.g. mammography and cervical cancer screening) please record the conclusions in separate topic-specific sections, as the conclusions will be reported in separate topic-specific evidence tables. Please use exact quotes from the review and provide the page numbers in the review where the quotes were found. Please highlight the quoted section(s) in the review and mark Q13 beside the sections. Report if results covering other topics are in the paper.

Question (14) - Were any of the systematic review's references marked as “retrieve” for full-text screening? Please review the reference list from the systematic review. Mark for “retrieval” any reference that meets either of the following criteria: (1) the title of the reference states it is a systematic review, quantitative review, meta-analysis or overview that evaluates the effectiveness and/or efficacy of cancer-control interventions that promote uptake of adult smoking cessation, adult healthy diet, mammography, cervical cancer screening, or control of cancer pain OR (2) the text of the review makes reference to other published systematic reviews that evaluate the efficacy and/or effectiveness of cancer control interventions promoting uptake in one or more of the areas addressed by this evidence report. Please remember to indicate the number of references marked for retrieval. This is for administrative purposes.

Question (15) - Any additional comments regarding this systematic review should be noted here. We are specifically interested in knowing if it is a Cochrane review, or if there are any unusual outcomes or notes of interest - additional articles included beyond their review strategy. Please comment if data is divided into particular ethnic groups or ages etc. when appropriate.

Guidelines for Full-text Relevance Screening for Questions Pertaining to Dissemination:

Systematic review of primary studies evaluating the diffusion or dissemination of cancer control interventions

Question #1 : Was the paper published in English?

Exclude if the study was not published in English.

Question #2 : Type of study.

Part II of the evidence report will include any papers that meet the criteria (outlined below) for a primary study, systematic review or overview of systematic review.

Primary study: To be included as a primary study the paper must be not be a position paper or any type of review. The paper must be a report of an experiment or investigation (e.g. RCT, non-RCT, cohort study, case series or survey). Primary studies published only as abstracts or dissertations will be excluded.

Question #3 : Does the study evaluate the diffusion or dissemination of a cancer control intervention?

Papers will be included if they have evaluated the process of dissemination or diffusion of a CCI (the scope of CCIs covered by this evidence report is completely defined in the Guidelines for full-text relevance screening for Part I). Examples of potential dissemination strategies are: train-the-trainer sessions, continuing medical education classes, mailed materials regarding CCIs (e.g. practice guidelines - [check s/c paper]), use of educators to inform health care providers about specific CCIs and mass media campaigns to increase awareness of a CCI.

Dissemination^*: the spread of knowledge from its source to a target audience. It includes any special efforts to ensure that individuals acquire a working acquaintance with that knowledge. Successful dissemination requires both accurate communication from the source and accurate understanding by the recipients (“competence”).

Diffusion^*: is a somewhat passive subset of dissemination in which no special efforts are made to promote the spread of knowledge. “Regular channels” for diffusion include journal articles and conversation.

Cancer control intervention: For the purpose of this evidence report we will be restricting the definition of cancer control interventions (CCI) to those interventions that promote the uptake of specific behaviors by health care providers, patients, or both. The term ‘behaviors’ is being used in a broad context, and encompasses a spectrum of outcomes from change in knowledge and attitudes to influencing practices. The following types of interventions will be included: audit and feedback, computer reminder systems, prompts, opinion leaders, continuing medical education, practice guidelines (as an intervention), incentives, mass media campaigns, peer-leaders, patient telephone counseling, educational outreach, mailed invitation/educational material, removal of barriers.

Question #4 : Topic area(s)

Check all topic areas included in the paper. The five topic areas addressed by this evidence report are:

• Adult smoking cessation – The following tobacco use areas will be excluded: prenatal smoking cessation, pre-operative smoking cessation, environmental tobacco smoke, preventing initiation of primary tobacco use or tobacco sales to minors.
• Adult healthy diet – Studies of the dissemination or diffusion of CCIs that promote increased consumption of fruits, vegetables, or fiber; diets that are low in fat; heart-smart diets or other diets that promote healthy eating will be included.
• Breast cancer screening – This evidence report will focus on mammography screening for breast cancer. The following breast cancer screening areas will be excluded: breast self-examination (BSE) and compliance after an abnormal mammography finding.
• Cervical cancer screening – This evidence report will focus on the PAP test to screen for cervical cancer. The topic of compliance after an abnormal PAP test will be excluded.
• Control of cancer pain – This evidence report will focus on interventions that promote control of cancer pain in adults. Interventions that exclusively focus on control of non-cancer related pain will be excluded.

Question #5 : Age Criteria

For the purpose of this systematic review all papers that focus exclusively on adolescents or children will be excluded.

Check the box “states adult” if the paper explicitly states that the age criteria was adult (e.g. 18 years of age or greater). For those papers that do not specify the age range of the subjects check the most appropriate age criteria (e.g. assume adult or not clear). For those papers which include both adolescents and adults check the box marked ‘adolescents and adults’.

Question #6 : Specific minority population targeted?

If the paper evaluates the diffusion or dissemination of cancer control interventions for specific populations (e.g. African-American, American-Indian, low socioeconomic status, or low educational level) check the “yes” box and specify the targeted population.

Question #7 : Exclusively focused on one or more of the five specified topic area(s) covered by this evidence report?

Indicate whether the paper exclusively focuses on one or more of the five topic areas included in this evidence report. If “yes”, simply check the “yes” box. If “no”, check the “no” box and specify ALL that apply (e.g. other cancer-related topic areas and/or other non-cancer-related areas).

Component Ratings of Study:

For each of the five components A - F, use the following descriptions as a roadmap.

A) SELECTION BIAS

Strong

1. The selected individuals are very likely to be representative of the target population (Q1 is 1); and
2. There is greater than 80% participation (Q2 is 1).

Moderate

1. The selected individuals are at least somewhat likely to be representative of the target population (Q1 is 1 or 2); and
2. There is 60 - 79% participation (Q2 is 2).
3. ‘Moderate’ may also be assigned if Q1 is 1 or 2 and Q2 is 5 (can't tell).

Weak

1. The selected individuals are not likely to be representative of the target population (Q1 is 3); or
2. There is less than 60% participation (Q2 is 3) or
3. Selection is not described (Q1 is 4); and the level of participation is not described (Q2 is 5).

B) DESIGN - a rating of:

Strong will be assigned to those articles that described RCTs and CCTs.

Moderate will be assigned to those that described a cohort analytic study, a case control study, a cohort design, or an interrupted time series.

Weak will be assigned to those that used any other method or did not state the method used.

C) CONFOUNDERS - a rating of:

Strong will be assigned to those articles that controlled for at least 80% of relevant confounders (Q1 is 2); or (Q2 is 1).

Moderate will be given to those studies that controlled for 60 – 79% of relevant confounders (Q1 is 1) and (Q2 is 2).

Weak will be assigned when less than 60% of relevant confounders were controlled (Q1 is 1) and (Q2 is 3) or control of confounders was not described (Q1 is 3) and (Q2 is 4).

D) BLINDING - a rating of:

Strong

1. The outcome assessor is not aware of the intervention status of participants (Q1 is 2); and
2. The study participants are not aware of the research question (Q2 is 2).

Moderate

1. The outcome assessor is not aware of the intervention status of participants (Q1 is 2); or
2. The study participants are not aware of the research question (Q2 is 2); or
3. Blinding is not described (Q1 is 3 and Q2 is 3).

Weak

1. The outcome assessor is aware of the intervention status of participants (Q1 is 1); and
2. The study participants are aware of the research question (Q2 is 1).

E) DATA COLLECTION METHODS - a rating of:

Strong

1. The data collection tools have been shown to be valid (Q1 is 1); and
2. The data collection tools have been shown to be reliable (Q2 is 1).

Moderate

1. The data collection tools have been shown to be valid (Q1 is 1); and
2. The data collection tools have not been shown to be reliable (Q2 is 2) or reliability is not described (Q2 is 3).

Weak

1. The data collection tools have not been shown to be valid (Q1 is 2) or both reliability and validity are not described (Q1 is 3 and Q2 is 3).

F) WITHDRAWALS AND DROP-OUTS - a rating of:

Strong will be assigned when the follow-up rate is 80% or greater (Q2 is 1).

Moderate will be assigned when the follow-up rate is 60 – 79% (Q2 is 2) OR Q2 is 5 (N/A).

Weak will be assigned when a follow-up rate is less than 60% (Q2 is 3) or if the withdrawals and drop-outs were not described (Q2 is 4).

Quality Assessment Tool for Quantitative Studies Dictionary

The purpose of this dictionary is to describe items in the tool thereby assisting raters to score study quality. Due to under-reporting or lack of clarity in the primary study, raters will need to make judgements about the extent that bias may be present. When making judgements about each component, raters should form their opinion based upon information contained in the study rather than making inferences about what the authors intended.

A) SELECTION BIAS

(Q1) Participants are more likely to be representative of the target population if they are randomly selected from a comprehensive list of individuals in the target population (score very likely). They may not be representative if they are referred from a source (e.g. clinic) in a systematic manner (score somewhat likely) or self-referred (score not likely).

(Q2) Refers to the % of subjects in the control and intervention groups that agreed to participate in the study before they were assigned to intervention or control groups.

B) STUDY DESIGN

In this section, raters assess the likelihood of bias due to the allocation process in an experimental study. For observational studies, raters assess the extent that assessments of exposure and outcome are likely to be independent. Generally, the type of design is a good indicator of the extent of bias. In stronger designs, an equivalent control group is present and the allocation process is such that the investigators are unable to predict the sequence.

Randomized Controlled Trial (RCT)

An experimental design where investigators randomly allocate eligible people to an intervention or control group. A rater should describe a study as an RCT if the randomization sequence allows each study participant to have the same chance of receiving each intervention and the investigators could not predict which intervention was next. If the investigators do not describe the allocation process and only use the words ‘random’ or ‘randomly’, the study is described as a controlled clinical trial.

See below for more details.

Was the study described as randomized?

Score YES, if the authors used words such as random allocation, randomly assigned, and random assignment.

Score NO, if no mention of randomization is made.

Was the method of randomization described?

Score YES, if the authors describe any method used to generate a random allocation sequence.

Score NO, if the authors do not describe the allocation method or describe methods of allocation such as alternation, case record numbers, dates of birth, day of the week, and any allocation procedure that is entirely transparent before assignment, such as an open list of random numbers of assignments.

If NO is scored, then the study is a controlled clinical trial.

Was the method appropriate?

Score YES, if the randomization sequence allowed each study participant to have the same chance of receiving each intervention and the investigators could not predict which intervention was next. Examples of appropriate approaches include assignment of subjects by a central office unaware of subject characteristics, or sequentially numbered, sealed, opaque envelopes.

Score NO, if the randomization sequence is open to the individuals responsible for recruiting and allocating participants or providing the intervention, since those individuals can influence the allocation process, either knowingly or unknowingly.

If NO is scored, then the study is a controlled clinical trial.

Controlled Clinical Trial (CCT)

An experimental study design where the method of allocating study subjects to intervention or control groups is open to individuals responsible for recruiting subjects or providing the intervention. The method of allocation is transparent before assignment, e.g. an open list of random numbers or allocation by date of birth, etc.

Cohort analytic (two group pre and post)

An observational study design where groups are assembled according to whether or not exposure to the intervention has occurred. Exposure to the intervention is not under the control of the investigators. Study groups might be non-equivalent or not comparable on some feature that affects outcome.

Case control study

A retrospective study design where the investigators gather ‘cases’ of people who already have the outcome of interest and ‘controls’ who do not. Both groups are then questioned or their records examined about whether they received the intervention exposure of interest.

Cohort (one group pre + post (before and after)

The same group is pretested, given an intervention, and tested immediately after the intervention. The intervention group, by means of the pretest, act as their own control group.

Interrupted time series

A time series consists of multiple observations over time. Observations can be on the same units (e.g. individuals over time) or on different but similar units (e.g. student achievement scores for particular grade and school). Interrupted time series analysis requires knowing the specific point in the series when an intervention occurred.

C) CONFOUNDERS

By definition, a confounder is a variable that is associated with the intervention or exposure and causally related to the outcome of interest. Even in a robust study design, groups may not be balanced with respect to important variables prior to the intervention. The authors should indicate if confounders were controlled in the design (by stratification or matching) or in the analysis. If the allocation to intervention and control groups is randomized, the authors must report that the groups were balanced at baseline with respect to confounders (either in the text or a table).

D) BLINDING

(Q1) Assessors should be described as blinded to which participants were in the control and intervention groups. The purpose of blinding the outcome assessors (who might also be the care providers) is to protect against detection bias.

(Q2) Study participants should not be aware of (e.g. blinded to) the research question. The purpose of blinding the participants is to protect against reporting bias.

E) DATA COLLECTION METHODS

Tools for primary outcome measures must be described as reliable and valid. If ‘face’ validity or ‘content’ validity has been demonstrated, this is acceptable. Some sources from which data may be collected are described below:

Self reported data includes data that is collected from participants in the study (e.g. completing a questionnaire, survey, answering questions during an interview, etc.).

Assessment/Screening includes objective data that is retrieved by the researchers. (e.g. observations by investigators).

Medical Records / Vital Statistics refers to the types of formal records used for the extraction of the data.

Reliability and validity can be reported in the study or in a separate study. For example, some standard assessment tools have known reliability and validity.

F) WITHDRAWALS AND DROP-OUTS

Score YES if the authors describe BOTH the numbers and reasons for withdrawals and drop-outs.

Score NO if either the numbers or reasons for withdrawals and drop-outs are not reported.

The percentage of participants completing the study refers to the % of subjects remaining in the study at the final data collection period in all groups (e.g. control and intervention groups).

G) INTERVENTION INTEGRITY

The number of participants receiving the intended intervention should be noted (consider both frequency and intensity). For example, the authors may have reported that at least 80 percent of the participants received the complete intervention. The authors should describe a method of measuring if the intervention was provided to all participants the same way. As well, the authors should indicate if subjects received an unintended intervention that may have influenced the outcomes. For example, co-intervention occurs when the study group receives an additional intervention (other than that intended). In this case, it is possible that the effect of the intervention may be over-estimated. Contamination refers to situations where the control group accidentally receives the study intervention. This could result in an under-estimation of the impact of the intervention.

H) ANALYSIS APPROPRIATE TO QUESTION

Was the quantitative analysis appropriate to the research question being asked?

An intention-to-treat analysis is one in which all the participants in a trial are analyzed according to the intervention to which they were allocated, whether they received it or not. Intention-to-treat analyses are favoured in assessments of effectiveness as they mirror the noncompliance and treatment changes that are likely to occur when the intervention is used in practice, and because of the risk of attrition bias when participants are excluded from the analysis.

Footnotes

*

Adapted from Lomas J, Haynes MA, Haynes BR. (1988). A taxonomy and critical review of tested strategies for the application of clinical practice recommendations: from “official” to “individual” clinical policy. Am.J.Prev.Med. 4(4suppl): 77–97.