Relevance Screening Forms

INSTRUCTIONS: PLEASE USE REVIEW-SPECIFIC DEFINITIONS IN SELECTING RESPONSES.

LEVEL 1 SCREENING:

1. Does this refer to breast cancer in women? YES NO CAN'T TELL
2. Does this refer to (the reporting/documentation of, or, patient-reported quality of life or patient-reported satisfaction relating to) its diagnosis, treatment (e.g., supportive care), or followup care?
   YES NO CAN'T TELL
3. Does this refer to at least one of the following?¹

   • Quality measure/ment(s) made available, published, or conducted beginning in 1993
   • Clinical practice guideline (CPG) published beginning in 1996
   • Systematic review (SR) published beginning in 1994
   • Commentary or editorial in a key general or cancer journal describing important (new) evidence that could alter practice/care recommendations, published beginning in 1994²
   • Evidence nominated by key informant, describing (new or missed) evidence that could alter practice/care recommendations³
     YES NO CAN'T TELL

Level 2 SCREENING:⁴

1. To which does it refer? (select all that apply)
   • ο Quality measure/ment(s) made available, published, or conducted beginning in 1993
   • ο Clinical practice guideline (CPG) published beginning in 1996
   • ο Systematic review (SR) published beginning in 1994
   • ο Commentary or editorial in a key general or cancer journal describing important (new) evidence that could alter practice/care recommendations, published beginning in 1994²
   • ο Evidence nominated by key informant, describing (new or missed) evidence that could alter practice/care recommendations³

LEVEL 3 SCREENING:⁵

1. Does this refer to a clinical practice guideline or systematic review? YES NO CAN'T TELL

Data Abstraction Form

Instructions: Please answer each question. Selecting response options means clicking on them. A text box requires you to provide specific information. When it is not reported (= NR), the question does not apply (= N/A), you cannot tell what/where it is (= CT), or you have no comment to make (= NC), type the relevant code in the text box. If the research report describes more than one quality indicator, answer in this eForm all the questions for the first reported quality indicator/measure while at the same time letting the review manager know that a data abstraction form is required for each additional one.

GENERAL INFORMATION:

1. Initials of reviewer: TEXT BOX (BOX)
2. Reference identification # (Refid#): BOX
3. Author, Year: BOX
4. Number of unique, review-relevant studies that this report describes: BOX
5. Other Refids that refer to this same research project:
6. Publication status (select one):
   • Peer-reviewed journal publication
   • Journal publication
   • Conference abstract/poster
   • Book
   • Book chapter
   • HTA/technical report
   • Thesis
   • Unpublished document
   • Study sponsor's internal report
   • Internet document
   • Other
7. If you answered “Other” to the preceding question, specify what you mean: BOX
8. Country in which the study was conducted (select all that apply):
   • Australia
   • Canada
   • United States
   • Japan
   • United Kingdom (not Ireland)
   • France
   • Germany
   • Italy
   • Finland
   • Russia
   • Other
   • Not reported
9. If you answered “Other” to the preceding question, specify what you mean: BOX
10. Number of sites: BOX
11. Funding source type (select all that apply):
    • Government
    • Industry
    • Private (non-industry)
    • Hospital
    • Other
    • Not reported
    • Can't tell
12. Specify the funding source(s): BOX
13. Number of unique quality measures (or measurements) [QMs] this report describes? BOX
14. Year(s) in which the QMs were assessed? BOX

DEFINITION:

1. Title of QM (the percentage of…): BOX
2. Bibliographic source(s) of QM: BOX
3. Category of QM = “breast cancer and…” (i.e., primary clinical component[s]) (select one):
   • Diagnosis (e.g., pathology)
   • Treatment
   • Follow-up
   • Supportive care
   • Reporting/documentation (e.g., pathology)
4. Institute of Medicine care domains (select all that apply):
   • Effectiveness
   • Patient-centeredness
   • Safety
   • Timeliness
5. Specific type of quality measure (select all that apply):
   • Appropriate use of diagnostic imaging
   • Quality of diagnostic imaging
   • Appropriate use of breast biopsy
   • Quality of diagnostic breast biopsy
   • Appropriate use of sentinel lymph node biopsy
   • Quality of diagnostic sentinel lymph node biopsy
   • Appropriate use of chest x-ray
   • Appropriate use of bone scan
   • Appropriate use of CT scans
   • Appropriate use of MRI
   • Appropriate use of blood tests
   • Availability of pathological staging
   • Accuracy of pathological staging
   • Availability of tumor marker status
   • Accuracy of tumor marker status
   • Availability of genetic testing
   • Accuracy of genetic testing
   • Appropriate use of genetic testing
   • None of the above
6. Specific type of quality measure (select all that apply):
   • Appropriate use of breast conserving surgery (BCS)
   • Quality of BCS
   • Appropriate use of mastectomy (including adequacy of surgical margins)
   • Quality of mastectomy (including adequacy of surgical margins)
   • Appropriate use of lymph node surgery
   • Quality of lymph node surgery
   • Appropriate use of reconstructive surgery
   • Quality of reconstructive surgery
   • Appropriate use of radiation therapy (RT) after BCS
   • Quality of RT after BCS
   • Appropriate use of RT post-mastectomy
   • Quality of RT post-mastectomy
   • Appropriate use of adjuvant and neo-adjuvant systemic therapy (chemotherapy; hormone therapy)
   • Quality of adjuvant and neo-adjuvant systemic therapy (chemotherapy; hormone therapy)
   • Appropriate use of hormonal and chemotherapy management of metastatic disease
   • Quality of hormonal and chemotherapy management of metastatic disease
   • Appropriate dosing of chemotherapy
   • Quality of dosing of chemotherapy
   • Appropriate use of dosing of radiotherapy
   • Quality of dosing of radiotherapy
   • None of the above
7. Specific type of quality measure (select all that apply):
   • Adequacy of documentation of pathology reports
   • Completeness of documentation of pathology reports
   • Adequacy of documentation of operative reports
   • Completeness of documentation of operative reports
   • Adequacy of documentation of radiation reports
   • Completeness of documentation of radiation reports
   • Adequacy of documentation of chemotherapy reports
   • Completeness of documentation of chemotherapy reports
   • None of the above
8. Specific type of quality measure (select all that apply):
   • Quality of follow-up (e.g., timeliness; interventions)
   • Quality of supportive care (e.g., interventions)
   • Quality of life
   • Patient satisfaction
9. Primary measure domain (select all that apply):
   • Structure (e.g., accreditation; number of certified specialists)
   • Access (e.g., attainment of timely & appropriate care)
   • Process (e.g., adherence to recommended care)
   • Outcome, including patient experience (e.g., QOL; patient satisfaction)
10. Specify the stated importance of, or need for, the QM (i.e., rationale; purpose: e.g., wide variation in quality of care; substandard care; over-use; under-use): BOX
11. Specify the references highlighting the importance of, or need for, the QM: BOX
12. Describe the QM's “denominator” (i.e., inclusion/exclusion criteria): BOX
13. Describe the QM's “numerator” (i.e., inclusion/exclusion criteria defining specific subset of denominator): BOX

PERFORMANCE:

1. Was this quality measure “systematically developed” to any degree? (select one)
   • Yes (e.g., pilot-testing: see Development questions)
   • No (a quality indicator whose performance has been merely measured)
   • Can't tell
2. Based on which type(s) of evidence was the criterion/standard defined (select all that apply)?
   • Clinical practice guideline: evidence-based
   • Clinical practice guideline: consensus-based
   • Systematic review of evidence
   • Selective/narrative review (i.e., unsystematic search of literature)
   • Expert (consensus) panel process
   • Other
3. If you answered ‘Other’ to the preceding question, specify what you mean: BOX
4. Name the evidence type(s) (e.g., guidelines), including its year of publication: BOX
5. Which methods of case identification (e.g., cancer registries; claims databases) were used to evaluate the performance of the QM? Please name them. BOX
6. Which data sources (e.g., patient self-report; medical records), per method of case identification, were used to evaluate the performance of the QM? BOX
7. Denominator time window (time period in which patients are reviewed for inclusion in the denominator), per method of case identification: BOX
8. Numerator time window (time period in which patients are reviewed for inclusion in the numerator), per method of case identification: BOX
9. Sample description, per set of identified cases (e.g., national convenience sample of women with…): BOX
10. Response rate (cases with complete data/eligible cases), per set of identified cases: BOX
11. Specify reasons for exclusion (& sample size), per set of identified cases (NOTE: the need for individualized care that contradicts recommended care would constitute an exclusion yet may not indicate poor/inappropriate care): BOX
12. Size of sample(s) analyzed, per set of identified cases: BOX
13. Specify tumor characteristics, per set of identified cases: BOX
14. Specify family history of breast cancer (first degree members), per set of identified cases: BOX
15. Specify proportion of patients in each stage of the disease at the time of study, per set of identified cases: BOX
16. Specify duration of the disease since diagnosis, per set of identified cases: BOX
17. Year diagnosed, per set of identified cases: BOX
18. How diagnosed, per set of identified cases: BOX
19. Sample age, per set of identified cases: BOX
20. Sample socioeconomic status, per set of identified cases: BOX
21. Sample race/ethnicity, per set of identified cases: BOX
22. Other demographic factors (e.g., location of permanent residence), per set of identified cases: BOX
23. Specify population's treatments (including surgery, radiotherapy, & systemic therapy), per set of identified cases: BOX
24. Specify type(s) of surgery (breast conserving surgery, mastectomy with or without reconstructive surgery, etc.), per set of identified cases: BOX
25. Overall concordance rate, per set of identified cases: BOX
26. Variations in rate of concordance according to stratification(s) of the population (e.g., age; vulnerable populations; hospitals; regions), per set of identified cases: BOX
27. Results re possible differences between groups (e.g., odds ratio) identified by stratification: BOX
28. Additional data (e.g., specificity; sensitivity; adaptability), per set of identified cases: BOX
29. Specify the evidence regarding the nature and adequacy of the (e.g., risk) adjustment(s) when cross-population or -database comparisons are made: BOX
30. Specify scientific evidence demonstrating a linkage to improvement in clinical or patient-reported outcomes, per set of identified cases: BOX
31. Results involving scores (e.g., QOL), per set of identified cases (i.e., overall score, with interpretation; scores per group identified by stratification; results reflecting possible differences between groups identified by stratification: e.g., differences in outcome [e.g., survival] associated with receipt/non-receipt of care ‘X’): BOX

CURRENT STATUS:

1. Describe the state of use (i.e., over the past 3 years: e.g., pilot testing; used by organizations yet discontinued by developer): BOX
2. Describe the current use (select all that apply):
   • Accreditation (accountability)
   • Internal quality improvement
   • Decision-making (accountability)
   • External quality oversight (accountability)
   • Quality of care reporting
   • Research
   • Not being used
   • Other
3. If you answered “Other” to the preceding question, specify what you mean: BOX
4. If not in use, specify the reason(s): BOX
5. Describe the care setting(s) in which the QM is employed: BOX
6. Who are the professional(s) most likely to use this QM? BOX
7. Additional comments: BOX

DEVELOPMENT:

1. Who developed this QM? BOX
2. How was the search for evidence performed to support the QM? BOX
3. Type of evidence supporting the measure (select all that apply):
   • Clinical practice guideline: evidence-based
   • Clinical practice guideline: consensus-based
   • Systematic review
   • Selective/narrative review (e.g., manual search of literature)
   • Expert (consensus) panel process
   • Other
4. If you answered “Other” to the preceding question, specify what you mean: BOX
5. Bibliographic databases searched: BOX
6. How was the evidence appraised (e.g., grading quality or level of evidence) BOX
7. How was the wording/phrasing of the QM initially formulated (e.g., expert consensus)? BOX
8. How was the wording/phrasing of the QM refined? BOX
9. How was the QM pilot-tested? BOX
10. Which methods of case identification (e.g., cancer registries; claims databases) were used to pilot test the QM? Please name them. BOX
11. Which data sources (e.g., patient self-report; medical records), per method of case identification, were used to pilot test the QM? BOX
12. Denominator time window (time period in which patients are reviewed for inclusion in the denominator), per method of case identification: BOX
13. Numerator time window (time period in which patients are reviewed for inclusion in the numerator), per method of case identification: BOX
14. Sample description, per set of identified cases (e.g., national convenience sample of women with…): BOX
15. Response rate (cases with complete data/eligible cases), per set of identified cases: BOX
16. Specify reasons for exclusion (& sample size), per set of identified cases (NOTE: the need for individualized care that contradicts recommended care would constitute an exclusion yet may not indicate poor/inappropriate care): BOX
17. Size of sample(s) analyzed, per set of identified cases: BOX
18. Specify tumor characteristics: BOX
19. Specify family history of breast cancer (first degree members), per set of identified cases: BOX
20. Specify proportion of patients in each stage of the disease at the time of study, per set of identified cases: BOX
21. Specify duration of the disease since diagnosis, per set of identified cases: BOX
22. Year diagnosed, per set of identified cases: BOX
23. How diagnosed, per set of identified cases: BOX
24. Sample age, per set of identified cases: BOX
25. Sample socioeconomic status, per set of identified cases: BOX
26. Sample race/ethnicity, per set of identified cases: BOX
27. Other demographic factors (e.g., location of permanent residence), per set of identified cases: BOX
28. Specify population's treatments (including surgery, radiotherapy, & systemic therapy), per set of identified cases: BOX
29. Specify type(s) of surgery (breast conserving surgery, mastectomy with or without reconstructive surgery, etc.), per set of identified cases: BOX
30. Specify the psychometric properties of the QM established through pilot-testing (e.g., reliability; validity; sensitivity; specificity), per set of identified cases: BOX
31. Specify the psychometric properties of the QM established through pilot-testing (e.g., reliability; validity; sensitivity; specificity), per stratification, per set of identified cases: BOX
32. Specify the evidence regarding the adaptability of the QM (e.g., its applicability in different contexts/settings re breast cancer): BOX
33. Specify the evidence regarding the nature and adequacy of the (e.g., risk) adjustment(s) when cross-population or -database comparisons are made: BOX
34. What additional explicit conditions of use are specified for this QM (e.g., sample size, settings)? BOX
35. Specify any changes made to the QM following pilot-testing? BOX
36. Overall concordance rate, per set of identified cases: BOX
37. Variations in rate of concordance according to stratification(s) of the population (e.g., age; vulnerable populations), per set of identified cases: BOX
38. Specify scientific evidence demonstrating a linkage to improvement in clinical or patient-reported outcomes, per set of identified cases: BOX
39. Results involving scores (e.g., QOL), per set of identified cases (i.e., overall score, with interpretation; scores per group identified by stratification; results reflecting possible differences between groups identified by stratification: e.g., differences in outcome [e.g., survival] associated with receipt/non-receipt of care “X”): BOX
40. Additional comments: BOX
41. Comments box

Trajectory of Scientific Development of Quality Indicators Used in Quality Measurement

Instructions: Select a level, then all sub-levels that apply from within it (e.g., I-ac).

Level I. Information/data are reported indicating that the quality indicator used in the present study to measure quality was developed prior to its implementation in the present study, according to the scientific principles by which any measure is formally developed (e.g., pilot testing, with appropriate rigor and data sources, its feasibility and ease of use, reliability, internal validity, etc.) AND :

1. (reference to) data from the pre-study developmental process indicate consistently sound ¹ psychometric properties (e.g., reliability; internal validity);
2. (reference to) data from the pre-study developmental process indicate consistently or inconsistently unsound psychometric properties;
3. data obtained/reported in the present study indicate consistently sound psychometric properties;
4. data obtained/reported in the present study indicate consistently or inconsistently unsound psychometric properties; or,
5. no pre-study developmental or study-related psychometric data are referred to or reported.

Level II. Information/data are reported indicating that the quality indicator used in the present study to measure quality was being actively developed in the present study, according to the scientific principles by which any measure is formally developed, AND :

1. data obtained/reported in the present study indicate consistently sound psychometric properties;
2. data obtained/reported in the present study indicate consistently or inconsistently unsound psychometric properties;
3. no study-related psychometric data are reported.

Level III. No information/data are reported indicating that the quality indicator used in the present study to measure quality has been, or in this study was being, developed according to the scientific principles by which any measure is formally developed, YET :

1. data obtained/reported in the present study indicate consistently sound psychometric properties;
2. data obtained/reported in the present study indicate consistently or inconsistently unsound psychometric properties;
3. no study-related psychometric data are reported.

Level IV. There is no (reference to) pre-study developmental or study-related evidence indicating that the quality indicator used in the present study to measure quality was ever developed according to the scientific principles by which any measure is formally developed.

Footnotes

1

Note that, after the final search update, and directly relating to the narrowed focus whereby CPGs and SRs were now outside the scope of the review, the eForm was modified to allow screeners to flag these two types of report for exclusion (see Modified QUOROM Flow Chart). This precluded having to order articles that, via the initial screening of pre-update bibliographic records, had been allowed to pass on to Level 2 screening.

2

These literature searches had not yet been conducted before the scope was narrowed.

3

Active solicitation had not yet begun before the scope was narrowed.

4

The key difference between Level 1 and Level 2 screening was that, in the latter, the following question was added to the three used in the former.

5

Only this one question was asked.

1

Consistently sound describes a situation involving a given psychometric property (e.g., inter-observer reliability; construct validity) observed across more than one study (e.g., two studies report sound reliability) and/or to different psychometric properties observed either within one study or across more than one study (e.g., sound reliability; sound construct validity). Consistently unsound refers to a situation involving a given psychometric property observed across more than one study (e.g., both report unsound reliability) and/or to different psychometric properties observed either within one study or across more than one study (e.g., unsound reliability; unsound construct validity). Inconsistently unsound points to a situation involving a given psychometric property observed across more than one study (e.g., one reports sound reliability while another reports unsound reliability) and/or to different psychometric properties observed either within one study or across more than one study (e.g., sound reliability; unsound construct validity).