Appendix B: Quality of Evidence Grids

Suzanne West; Valerie King; Timothy S Carey; Kathleen N Lohr; Nikki McKoy; Sonya F Sutton; Linda Lux

This publication is provided for historical reference only and the information may be out of date.

Appendix B: Quality of Evidence Grids

Publication Details

Table

Quality Grid 1A. Evaluation of Quality Rating Systems for Systematic Reviews.

Table

Quality Grid 1B. Description of Quality Rating Systems for Systematic Reviews.

Table

Quality Grid 2A. Evaluation of Quality Rating Systems for Randomized Controlled Trails.

Table

Quality Grid 2B. Description of Quality Rating Systems for Randomized Controlled Trails.

Table

Quality Grid 3A. Evaluation of Quality Rating Systems for Observational Studies.

Table

Quality Grid 3B. Description of Quality Rating Systems for Observational Studies.

Table

Quality Grid 4A. Evaluation of Quality Rating Systems for Diagnostic Studies.

Table

Quality Grid 4B. Description of Quality Rating Systems for Diagnostic Studies.

Publication Details

Copyright

Publisher

Agency for Healthcare Research and Quality (US), Rockville (MD)

NLM Citation

West S, King V, Carey TS, et al. Systems to Rate the Strength Of Scientific Evidence. Rockville (MD): Agency for Healthcare Research and Quality (US); 2002 Apr. (Evidence Reports/Technology Assessments, No. 47.) Appendix B: Quality of Evidence Grids.

Quality Grid 1A. Evaluation of Quality Rating Systems for Systematic Reviews

Instrument	Domains
Instrument	Study Question	Search Strategy	Inclusion/Exclusion	Interventions	Outcomes	Data Extraction	Study Quality/Validity	Data Synthesis & Analysis	Results	Discussion	Funding/Support
Oxman and Guyett, 1991⁴; Oxman et al., 1991⁵
Irwig et al., 1994⁶
Oxman et al., 1994¹⁵
Cook et al., 1995¹⁶
Sacks et al., 1996⁷
Auperin et al., 1997⁸
Beck, 1997⁹
Cranney et al., 1997¹⁷
de Vet et al., 1997¹⁸
Smith, 1997¹⁰
Barnes and Bero, 1998³
Pogue and Yusuf, 1998¹⁹
Sutton et al., 1998²⁰
Clarke and Oxman, 1999¹¹
Moher et al., 1999²¹
Khan et al., 2000¹²
New Zealand Guidelines Group, 2000¹³
NHMRC, 2000²²
Harbour and Miller 2001¹⁴
Stroup et al., 2000²³

: Note: For complete reference information, see reference list

Quality Grid 1B. Description of Quality Rating Systems for Systematic Reviews

Instrument	Description of Instrument to Assess Study Quality
Instrument	Generic or specific instrument	Type of instrument?	Quality concept discussed	Method used to select items	Rigorous development process	Inter-rater reliability reported	Instructions provided for instrument use?
Oxman and Guyatt 1991⁴; Oxman et al., 1991⁵	Generic	Checklist	No	Accepted	No	ICC=0.71 (95%CI:0.59-0.81)	Yes
Irwig et al., 1994⁶	Generic	Checklist	No	Accepted	No	No	Partial
Oxman et al., 1994¹⁵	Generic	Guidance	Partial	Accepted	No	No	Partial
Cook et al., 1995¹⁶	Generic	Guidance	Partial	Both	Partial	No	Yes
Sacks et al., 1996⁷	Generic	Checklist	No	Modified Sacks, et al., 1987¹³⁷	No	No	Yes
Auperin et al., 1997⁸	Generic	Checklist	No	Modified Sacks, et al., 1987¹³⁷	No	ICC = 0.89-0.96	Partial
Beck, 1997⁹	Generic	Checklist	No	Modified multiple sources	No	% Agreement 87-89%	No
Cranney et al., 1997¹⁷	Generic	Guidance	No	Modified Victor, 1995¹³⁸ and Cook, 1995¹⁶	Partial	No	No
de Vet et al., 1997¹⁸	Specific	Guidance	Partial	Accepted	No	No	Partial
Smith, 1997¹⁰	Generic	Checklist	No	Modified Mulrow 1987⁹⁴ and Oxman, et al., 1994¹⁵	No	No	Partial
Barnes and Bero, 1998³	Generic	Scale	Partial	Modified Oxman, et al., 1994¹⁵	No	r = 0.87	No
Pogue and Yusuf, 1998¹⁹	Generic	Guidance	Partial	Accepted	No	No	Partial
Sutton et al., 1998²⁰	Generic	Guidance	Yes	Modified multiple sources	No	No	Partial
Clarke and Oxman, 1999¹¹	Generic	Checklist	No	Both	Partial	No	Partial
Moher et al., 1999²¹	Generic	Guidance	Yes	Both	Partial	No	Partial
Khan et al., 2000¹²	Generic	Checklist	Yes	Both	No	No	Partial
New Zealand Guidelines Group, 2000¹³	Generic	Checklist	No	Both	No	No	Partial
NHMRC, 2000²²	Generic	Guidance	Yes	Modified Clarke and Oxman (1999)¹¹	No	No	Partial
Harbour and Miller 2001¹⁴	Generic	Checklist	Yes	Both	Partial	No	Yes
Stroup et al, 2000²³	Generic	Guidance	Partial	Both	No	No	Partial

: Note: For complete reference information, see reference list ICC = intraclass correlation coefficient k = kappa R = correlation coefficient

Quality Grid 2A. Evaluation of Quality Rating Systems for Randomized Controlled Trails

Instrument	Domains
Instrument	Study Question	Study Population	Randomization	Blinding	Interventions	Outcomes	Statistical Analysis	Results	Discussion	Funding/Support
Chalmers et al., 1981²⁴
DerSimonian et al., 1982⁴³
Evans and Pollock, 1985²⁵
Liberati et al., 1986²⁶
Poynard et al., 1987⁴⁴
Prendiville et al., 1988⁵²
Colditz et al., 1989²⁷
Gotzsche, 1989²⁸
Reisch et al., 1989⁴⁵
Imperiale and McCullough, 1990⁴⁶
Spitzer et al., 1990⁴⁷
Kleijnen et al., 1991²⁹
Detsky et al., 1992³⁰
Guyatt et al., 1993⁵⁴; Guyatt et al., 1994⁵³
Cho and Bero, 1994³¹
Goodman et al., 1994³²
Standards of Reporting Trials Group, 1994⁵⁵
Fahey et al., 1995³³
Schulz et al., 1995⁵¹
Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature, 1996⁵⁶
Moher et al., 2001⁵⁷
Jadad et al., 1996³⁴
Khan et al., 1996³⁵
van der Heijden et al., 1996³⁶
Bender and Halpern, 1997³⁷
de Vet et al., 1997¹⁸
Sindhu et al., 1997³⁸
van Tulder et al., 1997³⁹
Downs and Black, 1998⁴⁰
Moher et al., 1998⁴¹
Verhagen et al., 1998⁴⁸
Clarke and Oxman, 1999¹¹
Lohr and Carey, 1999¹
Khan et al., 2000¹²
New Zealand Guidelines Group, 2000¹³
NHMRC, 2000⁴⁹
Harbour and Miller 2001¹⁴
Turlik and Kushner, 2000⁴²
Zaza et al., 2000⁵⁰
EPC Quality Assessments
Aronson et al., 1999⁵⁸
Chestnut et al., 1999⁶⁰
Jadad et al., 1999⁶¹
Heidenreich et al., 1999⁶²
Mulrow et al., 1999⁶³
Vickrey et al., 1999⁶⁴
West et al., 1999⁶⁵
McNamara et al., 2001⁶⁶
Ross et al., 2000⁶⁷
Goudas et al., 2000⁶⁸ Lau et al., 2000⁵⁹

: Note: For complete reference information, see reference list

Quality Grid 2B. Description of Quality Rating Systems for Randomized Controlled Trails

Instrument	Description of Instrument to Assess Study Quality
Instrument	Generic or specific instrument	Type of instrument?	Quality concept discussed	Method used to select items	Rigorous development process	Inter-rater reliability reported	Instructions provided for instrument use
Chalmers et al., 1981²⁴	Generic	Scale	Yes	Accepted	No	No	Yes
DerSimonian et al., 1982⁴³	Generic	Checklist	Partial	Accepted	No	% Agreement 51-82%	Partial
Evans and Pollock, 1985²⁵	Generic	Scale	No	Accepted	No	No	Yes
Liberati et al., 1986²⁶	Generic	Scale	Yes	Modified Chalmers et al., 1981²⁴	No	No	Partial
Poynard et al., 1987⁴⁴	Generic	Checklist	No	Modified Chalmers 1981²⁴	No	No	No
Prendiville et al., 1988⁵²	Generic	Guidance	Yes	Accepted	No	No	Yes
Colditz et al., 1989²⁷	Generic	Scale	Yes	Modified DerSimonian et al., 1982⁴³	Partial	No	Partial
Gotzsche, 1989²⁸	Specific	Scale	No	Accepted	No	No	Yes
Reisch et al., 1989⁴⁵	Generic	Checklist	Partial	Accepted	No	Partial	Yes
Imperiale et al., 1990⁴⁶	Generic	Checklist	Yes	Accepted	No	k = 0.79	No
Spitzer et al., 1990⁴⁷	Generic	Checklist	Partial	Accepted	No	No	No
Kleijnen et al., 1991²⁹	Generic	Scale	Partial	Accepted	No	Partial	Partial
Detsky et al., 1992³⁰	Generic	Scale	Yes	Accepted	Partial	ICC = 0.92	Partial
Guyatt et al., 1993⁵⁴; Guyatt et al., 1994⁵³	Generic	Guidance	No	Accepted	No	No	Partial
Cho and Bero, 1994³¹	Generic	Scale	Yes	Modified Spitzer et al., 1990⁴⁷	Partial	r = 0.60 ± 0.13	Partial
Goodman et al., 1994³²	Generic	Scale	Yes	Both	Partial	ICC = 0.25	Yes
Standards of Reporting Trials Group, 1994⁵⁵	Generic	Guidance	Yes	Both	Partial	No	Yes
Fahey et al., 1995³³	Generic	Scale	Partial	Modified Clarke and Oxman (1999)¹¹	No	No	No
Schulz et al., 1995⁵¹	Generic	Component	Yes	Accepted	Yes	Partial	Yes
Asilomar Working Group, 1996⁵⁶	Generic	Guidance	No	Both	Partial	No	No
Moher et al., 2001⁵⁷	Generic	Guidance	Yes	Both	Partial	No	Yes
Jadad et al., 1996³⁴	Generic	Scale	Yes	Empiric	Yes	ICC = 0.66 (95% CI: 0.53-0.79)	Yes
Khan et al., 1996³⁵	Generic	Scale	Yes	Modified Jadad et al., 1996³⁴	Yes	k = 0.70-0.94	Yes
van der Heijden et al., 1996³⁶	Specific	Scale	Partial	Accepted	No	No	Yes
Bender et al., 1997³⁷	Generic	Scale	Partial	Modified Jadad et al., 1996³⁴	Yes	ICC = 0.85	Partial
de Vet et al., 1997¹⁸	Generic	Scale	Partial	Accepted	No	No	No
Sindhu et al., 1997³⁸	Generic	Scale	No	Both	Yes	R = 0.90-0.99	Partial
van Tulder et al., 1997³⁹	Generic	Scale	No	Accepted	No	No	No
Downs and Black, 1998⁴⁰	Generic	Scale	Partial	Both	Yes	r = 0.75	Yes
Moher et al., 1998⁴¹	Generic	Scale	Yes	Modified Jadad et al., 1996,³⁴ and Schulz et al., 1995⁵¹	Yes	No	Partial
Verhagen et al., 1998⁴⁸	Generic	Checklist	Yes	Both	Partial	No	No
Clarke and Oxman, 1999¹¹	Generic	Guidance	Yes	Both	No	No	Partial
Lohr and Carey, 1999¹	Generic	Guidance	Yes	Accepted	No	No	No
Khan et al., 2000¹²	Generic	Checklist	Yes	Both	No	No	Partial
New Zealand Guidelines Group, 2000¹³	Generic	Checklist	No	Both	No	No	Partial
NHMRC, 2000⁴⁹	Generic	Checklist	Yes	Both	No	No	No
Harbour and Miller 2001¹⁴	Generic	Checklist	Yes	Both	Partial	No	Yes
Turlik and Kushner, 2000⁴²	Specific	Scale	No	Both	No	No	No
Zaza et al., 2000⁵⁰	Generic	Checklist	No	Accepted	No	% Agreement 65.2-85.6 % (Median= 79.5%)	Yes
EPC Quality Assessments
Aronson et al., 1999⁵⁸	Specific	Checklist	No	Both	No	No	Partial
Chestnut et al., 1999⁶⁰	Specific	Checklist	No	Both	No	No	Partial
Jadad et al., 1999⁶¹	Specific	Scale	Yes	Both	No	No	Partial
Heidenreich et al., 1999⁶²	Specific	Checklist	No	Both	No	No	No
Mulrow et al., 1999⁶³	Specific	Scale	No	Both	No	No	Yes
Vickrey et al., 1999⁶⁴	Generic	Scale	Yes	Empiric	Yes	No	Yes
West et al., 1999⁶⁵	Specific	Scale	Yes	Accepted	Partial	k=0.66-0.88	Yes
McNamara et al., 2001⁶⁶	Specific	Scale	Partial	Modified Detsky et al., (1992)³⁰ and Clarke and Oxman (1999)¹¹	No	Partial	Partial
Ross et al., 2000⁶⁷	Generic	Scale	Yes	Modified Jadad et al., 1996³⁴	No	Partial	Yes
Goudas et al., 2000⁶⁸ Lau et al., 2000⁵⁹	Generic	Component	Yes	Accepted	No	No	Partial

: Note: For complete reference information, see reference list ICC = intraclass correlation coefficient k = kappa R = correlation coefficient

Quality Grid 3A. Evaluation of Quality Rating Systems for Observational Studies

Instrument	Domains
Instrument	Study Question	Study Population	Comparability Of Subjects	Exposure/Intervention	Outcome Measure	Statistical Analysis	Results	Discussion	Funding
Reisch et al., 1989⁴⁵
Spitzer et al., 1990⁴⁷
Carruthers et al., 1993⁷¹
Cho and Bero, 1994³¹
Goodman et al., 1994³²
Laupacis et al., 1994⁷²
Levine et al., 1994⁷³
Downs and Black, 1998⁴⁰
Angelillo and Villari, 1999⁷⁴
Corrao et al., 1999⁶⁹
Lohr and Carey, 1999¹
Ariens et al., 2000⁷⁰
Khan et al., 2000¹²
New Zealand Guidelines, 2000¹³
NHMRC, 2000⁴⁹
Harbour and Miller 2001¹⁴
Zaza et al., 2000⁵⁰
EPC Quality Assessments
Chestnut et al., 1999⁶⁰
Vickrey et al., 1999⁶⁴

: Note: For complete reference information, see reference list

Quality Grid 3B. Description of Quality Rating Systems for Observational Studies

Instrument	Description of Instrument to Assess Study Quality
Instrument	Generic or specific instrument	Type of instrument?	Quality concept discussed	Method used to select items	Rigorous development process	Inter-rater reliability reported	Instructions provided for instrument use?
Reisch et al., 1989⁴⁵	Generic	Checklist	Partial	Accepted	No	Partial	Yes
Spitzer et al., 1990⁴⁷	Generic	Checklist	Partial	Accepted	No	No	No
Carruthers et al., 1993⁷¹	Generic	Guidance	No	Accepted	No	No	No
Cho and Bero, 1994³¹	Generic	Scale	Yes	Modified Spitzer et al., 1990⁴⁷	Partial	r = 0.60 ± 0.13	Yes
Goodman et al., 1994³²	Generic	Scale	Yes	Both	Partial	ICC = 0.25	Yes
Laupacis et al., 1994⁷²	Generic	Guidance	No	Accepted	No	No	Partial
Levine et al., 1994⁷³	Generic	Guidance	No	Accepted	No	No	Partial
Downs et al., 1998⁴⁰	Generic	Scale	Partial	Both	Yes	r = 0.75	Yes
Angelillo and Villari, 1999⁷⁴	Specific	Guidance	Partial	Modified Chalmers et al., 1981,²⁴ and Longnecker, 1988¹³⁹	No	No	No
Corrao et al., 1999⁶⁹	Specific	Scale	No	Accepted	No	No	No
Lohr and Carey, 1999¹	Generic	Guidance	Yes	Accepted	No	No	No
Ariens et al., 2000⁷⁰	Specific	Checklist	Yes	Accepted	Partial	% Agreement between 2 reviewers = 84%	Partial
Khan et al., 2000¹²	Generic	Checklist	Yes	Accepted	No	No	Partial
New Zealand Guidelines Group, 2000¹³	Generic	Checklist	No	Both	No	No	Partial
NHMRC, 2000⁴⁹	Generic	Checklist	Yes	Both	No	No	Partial
Harbour and Miller 2001¹⁴	Generic	Checklist	Yes	Both	Partial	No	Yes
Zaza et al., 2000⁵⁰	Generic	Checklist	No	Accepted	No	% Agreement 65.2-85.6% (Median = 79.5%)	Yes
EPC Quality Assessments
Chestnut et al., 1999⁶⁰	Specific	Checklist	No	Both	No	No	Partial
Vickrey et al., 1999⁶⁴	Generic	Scale	No	Accepted	No	No	Partial

: Note: For complete reference information, see reference list ICC = intraclass correlation coefficient k = kappa R = correlation coefficient

Quality Grid 4A. Evaluation of Quality Rating Systems for Diagnostic Studies

Instrument	Domains
Instrument	Study Population	Adequate Description of Test	Appropriate Reference Standard	Blinded Comparison of Test and Reference	Avoidance of Verification Bias
Sheps and Schechter, 1984⁷⁵; Arroll et al., 1988⁷⁶
Begg, 1987¹⁴⁰
Working Group on methods for prognosis and decision making, 1990¹⁴¹
Hoffman et al., 1991¹¹¹
Pinson et al., 1991¹⁴²
Carruthers et al., 1993⁷¹
Jaeschke et al., 1994¹⁴³
Irwig et al., 1994⁶
Reid et al., 1995¹⁴⁴
Cochrane Methods Working Group, 1996⁷⁷
Bruns, 1997¹⁴⁵
Lijmer et al., 1999⁷⁸
Khan et al., 2000¹²
NHMRC, 2000⁴⁹
Harbour and Miller, 2001¹⁴
EPC Quality Assessments
McCrory et al., 1999⁷⁹
Ross et al., 1999⁸⁰
Goudas et al., 2000;⁶⁸ Lau et al., 2000⁵⁹

: Note: For complete reference information, see reference list

Quality Grid 4B. Description of Quality Rating Systems for Diagnostic Studies

Instrument	Description of Instrument to Assess Study Quality
Instrument	Generic or specific instrument	Type of instrument?	Quality concept discussed	Method used to select items	Rigorous development process	Inter-rater reliability reported	Instructions provided for instrument use?
Sheps and Schechter, 1984⁷⁵; Arroll et al., 1988⁷⁶	Generic	Checklist	No	Accepted	No	k = 0.81-1.0	Partial
Begg, 1987¹⁴⁰	Generic	Guidance	Partial	Accepted	No	No	Partial
Working Group, 1990¹⁴¹	Generic	Guidance	No	Accepted	Partial	No	Partial
Hoffman et al., 1991¹¹¹	Specific	Checklist	Partial	Based on multiple other systems	No	k = 0.61	Yes
Pinson et al., 1991¹⁴²	Specific	Guidance	No	Modified Becker, 1989¹⁴⁶	No	No	Yes
Carruthers et al., 1993⁷¹	Generic	Guidance	No	Accepted	No	No	no
Jaeschke et al., 1994¹⁴³	Generic	Guidance	No	Accepted	No	No	Partial
Irwig et al., 1994⁶	Generic	Guidance	Yes	Accepted	No	No	Partial
Reid et al., 1995¹⁴⁴	Generic	Guidance	Yes	Accepted	No	No	Yes
Cochrane Methods Working Group, 1996⁷⁷	Generic	Checklist	Partial	Accepted	No	No	Yes
Bruns, 1997¹⁴⁵	Generic	Guidance	Partial	Accepted	Partial	No	Partial
Lijmer et al., 1999⁷⁸	Generic	Checklist	Yes	Both	Partial	No	Partial
Khan et al., 2000¹²	Generic	Checklist	Yes	Both	No	No	Partial
NHMRC, 2000⁴⁹	Generic	Checklist	Yes	Modified Clarke and Oxman, 1999¹¹	No	No	Partial
Harbour and Miller 2001¹⁴	Generic	Checklist	Yes	Both	Partial	No	Yes
EPC Quality Assessments
McCrory et al., 1999⁷⁹	Generic	Scale	No	Accepted	No	No	Partial
Ross et al., 1999⁸⁰	Specific	Scale	Yes	Accepted	Partial	No	Partial
Goudas et al., 2000⁶⁸, Lau et al., 2000⁵⁹	Generic	Component	Yes	Accepted	No	No	Partial

: Note: For complete reference information, see reference list ICC = intraclass correlation coefficient k = kappa R = correlation coefficient